Thanks Andy, Now that is very interesting. 10million lines right? And on Solaris it can be done? Wonderful.
Does it mean that AIX has the problem? I don't know. We have setup ulimit variables in Unix to unlimited already. Bash-3.00# ulimit -a core file size (blocks, -c) 1048575 data seg size (kbytes, -d) 131072 file size (blocks, -f) unlimited max memory size (kbytes, -m) 32768 open files (-n) 2000 pipe size (512 bytes, -p) 64 stack size (kbytes, -s) 32768 cpu time (seconds, -t) unlimited max user processes (-u) 262144 virtual memory (kbytes, -v) unlimited I will try the suggestion by John Thompson and if that solves the issue I will update this thread. However we have found that vi cannot open this size file either, but it is a vi limitation which we changed and vi had no problem in editing the file. appreciate all of your suggestions. I will report back if there's any solutions found. regards jack andy baum-2 wrote: > > Jack, > > > Just wrote a little program to create a list with 10,000,000 lines of 10 > bytes each and then used ED to access it, with the following results. > > > >>ED &SAVEDLISTS& BIG > 10000000 lines long. > > ----: > 0001: 0002055482 > ----: SIZE > Size of "BIG" is 10000000 lines and 109999999 bytes (characters). > > > This is UV 10.2.2 running on Solaris 10. Not had any trouble with any > command (GET.LIST, EDIT.LIST etc.). Can you open the files with vi? If not > then the issue would appear to be with AIX, if you can then it looks like > an issue with the AIX build of UV. > > > Cheers, > > Andy > > > > ________________________________ > From: jig <[email protected]> > To: [email protected] > Sent: Monday, 6 June 2011, 14:02 > Subject: Re: [U2] Universe 11.1.2 list line limit > > > Thank you Jordan, > > You're right about OPENSEQ and READSEQ. However it depends on what you > want > to do with these long lists. > What we do is either merge the lists or find the difference or find the > union. And the way we have done it in the past is extremely efficient. > For instance if we want to find the difference between two lists we run > LIST.DIFF etc. However in order to do that the system sometimes has to > read > both lists. With normal lists up to 1 million elements it takes 1 or 2 > seconds, but with larger lists it just falls over. To do the same using > READSEQ it would take much longer than 1 or 2 secs and we only have a > couple > of secs to play with. > In the old PICK we used to do it with User exits (Assembler code). But the > LIST.DIFF, LIST.INTER etc are just as efficient when they work. > > The issue is not the how to do it but the why it fails so abruptly and > what > are the limits ? > Also is there a uvconfig variable that affects or can improve such a thing > ? > These are my specific questions. > > thanks for your suggestion anyway. > jack > > > David Jordan wrote: >> >> Hi Jack >> >> I would use the OPENSEQ and READSEQ to deal with such files. The problem >> with ED and READ apart from loading the whole record into memory is that >> to go to attribute 1M it goes to the start and then counts a million >> field >> marks. Then you want to read 1,000,001 it goes back to the beginning and >> reads 1,000,001 field marks it is extremely slow and unworkable. >> READSEQ, WRITESEQ leaves a pointer from the last attribute position. You >> have to process these files sequential but it would read 40MB in >> nanoseconds, far more efficient. >> >> Regards >> >> David Jordan >> >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of jig >> Sent: Monday, 6 June 2011 8:15 PM >> To: [email protected] >> Subject: Re: [U2] Universe 11.1.2 list line limit >> >> >> OK sorry Kate, >> >> I should have given a little more information. >> >> On our system (AIX v5.3) we recently upgraded to 11.1.2 of UV from v10.3, >> and we deal a lot with lists. Large lists with hundreds of thousands or >> even millions of lines (elements). These are just lists of keys to >> records >> in files. One in particular is a fixed length seven byte key >> (alphanumeric). >> However the largest file has 3.5 million records in it. This means that >> our lists can get to over 3 million lines ( this is about 40MB of a >> single >> file in a Unix directory). >> >> On version 10.3 when a basic program or the ED editor was used to read >> one >> of these lists (with anything over 2 million lines of keys in it) it >> would >> just core dump and not be able to read it. The same thing occurred >> whether >> one had the list in the &SAVEDLISTS& file or anywhere else. >> >> To get around this problem we wrote software to split these large lists. >> However that was cumbersome and not always worked to specification. >> >> With UV 11.1.2 (and because according to Rocket they did a lot of work to >> improve memory management) we thought we'd be better off, handling some >> of >> these large lists. However it seems we were wrong. >> A Basic routine cannot read a list that is over 4 million records in it. >> It just falls over and the process dies. The AE editor is a little more >> graceful with some information before it also dies trying to open any >> large list of keys. >> >> I know there are ways around this issue, but we expected a lot better >> performance from the new version. I dont have the exact error at the >> moment, because I am not at work, but I can add it here if requested. One >> can always try it themselves to see the error, because it happens every >> time. >> >> I am sure most companies do not work with such large lists, but even so, >> core dumping instead of a gracious exit is not good enough. This is the >> first thing we learned when we started out as programmers decades ago. >> >> regards >> jack >> >> >> Kate Stanton wrote: >>> >>> What limit? I do not understand. TIA., Kate >>> >>> On 6 June 2011 19:32, jig <[email protected]> wrote: >>>> >>>> Wondering if anyone is aware of the line limit in a saved list for >>>> the latest UV version? >>>> >>>> Has it improved since we went to 64-bit? >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p3178 >>>> 1147.html Sent from the U2 - Users mailing list archive at >>>> Nabble.com. >>>> >>>> _______________________________________________ >>>> U2-Users mailing list >>>> [email protected] >>>> http://listserver.u2ug.org/mailman/listinfo/u2-users >>>> >>> >>> >>> >>> -- >>> Kate Stanton >>> Walstan Systems Ltd >>> 4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand >>> Phone: + 64 9 360 5310 Mobile: + 64 21 400 486 >>> Email: [email protected] >>> _______________________________________________ >>> U2-Users mailing list >>> [email protected] >>> http://listserver.u2ug.org/mailman/listinfo/u2-users >>> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31782039.html >> Sent from the U2 - Users mailing list archive at Nabble.com. >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> > > -- > View this message in context: > http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31783169.html > Sent from the U2 - Users mailing list archive at Nabble.com. > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > -- View this message in context: http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31787563.html Sent from the U2 - Users mailing list archive at Nabble.com. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
