Jack,
Just wrote a little program to create a list with 10,000,000 lines of 10 bytes each and then used ED to access it, with the following results. >ED &SAVEDLISTS& BIG 10000000 lines long. ----: 0001: 0002055482 ----: SIZE Size of "BIG" is 10000000 lines and 109999999 bytes (characters). This is UV 10.2.2 running on Solaris 10. Not had any trouble with any command (GET.LIST, EDIT.LIST etc.). Can you open the files with vi? If not then the issue would appear to be with AIX, if you can then it looks like an issue with the AIX build of UV. Cheers, Andy ________________________________ From: jig <[email protected]> To: [email protected] Sent: Monday, 6 June 2011, 14:02 Subject: Re: [U2] Universe 11.1.2 list line limit Thank you Jordan, You're right about OPENSEQ and READSEQ. However it depends on what you want to do with these long lists. What we do is either merge the lists or find the difference or find the union. And the way we have done it in the past is extremely efficient. For instance if we want to find the difference between two lists we run LIST.DIFF etc. However in order to do that the system sometimes has to read both lists. With normal lists up to 1 million elements it takes 1 or 2 seconds, but with larger lists it just falls over. To do the same using READSEQ it would take much longer than 1 or 2 secs and we only have a couple of secs to play with. In the old PICK we used to do it with User exits (Assembler code). But the LIST.DIFF, LIST.INTER etc are just as efficient when they work. The issue is not the how to do it but the why it fails so abruptly and what are the limits ? Also is there a uvconfig variable that affects or can improve such a thing ? These are my specific questions. thanks for your suggestion anyway. jack David Jordan wrote: > > Hi Jack > > I would use the OPENSEQ and READSEQ to deal with such files. The problem > with ED and READ apart from loading the whole record into memory is that > to go to attribute 1M it goes to the start and then counts a million field > marks. Then you want to read 1,000,001 it goes back to the beginning and > reads 1,000,001 field marks it is extremely slow and unworkable. > READSEQ, WRITESEQ leaves a pointer from the last attribute position. You > have to process these files sequential but it would read 40MB in > nanoseconds, far more efficient. > > Regards > > David Jordan > > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of jig > Sent: Monday, 6 June 2011 8:15 PM > To: [email protected] > Subject: Re: [U2] Universe 11.1.2 list line limit > > > OK sorry Kate, > > I should have given a little more information. > > On our system (AIX v5.3) we recently upgraded to 11.1.2 of UV from v10.3, > and we deal a lot with lists. Large lists with hundreds of thousands or > even millions of lines (elements). These are just lists of keys to records > in files. One in particular is a fixed length seven byte key > (alphanumeric). > However the largest file has 3.5 million records in it. This means that > our lists can get to over 3 million lines ( this is about 40MB of a single > file in a Unix directory). > > On version 10.3 when a basic program or the ED editor was used to read one > of these lists (with anything over 2 million lines of keys in it) it would > just core dump and not be able to read it. The same thing occurred whether > one had the list in the &SAVEDLISTS& file or anywhere else. > > To get around this problem we wrote software to split these large lists. > However that was cumbersome and not always worked to specification. > > With UV 11.1.2 (and because according to Rocket they did a lot of work to > improve memory management) we thought we'd be better off, handling some of > these large lists. However it seems we were wrong. > A Basic routine cannot read a list that is over 4 million records in it. > It just falls over and the process dies. The AE editor is a little more > graceful with some information before it also dies trying to open any > large list of keys. > > I know there are ways around this issue, but we expected a lot better > performance from the new version. I dont have the exact error at the > moment, because I am not at work, but I can add it here if requested. One > can always try it themselves to see the error, because it happens every > time. > > I am sure most companies do not work with such large lists, but even so, > core dumping instead of a gracious exit is not good enough. This is the > first thing we learned when we started out as programmers decades ago. > > regards > jack > > > Kate Stanton wrote: >> >> What limit? I do not understand. TIA., Kate >> >> On 6 June 2011 19:32, jig <[email protected]> wrote: >>> >>> Wondering if anyone is aware of the line limit in a saved list for >>> the latest UV version? >>> >>> Has it improved since we went to 64-bit? >>> -- >>> View this message in context: >>> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p3178 >>> 1147.html Sent from the U2 - Users mailing list archive at >>> Nabble.com. >>> >>> _______________________________________________ >>> U2-Users mailing list >>> [email protected] >>> http://listserver.u2ug.org/mailman/listinfo/u2-users >>> >> >> >> >> -- >> Kate Stanton >> Walstan Systems Ltd >> 4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand >> Phone: + 64 9 360 5310 Mobile: + 64 21 400 486 >> Email: [email protected] >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> > > -- > View this message in context: > http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31782039.html > Sent from the U2 - Users mailing list archive at Nabble.com. > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > -- View this message in context: http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31783169.html Sent from the U2 - Users mailing list archive at Nabble.com. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
