Thanks John, It certainly sounds interesting and we will give it a go. Incidentally someone said that on Solaris there's no such issue, which initially points to an AIX limitation or an AIX implementation by UV.
Which ever way I will report back with what I found. Thanks again regards jack John Thompson-15 wrote: > > This probably does not help that much... but I did find out just recently > that a single unix process memory limit on an AIX 5.3 machine defaults to > 256 MB. > > This can be changed via an environment variable. However, in Universe > 10.3.4, I know for a fact that if you change it, the Universe session will > not even fire up. > > Have a looksy here: > > http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm > > Basically you can tell AIX with a Unix Environment variable to allow a > single unix process to use up to 2 GB of memory. This is providing I'm > reading that article right (there is my disclaimer) > > So... you could try the following from a unix prompt with a valid Universe > account. > > -Login to a user with enough of a shell and environment to run Universe. > Type the following at the shell prompt: > LDR_CNTRL=MAXDATA=0x20000000 > export LDR_CNTRL > cd /valid-uv-account-path > /path-to-uv-executable (on my machine its /usr/ibm/uv/bin/uv) > > If Universe fires up and takes you to TCL or what have you, then rejoice. > If it crashes with some crazy error, then you are most likely right that > they did not improve memory performance limits all that much on AIX at > least. > > If it does let you in, that setting above increases the limit to 512 MB > per > process. Try and edit your list now and see what happens. You can > experiment with any of those LDR_CNTRL values in that article all the way > up > to 2GB per process memory limit. > > If you only set that environment variable manually for your session, then > you can't hurt anything by trying. Just don't set it in some global > environment config file or script anywhere until you are sure that it > works. > > > > On Mon, Jun 6, 2011 at 10:57 AM, andy baum <[email protected]> wrote: > >> Jack, >> >> >> Just wrote a little program to create a list with 10,000,000 lines of 10 >> bytes each and then used ED to access it, with the following results. >> >> >> >> >ED &SAVEDLISTS& BIG >> 10000000 lines long. >> >> ----: >> 0001: 0002055482 >> ----: SIZE >> Size of "BIG" is 10000000 lines and 109999999 bytes (characters). >> >> >> This is UV 10.2.2 running on Solaris 10. Not had any trouble with any >> command (GET.LIST, EDIT.LIST etc.). Can you open the files with vi? If >> not >> then the issue would appear to be with AIX, if you can then it looks like >> an >> issue with the AIX build of UV. >> >> >> Cheers, >> >> Andy >> >> >> >> ________________________________ >> From: jig <[email protected]> >> To: [email protected] >> Sent: Monday, 6 June 2011, 14:02 >> Subject: Re: [U2] Universe 11.1.2 list line limit >> >> >> Thank you Jordan, >> >> You're right about OPENSEQ and READSEQ. However it depends on what you >> want >> to do with these long lists. >> What we do is either merge the lists or find the difference or find the >> union. And the way we have done it in the past is extremely efficient. >> For instance if we want to find the difference between two lists we run >> LIST.DIFF etc. However in order to do that the system sometimes has to >> read >> both lists. With normal lists up to 1 million elements it takes 1 or 2 >> seconds, but with larger lists it just falls over. To do the same using >> READSEQ it would take much longer than 1 or 2 secs and we only have a >> couple >> of secs to play with. >> In the old PICK we used to do it with User exits (Assembler code). But >> the >> LIST.DIFF, LIST.INTER etc are just as efficient when they work. >> >> The issue is not the how to do it but the why it fails so abruptly and >> what >> are the limits ? >> Also is there a uvconfig variable that affects or can improve such a >> thing >> ? >> These are my specific questions. >> >> thanks for your suggestion anyway. >> jack >> >> >> David Jordan wrote: >> > >> > Hi Jack >> > >> > I would use the OPENSEQ and READSEQ to deal with such files. The >> problem >> > with ED and READ apart from loading the whole record into memory is >> that >> > to go to attribute 1M it goes to the start and then counts a million >> field >> > marks. Then you want to read 1,000,001 it goes back to the beginning >> and >> > reads 1,000,001 field marks it is extremely slow and unworkable. >> > READSEQ, WRITESEQ leaves a pointer from the last attribute position. >> You >> > have to process these files sequential but it would read 40MB in >> > nanoseconds, far more efficient. >> > >> > Regards >> > >> > David Jordan >> > >> > >> > >> > -----Original Message----- >> > From: [email protected] >> > [mailto:[email protected]] On Behalf Of jig >> > Sent: Monday, 6 June 2011 8:15 PM >> > To: [email protected] >> > Subject: Re: [U2] Universe 11.1.2 list line limit >> > >> > >> > OK sorry Kate, >> > >> > I should have given a little more information. >> > >> > On our system (AIX v5.3) we recently upgraded to 11.1.2 of UV from >> v10.3, >> > and we deal a lot with lists. Large lists with hundreds of thousands or >> > even millions of lines (elements). These are just lists of keys to >> records >> > in files. One in particular is a fixed length seven byte key >> > (alphanumeric). >> > However the largest file has 3.5 million records in it. This means that >> > our lists can get to over 3 million lines ( this is about 40MB of a >> single >> > file in a Unix directory). >> > >> > On version 10.3 when a basic program or the ED editor was used to read >> one >> > of these lists (with anything over 2 million lines of keys in it) it >> would >> > just core dump and not be able to read it. The same thing occurred >> whether >> > one had the list in the &SAVEDLISTS& file or anywhere else. >> > >> > To get around this problem we wrote software to split these large >> lists. >> > However that was cumbersome and not always worked to specification. >> > >> > With UV 11.1.2 (and because according to Rocket they did a lot of work >> to >> > improve memory management) we thought we'd be better off, handling some >> of >> > these large lists. However it seems we were wrong. >> > A Basic routine cannot read a list that is over 4 million records in >> it. >> > It just falls over and the process dies. The AE editor is a little more >> > graceful with some information before it also dies trying to open any >> > large list of keys. >> > >> > I know there are ways around this issue, but we expected a lot better >> > performance from the new version. I dont have the exact error at the >> > moment, because I am not at work, but I can add it here if requested. >> One >> > can always try it themselves to see the error, because it happens every >> > time. >> > >> > I am sure most companies do not work with such large lists, but even >> so, >> > core dumping instead of a gracious exit is not good enough. This is the >> > first thing we learned when we started out as programmers decades ago. >> > >> > regards >> > jack >> > >> > >> > Kate Stanton wrote: >> >> >> >> What limit? I do not understand. TIA., Kate >> >> >> >> On 6 June 2011 19:32, jig <[email protected]> wrote: >> >>> >> >>> Wondering if anyone is aware of the line limit in a saved list for >> >>> the latest UV version? >> >>> >> >>> Has it improved since we went to 64-bit? >> >>> -- >> >>> View this message in context: >> >>> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p3178 >> >>> 1147.html Sent from the U2 - Users mailing list archive at >> >>> Nabble.com. >> >>> >> >>> _______________________________________________ >> >>> U2-Users mailing list >> >>> [email protected] >> >>> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >>> >> >> >> >> >> >> >> >> -- >> >> Kate Stanton >> >> Walstan Systems Ltd >> >> 4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand >> >> Phone: + 64 9 360 5310 Mobile: + 64 21 400 486 >> >> Email: [email protected] >> >> _______________________________________________ >> >> U2-Users mailing list >> >> [email protected] >> >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> >> >> >> > >> > -- >> > View this message in context: >> > >> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31782039.html >> > Sent from the U2 - Users mailing list archive at Nabble.com. >> > >> > _______________________________________________ >> > U2-Users mailing list >> > [email protected] >> > http://listserver.u2ug.org/mailman/listinfo/u2-users >> > _______________________________________________ >> > U2-Users mailing list >> > [email protected] >> > http://listserver.u2ug.org/mailman/listinfo/u2-users >> > >> > >> >> -- >> View this message in context: >> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31783169.html >> Sent from the U2 - Users mailing list archive at Nabble.com. >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> > > > > -- > John Thompson > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > -- View this message in context: http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31787675.html Sent from the U2 - Users mailing list archive at Nabble.com. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
