Hi everyone, Thanks for your help. We found the variable that makes a difference to handling large size saved lists in UV. Its the data segment size in Unix for AIX
After we increased the data seg default size by four times we can now edit these large lists. However there still is a limit but out of our immediate future requirements. Here's the new limit ulimit -a core file size (blocks, -c) 1048575 data seg size (kbytes, -d) 524288 file size (blocks, -f) unlimited max memory size (kbytes, -m) 32768 open files (-n) 2000 pipe size (512 bytes, -p) 64 stack size (kbytes, -s) 32768 cpu time (seconds, -t) unlimited max user processes (-u) 128 virtual memory (kbytes, -v) unlimited We tried doubling the ULIMIT variable in uvconfig but that didn't make any difference from what we have tested. Thanks to John Thompson's suggestion we found the issue in AIX. :clap: Hence another conclusion is that memory management changes recently performed by Rocket in this new release (v11.1.2) made no difference to the way its handling large lists in UV. regards jack John Thompson-15 wrote: > > This probably does not help that much... but I did find out just recently > that a single unix process memory limit on an AIX 5.3 machine defaults to > 256 MB. > > This can be changed via an environment variable. However, in Universe > 10.3.4, I know for a fact that if you change it, the Universe session will > not even fire up. > > Have a looksy here: > > http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm > > Basically you can tell AIX with a Unix Environment variable to allow a > single unix process to use up to 2 GB of memory. This is providing I'm > reading that article right (there is my disclaimer) > > So... you could try the following from a unix prompt with a valid Universe > account. > > -Login to a user with enough of a shell and environment to run Universe. > Type the following at the shell prompt: > LDR_CNTRL=MAXDATA=0x20000000 > export LDR_CNTRL > cd /valid-uv-account-path > /path-to-uv-executable (on my machine its /usr/ibm/uv/bin/uv) > > If Universe fires up and takes you to TCL or what have you, then rejoice. > If it crashes with some crazy error, then you are most likely right that > they did not improve memory performance limits all that much on AIX at > least. > > If it does let you in, that setting above increases the limit to 512 MB > per > process. Try and edit your list now and see what happens. You can > experiment with any of those LDR_CNTRL values in that article all the way > up > to 2GB per process memory limit. > > If you only set that environment variable manually for your session, then > you can't hurt anything by trying. Just don't set it in some global > environment config file or script anywhere until you are sure that it > works. > > > > On Mon, Jun 6, 2011 at 10:57 AM, andy baum <[email protected]> wrote: > >> Jack, >> >> >> Just wrote a little program to create a list with 10,000,000 lines of 10 >> bytes each and then used ED to access it, with the following results. >> >> >> >> >ED &SAVEDLISTS& BIG >> 10000000 lines long. >> >> ----: >> 0001: 0002055482 >> ----: SIZE >> Size of "BIG" is 10000000 lines and 109999999 bytes (characters). >> >> >> This is UV 10.2.2 running on Solaris 10. Not had any trouble with any >> command (GET.LIST, EDIT.LIST etc.). Can you open the files with vi? If >> not >> then the issue would appear to be with AIX, if you can then it looks like >> an >> issue with the AIX build of UV. >> >> >> Cheers, >> >> Andy >> >> >> >> ________________________________ >> From: jig <[email protected]> >> To: [email protected] >> Sent: Monday, 6 June 2011, 14:02 >> Subject: Re: [U2] Universe 11.1.2 list line limit >> >> >> Thank you Jordan, >> >> You're right about OPENSEQ and READSEQ. However it depends on what you >> want >> to do with these long lists. >> What we do is either merge the lists or find the difference or find the >> union. And the way we have done it in the past is extremely efficient. >> For instance if we want to find the difference between two lists we run >> LIST.DIFF etc. However in order to do that the system sometimes has to >> read >> both lists. With normal lists up to 1 million elements it takes 1 or 2 >> seconds, but with larger lists it just falls over. To do the same using >> READSEQ it would take much longer than 1 or 2 secs and we only have a >> couple >> of secs to play with. >> In the old PICK we used to do it with User exits (Assembler code). But >> the >> LIST.DIFF, LIST.INTER etc are just as efficient when they work. >> >> The issue is not the how to do it but the why it fails so abruptly and >> what >> are the limits ? >> Also is there a uvconfig variable that affects or can improve such a >> thing >> ? >> These are my specific questions. >> >> thanks for your suggestion anyway. >> jack >> >> >> David Jordan wrote: >> > >> > Hi Jack >> > >> > I would use the OPENSEQ and READSEQ to deal with such files. The >> problem >> > with ED and READ apart from loading the whole record into memory is >> that >> > to go to attribute 1M it goes to the start and then counts a million >> field >> > marks. Then you want to read 1,000,001 it goes back to the beginning >> and >> > reads 1,000,001 field marks it is extremely slow and unworkable. >> > READSEQ, WRITESEQ leaves a pointer from the last attribute position. >> You >> > have to process these files sequential but it would read 40MB in >> > nanoseconds, far more efficient. >> > >> > Regards >> > >> > David Jordan >> > >> > >> > >> > -----Original Message----- >> > From: [email protected] >> > [mailto:[email protected]] On Behalf Of jig >> > Sent: Monday, 6 June 2011 8:15 PM >> > To: [email protected] >> > Subject: Re: [U2] Universe 11.1.2 list line limit >> > >> > >> > OK sorry Kate, >> > >> > I should have given a little more information. >> > >> > On our system (AIX v5.3) we recently upgraded to 11.1.2 of UV from >> v10.3, >> > and we deal a lot with lists. Large lists with hundreds of thousands or >> > even millions of lines (elements). These are just lists of keys to >> records >> > in files. One in particular is a fixed length seven byte key >> > (alphanumeric). >> > However the largest file has 3.5 million records in it. This means that >> > our lists can get to over 3 million lines ( this is about 40MB of a >> single >> > file in a Unix directory). >> > >> > On version 10.3 when a basic program or the ED editor was used to read >> one >> > of these lists (with anything over 2 million lines of keys in it) it >> would >> > just core dump and not be able to read it. The same thing occurred >> whether >> > one had the list in the &SAVEDLISTS& file or anywhere else. >> > >> > To get around this problem we wrote software to split these large >> lists. >> > However that was cumbersome and not always worked to specification. >> > >> > With UV 11.1.2 (and because according to Rocket they did a lot of work >> to >> > improve memory management) we thought we'd be better off, handling some >> of >> > these large lists. However it seems we were wrong. >> > A Basic routine cannot read a list that is over 4 million records in >> it. >> > It just falls over and the process dies. The AE editor is a little more >> > graceful with some information before it also dies trying to open any >> > large list of keys. >> > >> > I know there are ways around this issue, but we expected a lot better >> > performance from the new version. I dont have the exact error at the >> > moment, because I am not at work, but I can add it here if requested. >> One >> > can always try it themselves to see the error, because it happens every >> > time. >> > >> > I am sure most companies do not work with such large lists, but even >> so, >> > core dumping instead of a gracious exit is not good enough. This is the >> > first thing we learned when we started out as programmers decades ago. >> > >> > regards >> > jack >> > >> > >> > Kate Stanton wrote: >> >> >> >> What limit? I do not understand. TIA., Kate >> >> >> >> On 6 June 2011 19:32, jig <[email protected]> wrote: >> >>> >> >>> Wondering if anyone is aware of the line limit in a saved list for >> >>> the latest UV version? >> >>> >> >>> Has it improved since we went to 64-bit? >> >>> -- >> >>> View this message in context: >> >>> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p3178 >> >>> 1147.html Sent from the U2 - Users mailing list archive at >> >>> Nabble.com. >> >>> >> >>> _______________________________________________ >> >>> U2-Users mailing list >> >>> [email protected] >> >>> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >>> >> >> >> >> >> >> >> >> -- >> >> Kate Stanton >> >> Walstan Systems Ltd >> >> 4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand >> >> Phone: + 64 9 360 5310 Mobile: + 64 21 400 486 >> >> Email: [email protected] >> >> _______________________________________________ >> >> U2-Users mailing list >> >> [email protected] >> >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> >> >> >> > >> > -- >> > View this message in context: >> > >> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31782039.html >> > Sent from the U2 - Users mailing list archive at Nabble.com. >> > >> > _______________________________________________ >> > U2-Users mailing list >> > [email protected] >> > http://listserver.u2ug.org/mailman/listinfo/u2-users >> > _______________________________________________ >> > U2-Users mailing list >> > [email protected] >> > http://listserver.u2ug.org/mailman/listinfo/u2-users >> > >> > >> >> -- >> View this message in context: >> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31783169.html >> Sent from the U2 - Users mailing list archive at Nabble.com. >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> > > > > -- > John Thompson > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > -- View this message in context: http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31787980.html Sent from the U2 - Users mailing list archive at Nabble.com. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
