This probably does not help that much... but I did find out just recently that a single unix process memory limit on an AIX 5.3 machine defaults to 256 MB.
This can be changed via an environment variable. However, in Universe 10.3.4, I know for a fact that if you change it, the Universe session will not even fire up. Have a looksy here: http://publib.boulder.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=/com.ibm.itame3.doc_5.1/am51_perftune113.htm Basically you can tell AIX with a Unix Environment variable to allow a single unix process to use up to 2 GB of memory. This is providing I'm reading that article right (there is my disclaimer) So... you could try the following from a unix prompt with a valid Universe account. -Login to a user with enough of a shell and environment to run Universe. Type the following at the shell prompt: LDR_CNTRL=MAXDATA=0x20000000 export LDR_CNTRL cd /valid-uv-account-path /path-to-uv-executable (on my machine its /usr/ibm/uv/bin/uv) If Universe fires up and takes you to TCL or what have you, then rejoice. If it crashes with some crazy error, then you are most likely right that they did not improve memory performance limits all that much on AIX at least. If it does let you in, that setting above increases the limit to 512 MB per process. Try and edit your list now and see what happens. You can experiment with any of those LDR_CNTRL values in that article all the way up to 2GB per process memory limit. If you only set that environment variable manually for your session, then you can't hurt anything by trying. Just don't set it in some global environment config file or script anywhere until you are sure that it works. On Mon, Jun 6, 2011 at 10:57 AM, andy baum <[email protected]> wrote: > Jack, > > > Just wrote a little program to create a list with 10,000,000 lines of 10 > bytes each and then used ED to access it, with the following results. > > > > >ED &SAVEDLISTS& BIG > 10000000 lines long. > > ----: > 0001: 0002055482 > ----: SIZE > Size of "BIG" is 10000000 lines and 109999999 bytes (characters). > > > This is UV 10.2.2 running on Solaris 10. Not had any trouble with any > command (GET.LIST, EDIT.LIST etc.). Can you open the files with vi? If not > then the issue would appear to be with AIX, if you can then it looks like an > issue with the AIX build of UV. > > > Cheers, > > Andy > > > > ________________________________ > From: jig <[email protected]> > To: [email protected] > Sent: Monday, 6 June 2011, 14:02 > Subject: Re: [U2] Universe 11.1.2 list line limit > > > Thank you Jordan, > > You're right about OPENSEQ and READSEQ. However it depends on what you want > to do with these long lists. > What we do is either merge the lists or find the difference or find the > union. And the way we have done it in the past is extremely efficient. > For instance if we want to find the difference between two lists we run > LIST.DIFF etc. However in order to do that the system sometimes has to read > both lists. With normal lists up to 1 million elements it takes 1 or 2 > seconds, but with larger lists it just falls over. To do the same using > READSEQ it would take much longer than 1 or 2 secs and we only have a > couple > of secs to play with. > In the old PICK we used to do it with User exits (Assembler code). But the > LIST.DIFF, LIST.INTER etc are just as efficient when they work. > > The issue is not the how to do it but the why it fails so abruptly and what > are the limits ? > Also is there a uvconfig variable that affects or can improve such a thing > ? > These are my specific questions. > > thanks for your suggestion anyway. > jack > > > David Jordan wrote: > > > > Hi Jack > > > > I would use the OPENSEQ and READSEQ to deal with such files. The problem > > with ED and READ apart from loading the whole record into memory is that > > to go to attribute 1M it goes to the start and then counts a million > field > > marks. Then you want to read 1,000,001 it goes back to the beginning and > > reads 1,000,001 field marks it is extremely slow and unworkable. > > READSEQ, WRITESEQ leaves a pointer from the last attribute position. You > > have to process these files sequential but it would read 40MB in > > nanoseconds, far more efficient. > > > > Regards > > > > David Jordan > > > > > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of jig > > Sent: Monday, 6 June 2011 8:15 PM > > To: [email protected] > > Subject: Re: [U2] Universe 11.1.2 list line limit > > > > > > OK sorry Kate, > > > > I should have given a little more information. > > > > On our system (AIX v5.3) we recently upgraded to 11.1.2 of UV from v10.3, > > and we deal a lot with lists. Large lists with hundreds of thousands or > > even millions of lines (elements). These are just lists of keys to > records > > in files. One in particular is a fixed length seven byte key > > (alphanumeric). > > However the largest file has 3.5 million records in it. This means that > > our lists can get to over 3 million lines ( this is about 40MB of a > single > > file in a Unix directory). > > > > On version 10.3 when a basic program or the ED editor was used to read > one > > of these lists (with anything over 2 million lines of keys in it) it > would > > just core dump and not be able to read it. The same thing occurred > whether > > one had the list in the &SAVEDLISTS& file or anywhere else. > > > > To get around this problem we wrote software to split these large lists. > > However that was cumbersome and not always worked to specification. > > > > With UV 11.1.2 (and because according to Rocket they did a lot of work to > > improve memory management) we thought we'd be better off, handling some > of > > these large lists. However it seems we were wrong. > > A Basic routine cannot read a list that is over 4 million records in it. > > It just falls over and the process dies. The AE editor is a little more > > graceful with some information before it also dies trying to open any > > large list of keys. > > > > I know there are ways around this issue, but we expected a lot better > > performance from the new version. I dont have the exact error at the > > moment, because I am not at work, but I can add it here if requested. One > > can always try it themselves to see the error, because it happens every > > time. > > > > I am sure most companies do not work with such large lists, but even so, > > core dumping instead of a gracious exit is not good enough. This is the > > first thing we learned when we started out as programmers decades ago. > > > > regards > > jack > > > > > > Kate Stanton wrote: > >> > >> What limit? I do not understand. TIA., Kate > >> > >> On 6 June 2011 19:32, jig <[email protected]> wrote: > >>> > >>> Wondering if anyone is aware of the line limit in a saved list for > >>> the latest UV version? > >>> > >>> Has it improved since we went to 64-bit? > >>> -- > >>> View this message in context: > >>> http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p3178 > >>> 1147.html Sent from the U2 - Users mailing list archive at > >>> Nabble.com. > >>> > >>> _______________________________________________ > >>> U2-Users mailing list > >>> [email protected] > >>> http://listserver.u2ug.org/mailman/listinfo/u2-users > >>> > >> > >> > >> > >> -- > >> Kate Stanton > >> Walstan Systems Ltd > >> 4 Kelmarna Ave, Herne Bay, Auckland 1011, New Zealand > >> Phone: + 64 9 360 5310 Mobile: + 64 21 400 486 > >> Email: [email protected] > >> _______________________________________________ > >> U2-Users mailing list > >> [email protected] > >> http://listserver.u2ug.org/mailman/listinfo/u2-users > >> > >> > > > > -- > > View this message in context: > > > http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31782039.html > > Sent from the U2 - Users mailing list archive at Nabble.com. > > > > _______________________________________________ > > U2-Users mailing list > > [email protected] > > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > > U2-Users mailing list > > [email protected] > > http://listserver.u2ug.org/mailman/listinfo/u2-users > > > > > > -- > View this message in context: > http://old.nabble.com/Universe-11.1.2-list-line-limit-tp31781147p31783169.html > Sent from the U2 - Users mailing list archive at Nabble.com. > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > -- John Thompson _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
