Harold You might want to look at splitting out some of those index records, e.g. where you have 10,000 fields in an index for CRIME, split this into CRIME, CRIME-1, CRIME-2 etc. with a maximum number of entries per index. Otherwise you are not going to get efficient storage at those sizes: if they are in a directory file these can be slow scanning and clashing on the lock table, and of course in a hashed file they will be in out of line overflow - again, slowing access. You can also adopt a scheme where, for example, the first entry in the base record holds the last sequence number of the series, so you don't have to read the intermediate ones when appending..
I realize this will mean changing the search routines, but it might help in the long run. I would also definitely echo what the others have said about sequentially accessing the source descriptions. Brian > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of > Oaks, Harold > Sent: Tuesday, July 21, 2009 4:54 PM > To: U2 Users List > Subject: Re: [U2] Universe just quits > > Barry: > Thanks for resonding. > > The main activity is running thru text files (narratives), > identifying the next word then updating a file (DEXNAR) used > to cross-reference all the words. For example, in record > ABC123 in the narrative field the next identified word is > DECOMPOSING. I read the record DECOMPOSING from the DEXNAR > file. The record DECOMPOSING has 10 lines in it, because 10 > other prior narratives had the word DECOMPOSING. The program > seeks the current record key (ABC123) in the fields using the > LOCATE command. If not found, the key ABC123 is appended to > the end of the DECOMPOSING record and DECOMPOSING it is > written out, now with 11 fields. > > Some of the records in DEXNAR get long as they are for fairly > common words, like ALIAS. (The most common words I do not > bother to cross-reference, of course, like the word THE). > Thus, the LOCATE is looking thru an ever increasing number of > fields as we go along. Some records currently have over > 11000 fields. Is that the problem? Maybe the sytem tries to > 'fit' the next very long record in memory to do the LOCATE > and it overflows something? > > The dropping out, however, occurs normally after updating > 25000+ records. Shorter runs seem to 'hold out'. What might > be building up in memory space? These are also overnight > runs, typically, so there are fewer users to contend with > making it unlikely that it's exceeded limits for all users, I > would think. > > The point of the cross-reference file is to allow users to > quickly search thru all the 700,000 available (crime) > narratives for the ones that match a set of entered search > words. Searching that many narratives using brute force > would probably take 5+ minutes for each search on our system, > which would really stifle the crime analysts. > > Thanks- > Harold > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Barry Rogen > Sent: Tuesday, July 21, 2009 1:03 PM > To: U2 Users List > Subject: Re: [U2] Universe just quits > > > Can you elaborate a little on these tasks > > Barry Rogen > PNY Technologies, Inc. > Senior Programmer/Analyst > (973) 515 - 9700 ext 5327 > bro...@pny.com > > ----------------------------------------------------- > We are continually faced with great opportunities > brilliantly disguised as insoluble problems. > > John W Gardner > ---------------------------------------------------------------- > P Before printing please think about your environmental responsibility > > > -----Original Message----- > From: u2-users-boun...@listserver.u2ug.org > [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of > Oaks, Harold > Sent: Tuesday, July 21, 2009 2:41 PM > To: u2-users@listserver.u2ug.org > Subject: [U2] Universe just quits > > I am having a very disconcerting problem. A long job I have > been running, processing a large text file and then loading a > Universe file, is simply quitting sometimes. No error > message of any kind, Universe just quits, the session drops > into unix. We have Universe 10.2 running over HPux 11.1. I > have reduced the job to smaller pieces to get thru it and > indicate data values so that I restart at about the dropout point. > So I'll be able to finish but I would like to understand > what's going on. > > Anybody seen this kind of thing before? Is there a Universe > parameter I should look at? A unix kernal parameter? > > Any ideas appreciated. > Thanks- > > Harold Oaks > Sr. Analyst/Programmer > Clark County Information Systems > Clark County, Washington > ph: (360) 397-6121 x4132 > > > > This e-mail and related attachments and any response may be > subject to public disclosure under state law. > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > 21/7/2009NOT INTENDED AS A SUBSTITUTE FOR A WRITING > > NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT MAY > BE A PART, OR IN ANY ATTACHMENTS THERETO, SHALL CONSTITUTE A > BINDING CONTRACT, OR ANY CONTRACTUAL OBLIGATION BY PNY, OR > ANY INTENT TO ENTER INTO ANY BINDING OBLIGATIONS, > NOTWITHSTANDING ANY ENACTMENT OF THE UNIFORM ELECTRONIC > TRANSACTIONS ACT, THE FEDERAL E-SIGN ACT, OR ANY OTHER STATE > OR FEDERAL LAW OF SIMILAR SUBSTANCE OR EFFECT. THIS EMAIL > MESSAGE, ITS CONTENTS AND ATTACHMENTS ARE NOT INTENDED TO > REPRESENT AN OFFER OR ACCEPTANCE OF AN OFFER TO ENTER INTO A > CONTRACT. NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF > WHICH IT MAY BE A PART, OR IN ANY ATTACHMENTS THERETO SHALL > ALTER THIS DISCLAIMER. > > This e-mail message from PNY Technologies, Inc. is for the > sole use of the intended recipient(s) and may contain > confidential and privileged information. Any unauthorized > review, use, disclosure or distribution is prohibited. If you > are not the intended recipient, please contact the sender by > reply e-mail and destroy all copies of the original message. > > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > > This e-mail and related attachments and any response may be > subject to public disclosure under state law. > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > U2-Users@listserver.u2ug.org > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
