Brian: I really like these suggestions and will give it a go.
Thanks- Harold -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Brian Leach Sent: Wednesday, July 22, 2009 8:44 AM To: 'U2 Users List' Subject: Re: [U2] Universe just quits Harold You might want to look at splitting out some of those index records, e.g. where you have 10,000 fields in an index for CRIME, split this into CRIME, CRIME-1, CRIME-2 etc. with a maximum number of entries per index. Otherwise you are not going to get efficient storage at those sizes: if they are in a directory file these can be slow scanning and clashing on the lock table, and of course in a hashed file they will be in out of line overflow - again, slowing access. You can also adopt a scheme where, for example, the first entry in the base record holds the last sequence number of the series, so you don't have to read the intermediate ones when appending.. I realize this will mean changing the search routines, but it might help in the long run. I would also definitely echo what the others have said about sequentially accessing the source descriptions. Brian > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Oaks, > Harold > Sent: Tuesday, July 21, 2009 4:54 PM > To: U2 Users List > Subject: Re: [U2] Universe just quits > > Barry: > Thanks for resonding. > > The main activity is running thru text files (narratives), identifying > the next word then updating a file (DEXNAR) used to cross-reference > all the words. For example, in record > ABC123 in the narrative field the next identified word is DECOMPOSING. > I read the record DECOMPOSING from the DEXNAR file. The record > DECOMPOSING has 10 lines in it, because 10 other prior narratives had > the word DECOMPOSING. The program seeks the current record key > (ABC123) in the fields using the LOCATE command. If not found, the key > ABC123 is appended to the end of the DECOMPOSING record and > DECOMPOSING it is written out, now with 11 fields. > > Some of the records in DEXNAR get long as they are for fairly common > words, like ALIAS. (The most common words I do not bother to > cross-reference, of course, like the word THE). > Thus, the LOCATE is looking thru an ever increasing number of fields > as we go along. Some records currently have over 11000 fields. Is > that the problem? Maybe the sytem tries to 'fit' the next very long > record in memory to do the LOCATE and it overflows something? > > The dropping out, however, occurs normally after updating > 25000+ records. Shorter runs seem to 'hold out'. What might > be building up in memory space? These are also overnight runs, > typically, so there are fewer users to contend with making it unlikely > that it's exceeded limits for all users, I would think. > > The point of the cross-reference file is to allow users to quickly > search thru all the 700,000 available (crime) narratives for the ones > that match a set of entered search words. Searching that many > narratives using brute force would probably take 5+ minutes for each > search on our system, which would really stifle the crime analysts. > > Thanks- > Harold > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Barry Rogen > Sent: Tuesday, July 21, 2009 1:03 PM > To: U2 Users List > Subject: Re: [U2] Universe just quits > > > Can you elaborate a little on these tasks > > Barry Rogen > PNY Technologies, Inc. > Senior Programmer/Analyst > (973) 515 - 9700 ext 5327 > [email protected] > > ----------------------------------------------------- > We are continually faced with great opportunities brilliantly > disguised as insoluble problems. > > John W Gardner > ---------------------------------------------------------------- > P Before printing please think about your environmental responsibility > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Oaks, > Harold > Sent: Tuesday, July 21, 2009 2:41 PM > To: [email protected] > Subject: [U2] Universe just quits > > I am having a very disconcerting problem. A long job I have been > running, processing a large text file and then loading a Universe > file, is simply quitting sometimes. No error message of any kind, > Universe just quits, the session drops into unix. We have Universe > 10.2 running over HPux 11.1. I have reduced the job to smaller pieces > to get thru it and indicate data values so that I restart at about the > dropout point. > So I'll be able to finish but I would like to understand what's going > on. > > Anybody seen this kind of thing before? Is there a Universe parameter > I should look at? A unix kernal parameter? > > Any ideas appreciated. > Thanks- > > Harold Oaks > Sr. Analyst/Programmer > Clark County Information Systems > Clark County, Washington > ph: (360) 397-6121 x4132 > > > > This e-mail and related attachments and any response may be subject to > public disclosure under state law. > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > 21/7/2009NOT INTENDED AS A SUBSTITUTE FOR A WRITING > > NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT MAY BE A > PART, OR IN ANY ATTACHMENTS THERETO, SHALL CONSTITUTE A BINDING > CONTRACT, OR ANY CONTRACTUAL OBLIGATION BY PNY, OR ANY INTENT TO ENTER > INTO ANY BINDING OBLIGATIONS, NOTWITHSTANDING ANY ENACTMENT OF THE > UNIFORM ELECTRONIC TRANSACTIONS ACT, THE FEDERAL E-SIGN ACT, OR ANY > OTHER STATE OR FEDERAL LAW OF SIMILAR SUBSTANCE OR EFFECT. THIS EMAIL > MESSAGE, ITS CONTENTS AND ATTACHMENTS ARE NOT INTENDED TO REPRESENT AN > OFFER OR ACCEPTANCE OF AN OFFER TO ENTER INTO A CONTRACT. NOTHING IN > THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT MAY BE A PART, OR IN ANY > ATTACHMENTS THERETO SHALL ALTER THIS DISCLAIMER. > > This e-mail message from PNY Technologies, Inc. is for the sole use of > the intended recipient(s) and may contain confidential and privileged > information. Any unauthorized review, use, disclosure or distribution > is prohibited. If you are not the intended recipient, please contact > the sender by reply e-mail and destroy all copies of the original > message. > > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > > This e-mail and related attachments and any response may be subject to > public disclosure under state law. > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > > _______________________________________________ > U2-Users mailing list > [email protected] > http://listserver.u2ug.org/mailman/listinfo/u2-users > _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users This e-mail and related attachments and any response may be subject to public disclosure under state law. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
