"Real" Indexing should win - compound key based on wo...@id, index on word and then traverse keys in Basic (or via a select) ... saves having to juggle your own key blocks for larger intersects/more popular words
Ross Ferris Stamina Software Visage > Better by Design! >-----Original Message----- >From: [email protected] [mailto:u2-users- >[email protected]] On Behalf Of Brian Leach >Sent: Thursday, 23 July 2009 1:44 AM >To: 'U2 Users List' >Subject: Re: [U2] Universe just quits > >Harold > >You might want to look at splitting out some of those index records, >e.g. >where you have 10,000 fields in an index for CRIME, split this into >CRIME, >CRIME-1, CRIME-2 etc. with a maximum number of entries per index. >Otherwise >you are not going to get efficient storage at those sizes: if they are >in a >directory file these can be slow scanning and clashing on the lock >table, >and of course in a hashed file they will be in out of line overflow - >again, >slowing access. You can also adopt a scheme where, for example, the >first >entry in the base record holds the last sequence number of the series, >so >you don't have to read the intermediate ones when appending.. > >I realize this will mean changing the search routines, but it might help >in >the long run. > >I would also definitely echo what the others have said about >sequentially >accessing the source descriptions. > >Brian > >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of >> Oaks, Harold >> Sent: Tuesday, July 21, 2009 4:54 PM >> To: U2 Users List >> Subject: Re: [U2] Universe just quits >> >> Barry: >> Thanks for resonding. >> >> The main activity is running thru text files (narratives), >> identifying the next word then updating a file (DEXNAR) used >> to cross-reference all the words. For example, in record >> ABC123 in the narrative field the next identified word is >> DECOMPOSING. I read the record DECOMPOSING from the DEXNAR >> file. The record DECOMPOSING has 10 lines in it, because 10 >> other prior narratives had the word DECOMPOSING. The program >> seeks the current record key (ABC123) in the fields using the >> LOCATE command. If not found, the key ABC123 is appended to >> the end of the DECOMPOSING record and DECOMPOSING it is >> written out, now with 11 fields. >> >> Some of the records in DEXNAR get long as they are for fairly >> common words, like ALIAS. (The most common words I do not >> bother to cross-reference, of course, like the word THE). >> Thus, the LOCATE is looking thru an ever increasing number of >> fields as we go along. Some records currently have over >> 11000 fields. Is that the problem? Maybe the sytem tries to >> 'fit' the next very long record in memory to do the LOCATE >> and it overflows something? >> >> The dropping out, however, occurs normally after updating >> 25000+ records. Shorter runs seem to 'hold out'. What might >> be building up in memory space? These are also overnight >> runs, typically, so there are fewer users to contend with >> making it unlikely that it's exceeded limits for all users, I >> would think. >> >> The point of the cross-reference file is to allow users to >> quickly search thru all the 700,000 available (crime) >> narratives for the ones that match a set of entered search >> words. Searching that many narratives using brute force >> would probably take 5+ minutes for each search on our system, >> which would really stifle the crime analysts. >> >> Thanks- >> Harold >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Barry Rogen >> Sent: Tuesday, July 21, 2009 1:03 PM >> To: U2 Users List >> Subject: Re: [U2] Universe just quits >> >> >> Can you elaborate a little on these tasks >> >> Barry Rogen >> PNY Technologies, Inc. >> Senior Programmer/Analyst >> (973) 515 - 9700 ext 5327 >> [email protected] >> >> ----------------------------------------------------- >> We are continually faced with great opportunities >> brilliantly disguised as insoluble problems. >> >> John W Gardner >> ---------------------------------------------------------------- >> P Before printing please think about your environmental responsibility >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of >> Oaks, Harold >> Sent: Tuesday, July 21, 2009 2:41 PM >> To: [email protected] >> Subject: [U2] Universe just quits >> >> I am having a very disconcerting problem. A long job I have >> been running, processing a large text file and then loading a >> Universe file, is simply quitting sometimes. No error >> message of any kind, Universe just quits, the session drops >> into unix. We have Universe 10.2 running over HPux 11.1. I >> have reduced the job to smaller pieces to get thru it and >> indicate data values so that I restart at about the dropout point. >> So I'll be able to finish but I would like to understand >> what's going on. >> >> Anybody seen this kind of thing before? Is there a Universe >> parameter I should look at? A unix kernal parameter? >> >> Any ideas appreciated. >> Thanks- >> >> Harold Oaks >> Sr. Analyst/Programmer >> Clark County Information Systems >> Clark County, Washington >> ph: (360) 397-6121 x4132 >> >> >> >> This e-mail and related attachments and any response may be >> subject to public disclosure under state law. >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> 21/7/2009NOT INTENDED AS A SUBSTITUTE FOR A WRITING >> >> NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT MAY >> BE A PART, OR IN ANY ATTACHMENTS THERETO, SHALL CONSTITUTE A >> BINDING CONTRACT, OR ANY CONTRACTUAL OBLIGATION BY PNY, OR >> ANY INTENT TO ENTER INTO ANY BINDING OBLIGATIONS, >> NOTWITHSTANDING ANY ENACTMENT OF THE UNIFORM ELECTRONIC >> TRANSACTIONS ACT, THE FEDERAL E-SIGN ACT, OR ANY OTHER STATE >> OR FEDERAL LAW OF SIMILAR SUBSTANCE OR EFFECT. THIS EMAIL >> MESSAGE, ITS CONTENTS AND ATTACHMENTS ARE NOT INTENDED TO >> REPRESENT AN OFFER OR ACCEPTANCE OF AN OFFER TO ENTER INTO A >> CONTRACT. NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF >> WHICH IT MAY BE A PART, OR IN ANY ATTACHMENTS THERETO SHALL >> ALTER THIS DISCLAIMER. >> >> This e-mail message from PNY Technologies, Inc. is for the >> sole use of the intended recipient(s) and may contain >> confidential and privileged information. Any unauthorized >> review, use, disclosure or distribution is prohibited. If you >> are not the intended recipient, please contact the sender by >> reply e-mail and destroy all copies of the original message. >> >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> >> This e-mail and related attachments and any response may be >> subject to public disclosure under state law. >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> > >_______________________________________________ >U2-Users mailing list >[email protected] >http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
