Ross: Interesting idea! I will try it both ways and see which is faster.
Thanks so much- Harold -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ross Ferris Sent: Wednesday, July 22, 2009 6:42 PM To: U2 Users List Subject: Re: [U2] Universe just quits "Real" Indexing should win - compound key based on wo...@id, index on word and then traverse keys in Basic (or via a select) ... saves having to juggle your own key blocks for larger intersects/more popular words Ross Ferris Stamina Software Visage > Better by Design! >-----Original Message----- >From: [email protected] [mailto:u2-users- >[email protected]] On Behalf Of Brian Leach >Sent: Thursday, 23 July 2009 1:44 AM >To: 'U2 Users List' >Subject: Re: [U2] Universe just quits > >Harold > >You might want to look at splitting out some of those index records, >e.g. >where you have 10,000 fields in an index for CRIME, split this into >CRIME, CRIME-1, CRIME-2 etc. with a maximum number of entries per >index. >Otherwise >you are not going to get efficient storage at those sizes: if they are >in a directory file these can be slow scanning and clashing on the lock >table, and of course in a hashed file they will be in out of line >overflow - again, slowing access. You can also adopt a scheme where, >for example, the first entry in the base record holds the last sequence >number of the series, so you don't have to read the intermediate ones >when appending.. > >I realize this will mean changing the search routines, but it might help >in >the long run. > >I would also definitely echo what the others have said about >sequentially accessing the source descriptions. > >Brian > >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Oaks, >> Harold >> Sent: Tuesday, July 21, 2009 4:54 PM >> To: U2 Users List >> Subject: Re: [U2] Universe just quits >> >> Barry: >> Thanks for resonding. >> >> The main activity is running thru text files (narratives), >> identifying the next word then updating a file (DEXNAR) used to >> cross-reference all the words. For example, in record >> ABC123 in the narrative field the next identified word is >> DECOMPOSING. I read the record DECOMPOSING from the DEXNAR file. >> The record DECOMPOSING has 10 lines in it, because 10 other prior >> narratives had the word DECOMPOSING. The program seeks the current >> record key (ABC123) in the fields using the LOCATE command. If not >> found, the key ABC123 is appended to the end of the DECOMPOSING >> record and DECOMPOSING it is written out, now with 11 fields. >> >> Some of the records in DEXNAR get long as they are for fairly common >> words, like ALIAS. (The most common words I do not bother to >> cross-reference, of course, like the word THE). >> Thus, the LOCATE is looking thru an ever increasing number of fields >> as we go along. Some records currently have over 11000 fields. Is >> that the problem? Maybe the sytem tries to 'fit' the next very long >> record in memory to do the LOCATE and it overflows something? >> >> The dropping out, however, occurs normally after updating >> 25000+ records. Shorter runs seem to 'hold out'. What might >> be building up in memory space? These are also overnight runs, >> typically, so there are fewer users to contend with making it >> unlikely that it's exceeded limits for all users, I would think. >> >> The point of the cross-reference file is to allow users to quickly >> search thru all the 700,000 available (crime) narratives for the ones >> that match a set of entered search words. Searching that many >> narratives using brute force would probably take 5+ minutes for each >> search on our system, which would really stifle the crime analysts. >> >> Thanks- >> Harold >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Barry Rogen >> Sent: Tuesday, July 21, 2009 1:03 PM >> To: U2 Users List >> Subject: Re: [U2] Universe just quits >> >> >> Can you elaborate a little on these tasks >> >> Barry Rogen >> PNY Technologies, Inc. >> Senior Programmer/Analyst >> (973) 515 - 9700 ext 5327 >> [email protected] >> >> ----------------------------------------------------- >> We are continually faced with great opportunities brilliantly >> disguised as insoluble problems. >> >> John W Gardner >> ---------------------------------------------------------------- >> P Before printing please think about your environmental responsibility >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Oaks, >> Harold >> Sent: Tuesday, July 21, 2009 2:41 PM >> To: [email protected] >> Subject: [U2] Universe just quits >> >> I am having a very disconcerting problem. A long job I have been >> running, processing a large text file and then loading a Universe >> file, is simply quitting sometimes. No error message of any kind, >> Universe just quits, the session drops into unix. We have Universe >> 10.2 running over HPux 11.1. I have reduced the job to smaller >> pieces to get thru it and indicate data values so that I restart at >> about the dropout point. >> So I'll be able to finish but I would like to understand what's going >> on. >> >> Anybody seen this kind of thing before? Is there a Universe >> parameter I should look at? A unix kernal parameter? >> >> Any ideas appreciated. >> Thanks- >> >> Harold Oaks >> Sr. Analyst/Programmer >> Clark County Information Systems >> Clark County, Washington >> ph: (360) 397-6121 x4132 >> >> >> >> This e-mail and related attachments and any response may be subject >> to public disclosure under state law. >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> 21/7/2009NOT INTENDED AS A SUBSTITUTE FOR A WRITING >> >> NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT MAY BE A >> PART, OR IN ANY ATTACHMENTS THERETO, SHALL CONSTITUTE A BINDING >> CONTRACT, OR ANY CONTRACTUAL OBLIGATION BY PNY, OR ANY INTENT TO >> ENTER INTO ANY BINDING OBLIGATIONS, NOTWITHSTANDING ANY ENACTMENT OF >> THE UNIFORM ELECTRONIC TRANSACTIONS ACT, THE FEDERAL E-SIGN ACT, OR >> ANY OTHER STATE OR FEDERAL LAW OF SIMILAR SUBSTANCE OR EFFECT. THIS >> EMAIL MESSAGE, ITS CONTENTS AND ATTACHMENTS ARE NOT INTENDED TO >> REPRESENT AN OFFER OR ACCEPTANCE OF AN OFFER TO ENTER INTO A >> CONTRACT. NOTHING IN THIS E-MAIL, IN ANY E-MAIL THREAD OF WHICH IT >> MAY BE A PART, OR IN ANY ATTACHMENTS THERETO SHALL ALTER THIS >> DISCLAIMER. >> >> This e-mail message from PNY Technologies, Inc. is for the sole use >> of the intended recipient(s) and may contain confidential and >> privileged information. Any unauthorized review, use, disclosure or >> distribution is prohibited. If you are not the intended recipient, >> please contact the sender by reply e-mail and destroy all copies of >> the original message. >> >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> >> This e-mail and related attachments and any response may be subject >> to public disclosure under state law. >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> >> _______________________________________________ >> U2-Users mailing list >> [email protected] >> http://listserver.u2ug.org/mailman/listinfo/u2-users >> > >_______________________________________________ >U2-Users mailing list >[email protected] >http://listserver.u2ug.org/mailman/listinfo/u2-users _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users This e-mail and related attachments and any response may be subject to public disclosure under state law. _______________________________________________ U2-Users mailing list [email protected] http://listserver.u2ug.org/mailman/listinfo/u2-users
