You could always build an index by date. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Mark Johnson Sent: Thursday, October 06, 2005 7:36 AM To: [email protected] Subject: Re: [U2] Fw: More U2 programming hints
I'll add another observation. One file on a client's system (D3, proper mod) contains over 7 million records. It's 1.25 GB. The client wanted to know what the oldest record was. I don't dare "SORT BY DATE" so I wrote a small basic program to BASIC SELECT and keep the lowest number date field. Out of pure habit I used BASIC SELECT as I know that at TCL a COUNT of this file takes around 20 minutes. So I can imagine a EXECUTE SELECT would have to wait at least 20 minutes before evaluating the first record. I don't know everything about select buffers, but something tells me that the 20 minutes (from COUNT) would be much longer as it has to prepare the active list of 7 million elements. As before, I can't imagine the EXECUTE SELECT making up for the lost time going through the file from beginning to end *and* preparing an isolated entity (active list) to READNEXT from. Please no flames on my exposed word called 'habit'. We all have habits, both good and bad and only when proven that my habits are bad can I consider changing them. Actual proof, not guesses and/or first-training. Thanks Mark Johnson ----- Original Message ----- From: "Allen E. Elwood" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Tuesday, October 04, 2005 5:45 PM Subject: RE: [U2] Fw: More U2 programming hints > This is the way it was explained to me way back in '88. The internal select > is slower on the whole file, but immediate in response. It works the same > as LIST. If I list a file with 2,000,000 records I get immediate response. > > If I want to process an entire file, then external select is slower on > response, i.e. I have to wait for 2 million records to be selected before > processing begins, but is quicker in processing all records. > > The internal is slower due to the system having to stop what it's doing, > find the next group, break out the individual ID's from that group, and then > return it to the program - over and over again as it makes it's way through > the file. > > hth! > > Allen > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf Of Stevenson, > Charles > Sent: Tuesday, October 04, 2005 14:24 > To: [email protected] > Cc: Louis Windsor > Subject: RE: [U2] Fw: More U2 programming hints > > > This is a bit disconcerting. > BASIC SELECT should be faster than EXECUTE "SELECT..." > Maybe the smart people can weigh in on this: > > > From: Louis Windsor > > > > A few years ago we used the BASIC SELECT FILE as opposed to > > the EXECUTE "SELECT FILE". > > > > We updated UniVerse (don't ask from what version to what > > version as I don't remember) and overnight ALL our programs > > ran five or six times longer. > > Completely contrary to my experience and counter-intuitive, too. > > > We were told (by VMark) that the BASIC SELECT now selected > > each group but it could be optioned to work the "old" way. > > Hmmm, do I vaguely, hazily remember something about that? Maybe on this > list? Maybe in release notes? No uvconfig option jumps out at me. > I don't think flavor would matter, or $OPTIONS [-]VAR.SELECT. > $OPTIONS FSELECT would slow the BASIC SELECT down to approximately the > same as EXECUTE "SELECT...", but not make it slower. > Louis, do you, perchance, use $OPTIONS FSELECT? Maybe buried in a > $include file common to every program? > > > I wrote a conversion program to change ALL BASIC SELECTs to > > executed SELECTs in the source and recompiled and that is the > > way we have done it ever since. > > > > I don't know if things are different now but we have grown to > > prefer EXECUTEd selects as selection criteria can be included. > > Louis, can you run a simple benchmark and see if this is still true? > Or show us an example of your own? > > INTERNAL: > OPEN "[really big file]" TO F ELSE STOP > CRT 'I1', TIMEDATE(), SYSTEM(9) > SELECT F > CRT 'I2', TIMEDATE(), SYSTEM(9) > LOOP WHILE READNEXT ID > READ REC FROM F, ID ELSE NULL > REPEAT > CRT 'I3', TIMEDATE(), SYSTEM(9) > > EXECUTED: > OPEN "[really big file]" TO F ELSE STOP > CRT 'E1', TIMEDATE(), SYSTEM(9) > EXECUTE "SELECT [really big file]" > CRT 'E2', TIMEDATE(), SYSTEM(9) > LOOP WHILE READNEXT ID > READ REC FROM F, ID ELSE NULL > REPEAT > CRT 'E3', TIMEDATE(), SYSTEM(9) > > (Run each a couple times, to allow for i/o differences in loading data > buffer cache.) > > There should be virtually no elapsed time between I1:I2 above, but long > elapsed time between E1:E2. > I expect I2:I3 to approximately equal E2:E3. > > > Let me explain why this is counter-intuitive. > > Normally, the BASIC SELECT statement itself does not actually do any > select on the file. It merely sets things up behind the scenes so that > subsequent READNEXTs each get the next id from the file opened to > F.FILE, ("next" meaning as stored on disk). > UV keeps track of where it is in the file, unbeknownst to you. Sorta > like it keeps track of where it is for REMOVE or attribute-level > EXTRACTs. > > > > Exceptions to internal being faster than executed: > > 1.SSELECT FILEVAR (i.e., 2 S's, SortSelect). > You gotta read the whole file First to sort the keys. > (and it's an alpha-type sort, even for numeric keys.) > > 2. $OPTIONS FESLECT > Makes SELECT FILEVAR populate @SELECTED and to do so means traversing > the file. > > 3. Louis Windsor. Poor bloke, they're out to get him. > > > Chuck Stevenson > ------- > u2-users mailing list > [email protected] > To unsubscribe please visit http://listserver.u2ug.org/ > ------- > u2-users mailing list > [email protected] > To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
