RE: [U2] Internal Cache in Unidata
What OS? How much RAM does your server have? Etc etc. What's the likelihood that the data was on disk when you first ran the query, and was pulled into the OS cache as a result? Second time round, the OS didn't bother to access the disk because it was all in cache ... Especially if your query is only accessing one file, I'd write a little macro in ED to just pull in every record in the file, and then see if your query flies the first time. If it does fly (and if you're on Unix that's especially likely) then it's down to OS caching, not UniData. Certainly when I tried writing the file access code for MaVerick, I deliberately ignored caching the files on the basis that linux would do a better job than I could ... Cheers, Wol -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeffrey Butera Sent: 24 September 2007 13:34 To: u2-users@listserver.u2ug.org Subject: [U2] Internal Cache in Unidata Can anyone speak factually about internal cache in Unidata? In short... I often look at performance and attempt to improve things by changing queries, using indicies, rewriting source and so on. In doing so I perform timing and consistently note that the second time a query is performed, it executes considerably faster than the first. As an example, here's the first execution of a series of queries: SELECT H08.CR.DIV.DATA WITH XCDD.HAMP.CMTE.MEMBER=0202736 time=18ms SELECT H08.CR.DIV.DATA WITH XCDD.DIV EQ D2 time=988ms SELECT H08.CR.DIV.DATA WITH XCDD.REV EQ P time=5ms And then immediately peforming these again: SELECT H08.CR.DIV.DATA WITH XCDD.HAMP.CMTE.MEMBER=0202736 time=18ms SELECT H08.CR.DIV.DATA WITH XCDD.DIV EQ D2 time=9ms SELECT H08.CR.DIV.DATA WITH XCDD.REV EQ P time=5ms To be clear - the difference between 988ms and 9ms is not a typo, it's real. Likewise this is not the result of server load. I can consistently reproduce the above scenario with different queries at different times of the day (and wee hours of the morning with few users). I'm not complaining - it's a perfectly good feature - it's just hard to get accurate timings on things so I often perform queries numerous times and dump the first few executions. Of course, this isn't reality as many queries only get executed once in a blue moon... -- Jeff Butera, Ph.D. Administrative Systems Hampshire College [EMAIL PROTECTED] 413-559-5556 But Mom - Daddy knows everything. Catherine Butera --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Internal Cache in Unidata
Can anyone speak factually about internal cache in Unidata? I would be looking more at the operating system than UniData when it comes to results like this. With the amount of memory available on today's systems, it's likely that major portions (if not all) of the file(s) are remaining in O/S cache. So for second and subsequent queries you're pulling from memory instead of disk. If you really want to get a fair estimation of one query method versus another, you really should reboot between each invocation. OK - that's generally not practical, so you probably want to perform each query twice, ignoring the first one each time. That won't give you an indication of how things will perform in real life, but it will give you some sort of idea of how one method compares to another. It's interesting to note that your second of three queries changes drastically during the second invocation while the others remain the same. Without knowing anything about your files or the dictionary items involved, I'm going to guess that XCDD.DIV involves one or more file translates. The records in the file(s) being referenced are being cached during the first invocation and available in memory the second time around. Tim Snyder Consulting I/T Specialist U2 Lab Services Information Management, IBM Software Group --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
RE: [U2] Internal Cache in Unidata
This is a normal feature of the O/S not Unidata. Both Unix and Windows attempt to load files into memory based on last use. Making it VERY hard to test file access methods..! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Jeffrey Butera Sent: Monday, September 24, 2007 05:34 To: u2-users@listserver.u2ug.org Subject: [U2] Internal Cache in Unidata Can anyone speak factually about internal cache in Unidata? In short... I often look at performance and attempt to improve things by changing queries, using indicies, rewriting source and so on. In doing so I perform timing and consistently note that the second time a query is performed, it executes considerably faster than the first. As an example, here's the first execution of a series of queries: SELECT H08.CR.DIV.DATA WITH XCDD.HAMP.CMTE.MEMBER=0202736 time=18ms SELECT H08.CR.DIV.DATA WITH XCDD.DIV EQ D2 time=988ms SELECT H08.CR.DIV.DATA WITH XCDD.REV EQ P time=5ms And then immediately peforming these again: SELECT H08.CR.DIV.DATA WITH XCDD.HAMP.CMTE.MEMBER=0202736 time=18ms SELECT H08.CR.DIV.DATA WITH XCDD.DIV EQ D2 time=9ms SELECT H08.CR.DIV.DATA WITH XCDD.REV EQ P time=5ms To be clear - the difference between 988ms and 9ms is not a typo, it's real. Likewise this is not the result of server load. I can consistently reproduce the above scenario with different queries at different times of the day (and wee hours of the morning with few users). I'm not complaining - it's a perfectly good feature - it's just hard to get accurate timings on things so I often perform queries numerous times and dump the first few executions. Of course, this isn't reality as many queries only get executed once in a blue moon... -- Jeff Butera, Ph.D. Administrative Systems Hampshire College [EMAIL PROTECTED] 413-559-5556 But Mom - Daddy knows everything. Catherine Butera --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/ --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Internal Cache in Unidata
quote who='Anthony Youngman' date='Monday 24 September 2007' What OS? How much RAM does your server have? Etc etc. What's the likelihood that the data was on disk when you first ran the query, and was pulled into the OS cache as a result? Second time round, the OS didn't bother to access the disk because it was all in cache ... Good point. This is UD 7.1.8 on Solaris, so your thoughts are on target. -- Jeff Butera, Ph.D. Administrative Systems Hampshire College [EMAIL PROTECTED] 413-559-5556 But Mom - Daddy knows everything. Catherine Butera --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Internal Cache in Unidata
At 07:33 AM 9/24/2007, you wrote: Can anyone speak factually about internal cache in Unidata? In short... A cache buffer is a perfectly normal operation of the OS, not the database. This is why you are seeing the results that you are seeing. It is often tunable in the OS on how big the buffer is as well. The main tunable items within the database have to do with the size of the rotating file pool, size of shared memory for processes and database file sizing. Doug Miller [EMAIL PROTECTED] Manager of Technical Services Strategy 7Dallas TX --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
Re: [U2] Internal Cache in Unidata
It's interesting to note that your second of three queries changes drastically during the second invocation while the others remain the same. Without knowing anything about your files or the dictionary items involved, I'm going to guess that XCDD.DIV involves one or more file translates. Nope. For all three of those queries the fields are Data, single valued and indexed. That's why I'm looking at this... -- Jeff Butera, Ph.D. Administrative Systems Hampshire College [EMAIL PROTECTED] 413-559-5556 But Mom - Daddy knows everything. Catherine Butera --- u2-users mailing list u2-users@listserver.u2ug.org To unsubscribe please visit http://listserver.u2ug.org/
