See the minimum modulus of 1 & the current mod of 240052? your file has split 240051 times, and a lot of your writes are probably going to involve a split operation. Check this again in a couple of days to see what the current mod is to see how many split operations you've done. An interesting thing to see would be the sizes of the DATA.30 & OVER.30 files; with a mod of 240052 & a group size of 2048, the file is sized for 469Mb, yet the actual data size is 860Gb. My first guess would be that you are storing a lot of oversized items (>1600b, the Large Record Size) - nearly 400Mb. IIRC, UV puts a pointer in the primary group to the overflow, so each I/O will be at least 2 disk accesses for an oversized item (this is efficient: if it tried to store the first 1600 bytes here in primary space, you'd have to read all of the oversized items to get to the most recently accessed items at the end of the group). If a large fraction of your oversized items are under, say 3200b, doubling the group size might help significantly, bearing in mind that you don't want to exceed the page size for your OS (typically 4K)(think about it a moment: if data is stored on disk in 4k pages, and you have a group size above that, every read is two physical disk reads, cutting your throughput capacity in half).
To look at this, I use a small program that opens the file, loops through a basic select while readnext, reads each record, does a len(rec), & then increments a counter in an attribute where <1> is a count of records from 0-500b in size, <2> is 501-1000, <3> is 1001-1500, etc. I've often found it useful to also make from attribute 20 (30? 40? your choice) on up multivalued, with the 1st mv being the counter & then a subvalue list of item ids in the 2nd mv. This way, you can see a histogram of your record sizes, and make more informed file sizing/typing decisions. > Subject: RE: [U2] static/dynamic file(s) opinions wanted!> Date: Wed, 23 Jul 2008 07:55:32 -0400> From: [EMAIL PROTECTED]> To: [email protected]> > Wow,> Super detailed explanation. I want to avoid the Katrina effect but> > 1. I am not sure what to check for, I can run ANALYZE.FILE and get the> below results (i his the right way to inspect my "levees" or should I be> using something else? I am just used to using jRF -R jASE report only)> and HASH.HELP and I am not sure what to use for dynamic files to> "inspect" them.> > >ANALYZE.FILE PARTS> File name .................. PARTS> Pathname ................... PARTS> File type .................. DYNAMIC> Hashing Algorithm .......... GENERAL> No. of groups (modulus) .... 240052 current ( minimum 1 )> Large record size .......... 1600 bytes> Group size ................. 2048 bytes> Load factors ............... 80% (split), 50% (merge) and 35% (actual)> Total size ................. 907905024 bytes> > Thanks for the great explanation > > Dougc> > Ps> > On a semi related subject, I sure do miss fry's, there is not one here> in NC .........> > > -----Original Message-----> From: [EMAIL PROTECTED]> [mailto:[EMAIL PROTECTED] On Behalf Of Dan Fitzgerald> Sent: Tuesday, July 22, 2008 7:28 PM> To: [email protected]> Subject: RE: [U2] static/dynamic file(s) opinions wanted!> > And once a dynamic file starts splitting, writes become very> inefficient. I'd> say that the most efficient file is a well-sized static one. You want it> "wide> & shallow". I'm not against having a mod larger than the number of items> anymore, especially if the file is going to grow. If I know that the> file is> going to grow to 1.5Gb, I'll create it that large. With today's storage> technologies, having empty groups isn't all that bad a thing. The old> "empty> groups impact SELECT statements" isn't as true as it once was, and if> you do> so many selects that it's an issue, you might want to look into indices.> I saw> a 1Tb drive at Fry's for $170 the other day, so space isn't an issue> anymore,> either. Dynamic files used to be sold as "you don't have to maintain> them".> You do, though. If your current mod is GT your minimum mod, you should> resize> up with a different min mod. That's maintenance. The reason is that your> file> has reached a point where you're adding data to a file with an> increasing> chance that the group is already near the split %. When that happens,> your> write turns into a beast. It has to go to whatever mechanism your OS has> for> finding free space, attaching that, updating that table, dividing the> data in> RAM between the two groups (old & new), read in the header (& there's> only one> per file, no matter how many users want it), lock it, update it with the> new> structure, write it, unlock it, write out the old group, write out the> new> group. Undersized static files can be as bad, but dynamic files have the> "Katrina effect". Everybody expects that the levees will hold, and FEMA> will> rescue them, and dynamic files don't need maintenance, so they don't> check to> be sure...> > > From: [EMAIL PROTECTED]> To: [email protected]>> Subject: RE:> [U2] static/dynamic file(s) opinions wanted!> Date: Tue, 22 Jul 2008> 18:25:50> -0400> > Thanks Kevin but I am still missing something, if dynamic files> are> well> "dynamic" why would one need to resize them and how would one know> what> to> resize them to? I am sorry if this is basic stuff but I come from a> jBASE>> world and have never really used dynamic files. > > you cannot run> HASH.HELP> on a dynamic file and ANALYZE.FILE does not seem to> return anything> useful in> resizing, so how would one determaine (without> using a rool like the> ones> previously mentioned). I LOVE to use tools like> these BUT I also like> to know> how to do it manually in case I cannot use> these tool(s) (for whatever> reason).> > I guess the gist of my question(s) would be > > 1. how to> tell if> a dynamic file needs to be resized> 2. my other question about which is> better> seems to be that dynamic files> are if you have LOTS of files and they> grow a> lot and you cannot maintain> them> > thanks everyone!> > dougc> > >> -----Original Message-----> From: [EMAIL PROTECTED]>> [mailto:[EMAIL PROTECTED] On Behalf Of Kevin King>> Sent:> Tuesday, July 22, 2008 4:36 PM> To: [email protected]>> Subject: Re:> [U2] static/dynamic file(s) opinions wanted!> > Files that grow at a> controlled rate and especially files that could exceed> 2G are good> candidates> for dynamic files. Files that are cleared, or files> that have masses of> data> loaded or removed from them, are not. Work files,> for example, can be a> horrible use of dynamic files. As was stated earlier,> it's crucial that> the> original block size of the dynamic file be set> properly, otherwise the> file> could split way too often.> > On Tue, Jul 22, 2008 at 1:02 PM, Symeon> Breen> <[EMAIL PROTECTED]> wrote:> > > We use dynamic files no problem - yes I> suppose in certain circumstances> > there is an overhead, but it would> still> be faster than a badly sized> > static> > file. The conclusion we have> is if> you are really on top of your file> sizes> > and administrating things> daily> there is probably less need for dynamic> > files. If however you have> hundreds> of accounts and files then dynamic> > files> > are easier to admin and> hence> probably faster in the long term.> >> >> >> > Symeon.> >> >> >> >> -----Original Message-----> > From: [EMAIL PROTECTED]>> >> [mailto:[EMAIL PROTECTED] On Behalf Of doug chanco> >> Sent:> 22 July 2008 16:36> > To: [email protected]> > Subject: [U2]> static/dynamic file(s) opinions wanted!> >> > hey all,> > I have "heard"> bad> things about using dynamic files versus> > hashed/static ones. Can> anyone> share any thoughts on which is better> > (in particular on a system> where the> files grow at a fairly steady rate).> >> > I always understood that> dynamic> files were best on files that did not> > change "that much that fast "> as the> constant need to resize would> > outweigh the manual effort of resizing> the> files manually (or with a> > program).> >> > I am looking for insight> (or> where to find some insight) on universe and> > best file practices> (right now> I am reading the system description> > manual and its helping but lacks> insights that I am sure some of the old> > pickies on here have)> >> >> so any> thoughts/suggestions/ideas/comments are welcomed!> >> > thanks> >> >> dougc> >>> > ps> >> > universe 10.1 and aix 5.2> > -------> > u2-users mailing> list> >> [email protected]> > To unsubscribe please visit> http://listserver.u2ug.org/> > No virus found in this incoming message.>> >> Checked by AVG - http://www.avg.com> > Version: 8.0.138 / Virus> Database:> 270.5.3/1565 - Release Date: 7/21/2008> > 6:36 PM> > -------> > u2-users> mailing list> > [email protected]> > To unsubscribe please> visit> http://listserver.u2ug.org/> >> > > > -- > -Kevin>> http://www.PrecisOnline.com> -------> u2-users mailing list>> [email protected]> To unsubscribe please visit> http://listserver.u2ug.org/> > > __________ Information from ESET NOD32> Antivirus, version of virus signature> database 3289 (20080722)> __________> >> The message was checked by ESET NOD32 Antivirus.> > http://www.eset.com>> > > >> __________ Information from ESET NOD32 Antivirus, version of virus> signature>> database 3289 (20080722) __________> > The message was checked by ESET> NOD32> Antivirus.> > http://www.eset.com> -------> u2-users mailing list>> [email protected]> To unsubscribe please visit> http://listserver.u2ug.org/> _________________________________________________________________> Use video conversation to talk face-to-face with Windows Live Messenger.> http://www.windowslive.com/messenger/connect_your_way.html?ocid=TXT_TAGL> M_WL_> Refresh_messenger_video_072008> -------> u2-users mailing list> [email protected]> To unsubscribe please visit http://listserver.u2ug.org/> > -- > This message has been scanned for viruses and> dangerous content by SecureMail, and is> believed to be clean.> -------> u2-users mailing list> [email protected]> To unsubscribe please visit http://listserver.u2ug.org/ _________________________________________________________________ Keep your kids safer online with Windows Live Family Safety. http://www.windowslive.com/family_safety/overview.html?ocid=TXT_TAGLM_WL_fami ly_safety_072008 ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
