Just a thought on this Jeff: Is there any logical way to break-down the file so that it does not have to have so many records in it? Either a time-period based breakdown or some other logical distinguishable break-point? In the end, maybe you might need to restructure that application a bit to accommodate multiple files, but I would think overall performance & overhead wise, things would be much easier to control as content & number of records grow.
Regards, Scott Richardson ----- Original Message ----- From: "Jeff Butera" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Thursday, July 05, 2007 9:44 AM Subject: [U2] Unidata Dynamic files > hpux, unidata 7.1.8 > %<----- snip > Of course, I'm dealing with a headache of a dynamic file that just isn't > happy. In short, it's got about 33,000,000 records, each in under 512 in > size with purely integer keys. I can create a dynamic file with hash type > 0, modulo around 3,000,000 and block size 1024 which looks good at the > start: it's got 3 dat segments and 1 small ovr segment. > > When I copy records from the original to the new dynamic file and get > somewhere around 20,000,000 records, it start creating ovr segments lie > crazy. I end up with 4 dat segments and 5 ovr segments. > > When I run guide on resulting file, it's taking up about 9Gig on disk, but > only contains about 1.1Gig of data. I've tried split/merge loads of 20/10 > and 60/40 with similar results. > > Hence, I'm trying to determine why it's splitting like mad even thought > it's mostly empty. With the purely integer (sequential) keys, it's got a > nice even distribution in keys. I can avoid the overflow by creating > with a rediculously large modulo (eg: 10,000,000) but again, I get a file > that's 80% empty in the end. > > Anyadvice appreciated. > > Jeff Butera, Ph.D. > Administrative Systems > Hampshire College > [EMAIL PROTECTED] > 413-559-5556 > > "Daddy - did you lose your mind?" > Catherine Butera > ------- > u2-users mailing list > [email protected] > To unsubscribe please visit http://listserver.u2ug.org/ ------- u2-users mailing list [email protected] To unsubscribe please visit http://listserver.u2ug.org/
