On 03/06/11 15:03, George Gallen wrote: > Well, this process is still running....and it's up to just about 15M (1.3g) > so hopefully, it will be done very soon - I estimated this to take > about an hour or so, well, thats turning into almost 20 hours. > > I don't know if I have the 64bit option available, I could but something > tells me I don't - I'm on version 10.0.2 of UV > > But if it crashes the process because the file gets too big, I'll have to > pare down what gets written out - just less history, this is a file > that once it's created, it's exported and rarely used, so it's getting the > slow > part out of the way once, after that, it runs fast for updating.
Hmm... If you're creating a dynamic file and then dumping into it, think about using MINIMUM.MODULUS. Or using a static file. The resizing could be what's crippling your performance. I know resizing a dynamic file is fast, but if you're firing loads of small records at the file, it's still going to be relatively expensive. Assuming each record (including key) is, say, 100 bytes and you've got the standard 2K bucket, every 16 records you add, you're going to have to rehash 16 more to split a bucket. If you pick a sufficiently big MINIMUM.MODULUS, so that all the records should just about fit in the file, you've gone and saved all that calculation. It looks like choosing a value round about a million would be a good idea. Maybe a bit less. And if you don't have enough ram to cache that file in memory, I guess all that io could be killing you. You don't need to add the cost of resizing as well ... Cheers, Wol _______________________________________________ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
