The db4o branch is rapidly nearing completion, with most of the serious bugs and performance issues worked out. Instant download resuming, and support for a nearly unlimited volume of downloads while using very little RAM, are the main features.
However, there are memory usage spikes at certain points during inserts and requests. In particular, if you attempt a 4GB ISO insert on a node with a 128MB memory limit, it will go into 100% CPU for some time (Full GC's), due to running out of memory and having to do lots of garbage collection, and probably eventually fail with an OutOfMemoryException. So we will probably want to keep the default memory usage limit of 192MB. However, with the exception of *really* big inserts (well over 4GB), 192MB should be enough for almost all users with db4o, so there is no reason to ask the end-user about it any more IMHO. In fact, it will allow for multiple 4GB inserts, since any one of them will only cause a spike once. 25GB single file inserts would obviously produce a larger spike - somewhere in the region of 1GB of RAM would be needed. It would be possible to remove this limit completely, but it would require more work and complicate the API, and reduce performance starting big inserts slightly on nodes with lots of RAM; it wouldn't necessarily take more than a day, but since IMHO it could be done post-merge without too back compatibility issues, it can be safely postponed. Note that with the exception of the memory spikes while starting an insert (or a request), db4o will need very little memory. Almost everything is kept in the database and is only cached in RAM, with the exception of the request bloom filters, which are approximately 3.7MB per 10GB of queued downloads. One problem remaining is that inserts are quite resource intensive (lots of database access / disk I/O) on the db4o branch, more so than on the current code. This largely results from the current inefficient architecture of splitfile inserts; I could fix this before merging, and it would avoid complications with back compatibility later, but it would be somewhat significant work, and the back compatibility issues for doing it later will be surmountable, so I probably won't deal with inserts before merging db4o. Obviously the db4o branch does more disk i/o than the current code. IMHO this is an acceptable tradeoff, especially for downloads. Remaining work: - Finish the unused structure deletion code. - Consider temp file garbage collection. - Recently-succeeded preference in request selection. - Consider removing use of obsolete structures. - More debugging.
pgpC2Gb7LB3ch.pgp
Description: PGP signature
_______________________________________________ Devl mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
