Hi. There is an interesting issue involving very large datastores on machines with less than a hell of a lot of RAM. Basically, it is that we keep a structure (NativeBuffer) for each file in the datastore in RAM. This is necessary for two reasons: a) To keep the LRU list up to date. b) To optimize the common case datastore lookup, which is where we are asked for a file and we probably don't have it. This would result in a syscall (and maybe even some disk seeking) if we don't keep the above datastructure.
Now, we can solve the above problems and greatly reduce the amount of memory used by: * Implementing a hack, storeSize=0, which does not attempt to limit store space usage, and therefore does not need the LRU list. Suggested by Pascal. * Using a bit-vector hash to optimize lookups. A largish datastore might have 200,000 files; with a scarcity factor of 50, this is only ~10,000,000 bits, i.e. just over a meg. This would be a config parameter - the power of two to use for the number of bits. I don't consider this to be a priority at the moment... however, it is a long term problem, and a potential sponsor has emerged (Pascal). I suggest that it will take up to approximately three days including maintenance and debugging (wild guess :))... I am still working for freenet project, and there's a good chance it will be able to renew my contract, so it seems reasonable to do it through the project. I suggest that we should charge a commission of say 50% to cover unexpected events, and this item not being on the current perceived critical path... so $41*3*1.5 ~= $180 donation (3 days may be pessimistic... but experience has taught me to overestimate). The sponsor has agreed to this. Comments? I'm going to start working on this on Saturday or Monday (depending on various factors including whether I decide to work on Saturday [I'm somewhat behind on hours due to recent illness, I regard myself as a contractor and I need the flexibility to take time off and delay the next payment when necessary]) if nobody objects. -- Matthew Toseland toad at amphibian.dyndns.org amphibian at users.sourceforge.net Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/1/03 http://freenetproject.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20021227/ad31676a/attachment.pgp>
