On Wed, May 6, 2009 at 1:09 PM, Matthew Toseland <toad at amphibian.dyndns.org> wrote: > On Wednesday 06 May 2009 17:34:41 Juiceman wrote: >> On Wed, May 6, 2009 at 11:29 AM, Matthew Toseland >> <toad at amphibian.dyndns.org> wrote: >> > IMHO the number of disk reads is simply a function of how much memory > there >> > is, how much can be used to cache the node.db4o by the operating system. > We >> > can do more caching at the JVM level, this will reduce CPU usage but not >> > actual IOs hitting the disk, and will cost more heap memory. >> > >> > But we can reduce the number of disk writes further for downloads: >> > >> > Turn off scheduling by retry count. Schedule solely by priority then >> > round-robin over the different request clients and requests then randomly >> > over the individual SendableGet's: as we do now, but without the retry > count >> > layer. >> > >> > Only update the retry count for an individual block if MaxRetries != -1. >> > >> > Keep the cooldown queue entirely in RAM. Two structures so that we can > select >> > requests quickly: >> > >> > 1) A structure tracking all the keys fetched in the last half hour. For > each >> > key, the 3 times at which it was last fetched. When selecting requests, we >> > would not send any request for a key in this list - at least not for >> > persistent requests. >> > >> > 2) A structure tracking all the BaseSendableGet's sent in the last half > hour. >> > For each, record the start and finish times for the last 3 times it was >> > fetched. When selecting requests, we can shortcut having to ask the first >> > structure for each block by excluding a whole BaseSendableGet through this >> > structure. >> > >> > Memory cost? Well, we do actually keep the first structure already, in the >> > FailureTable ... some changes would be needed perhaps. The second > structure >> > would be new, and somewhat problematic, as it would keep these objects in >> > memory even if they are deactivated. We could introduce a unique > identifier >> > or even use db4o's object IDs to avoid this (probably the best solution). > A >> > simple hashset of the objects themselves probably would not work well for >> > activation reasons. >> > >> > Network performance cost? Well, what would be the impact of not scheduling > by >> > retry count any more? I dunno, any opinions? >> > >> > CPU cost? We might spend more time trying to find requests to run ... this >> > might even result in more disk reads ... ? But on the whole it ought to be >> > positive ... >> > >> > Code changes? Well, we could keep SplitFileFetcherSubSegment, and be > backward >> > compatible, it'd be a bit awkward though... eventually we'd want a new > class >> > merging SFFS and SFFSS... >> >> RAM is cheap nowadays and if we make this an option then low memory >> nodes can still trade disk io for RAM. > > No, it cannot be optional. These sorts of architectural changes are big enough > that they are either part of the code or not. > >> I give Freenet 512MB and don't >> miss it. ?The disk io, however is unacceptable and causes me to only >> run Freenet when I am not using my PC since db4o was merged... > > And I do not yet understand why. As I asked you on the support list, are you > doing any *up*loads, or is it all downloads? >
No uploads. -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin
