On Thursday 15 January 2009 16:23, Robert Hailey wrote: > > On Jan 15, 2009, at 8:09 AM, Matthew Toseland wrote: > > > We have several conflicting goals here: > > - Minimise request latency for fproxy. > > - Maximise the probability of a request succeeding. > > - Maximise throughput for large downloads. > > - Use all available upstream bandwidth. > > - Don't break routing by causing widespread backoff. > > Perhaps the only issue is the last one (causing backoff)? > > Right now we meter bandwidth by request at accept time (which, seems > logical, as one must accept/deny a request, right?). But then, for any > request, we send it to the closest (non-backed-off) node without > respect to that nodes bandwidth (or what we could guess of it).
It can of course reject it. > Therefore (since we accept the average ability/bandwidth of our peers) > wouldn't we expect that ~50% would be backed off (1/2 has more > bandwidth, 1/2 has less). [I know, it sounds like I favor token > passing now...] Not sure I follow. > > > What makes this even more problematic: > > - The probability of a CHK request succeeding is under 10%. > > But... why do they not succeed? getting lost? not existing? existing- > but-timeout? Mostly lost data as I understand it. Lots more data has been inserted into Freenet than is available at any given time. > > > This may improve, but probably not in the near future. > > > > Therefore, in order to use up our bandwidth, we need to accept > > enough requests > > that we usually have a queue of data to send, even though most of > > them will > > fail. Which means that sometimes we will have a lot of requests > > succeed, and > > the request latency rises dramatically - which is bad for fproxy. > > > > Hence IMHO we need two kinds of requests: > > - Bulk requests: The vast majority of CHK requests are long-term > > queued > > requests for large files. Latency doesn't matter much for these. > > - Realtime requests: Fproxy requests are fewer, but have strong > > latency > > requirements, as well as needing a reasonable psuccess. They are > > especially > > important for new users trying out Freenet. > > > > I appreciate this has been suggested before on the lists and I have > > opposed > > it ... I now think I was wrong. Who was it suggested it first? > > Thelema maybe? > > The main reason I opposed it was that it makes it easier to > > distinguish > > requests. It also increases complexity. IMHO it is necessary > > nonetheless, and > > anyway later on when we have passive requests (IMHO vital both for > > low uptime > > and low latency/polling) we will need something similar. > > It also means one could write an app which would misbehave, making all > the requests realtime. Gains you absolutely nothing, because there are a limited number of realtime slots. That's the point here: bulk is better if you want to do lots of requests. > > Perhaps it is enough for each node to specify to it's peers which > transfers it wants 'now' (with absolute-transfer priority/no other chk > transfers for that time). I guess it would need to chain to the source > to be successful and could be denied at any step. That's the same thing isn't it? > > > The first thing is to implement the turtling proposal. No request > > should take > > more than 60 seconds to transfer, only the node responsible for a > > severe > > delay should be backed off from, the rest of the chain should treat > > it as a > > quick transfer failure / DNF, and the node closest should transfer > > the data > > anyway and offer it via ULPRs. The node sending should treat it as a > > normal > > request. Hopefully this will be working today. After that maybe some > > tuning > > of the load limiting code, but I suspect there isn't much we can do > > given the > > opposing forces mentioned above. > > I think we need to understand the problem a little bit better, as > (given the consistency) I think that the problem is a math/logic > consequence of: > (1) how we average everything for bandwidth (assuming all requests the > same, can we take another one) > (2) treating all requests the same... > (a) in how we transfer them (as you said) > (b) in how we route them (?) > (3) transfers going only as fast as the slowest node I thought I had explained this? Because few requests succeed on average, to get good throughput we need to accept a lot of requests ... but that causes big spikes in latency sometimes...
pgpO8NYIbHbVs.pgp
Description: PGP signature
_______________________________________________ Devl mailing list [email protected] http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
