On Jan 15, 2009, at 8:09 AM, Matthew Toseland wrote:
> We have several conflicting goals here:
> - Minimise request latency for fproxy.
> - Maximise the probability of a request succeeding.
> - Maximise throughput for large downloads.
> - Use all available upstream bandwidth.
> - Don't break routing by causing widespread backoff.
Perhaps the only issue is the last one (causing backoff)?
Right now we meter bandwidth by request at accept time (which, seems
logical, as one must accept/deny a request, right?). But then, for any
request, we send it to the closest (non-backed-off) node without
respect to that nodes bandwidth (or what we could guess of it).
Therefore (since we accept the average ability/bandwidth of our peers)
wouldn't we expect that ~50% would be backed off (1/2 has more
bandwidth, 1/2 has less). [I know, it sounds like I favor token
passing now...]
> What makes this even more problematic:
> - The probability of a CHK request succeeding is under 10%.
But... why do they not succeed? getting lost? not existing? existing-
but-timeout?
>
>
> This may improve, but probably not in the near future.
>
> Therefore, in order to use up our bandwidth, we need to accept
> enough requests
> that we usually have a queue of data to send, even though most of
> them will
> fail. Which means that sometimes we will have a lot of requests
> succeed, and
> the request latency rises dramatically - which is bad for fproxy.
>
> Hence IMHO we need two kinds of requests:
> - Bulk requests: The vast majority of CHK requests are long-term
> queued
> requests for large files. Latency doesn't matter much for these.
> - Realtime requests: Fproxy requests are fewer, but have strong
> latency
> requirements, as well as needing a reasonable psuccess. They are
> especially
> important for new users trying out Freenet.
>
> I appreciate this has been suggested before on the lists and I have
> opposed
> it ... I now think I was wrong. Who was it suggested it first?
> Thelema maybe?
> The main reason I opposed it was that it makes it easier to
> distinguish
> requests. It also increases complexity. IMHO it is necessary
> nonetheless, and
> anyway later on when we have passive requests (IMHO vital both for
> low uptime
> and low latency/polling) we will need something similar.
It also means one could write an app which would misbehave, making all
the requests realtime.
Perhaps it is enough for each node to specify to it's peers which
transfers it wants 'now' (with absolute-transfer priority/no other chk
transfers for that time). I guess it would need to chain to the source
to be successful and could be denied at any step.
> The first thing is to implement the turtling proposal. No request
> should take
> more than 60 seconds to transfer, only the node responsible for a
> severe
> delay should be backed off from, the rest of the chain should treat
> it as a
> quick transfer failure / DNF, and the node closest should transfer
> the data
> anyway and offer it via ULPRs. The node sending should treat it as a
> normal
> request. Hopefully this will be working today. After that maybe some
> tuning
> of the load limiting code, but I suspect there isn't much we can do
> given the
> opposing forces mentioned above.
I think we need to understand the problem a little bit better, as
(given the consistency) I think that the problem is a math/logic
consequence of:
(1) how we average everything for bandwidth (assuming all requests the
same, can we take another one)
(2) treating all requests the same...
(a) in how we transfer them (as you said)
(b) in how we route them (?)
(3) transfers going only as fast as the slowest node
--
Robert Hailey
_______________________________________________
Devl mailing list
[email protected]
http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
