oo@lkXpu0~CDV6dh0Idyw4MBwkUSgn~h~Bs3qqVXYOXSaY wrote: > toad-notrust@h2RzPS4fEzP0zU43GAfEgxqK2Y55~kEUNR01cWvYApI wrote: > >> liberty-test-of-jFniki@fUGgvPvBMLTdR2C7rk8G-Om9xfMEcLgaQyKimk2pXvU wrote: >> >>> FuckSárközi@lJpyCPT1hcDMA9~Lfa6LoGDqfH7ymInRuaqwH6CZwAE wrote : >>>> oo@lkXpu0~CDV6dh0Idyw4MBwkUSgn~h~Bs3qqVXYOXSaY wrote : >>>>> toad-notrust@h2RzPS4fEzP0zU43GAfEgxqK2Y55~kEUNR01cWvYApI wrote: >>>>> >>>>>>> We are getting downloads stale for days at 0 blocks. There is some >>>>>>> bug in the wakeup mechanism. >>>>> >>>>> Yes, there is such a bug. >>>>> <toad-mode on>Most likely<toad-mode off> once stalled all your >>>>> downloads will proceed again if you change priority of an arbitrary >>>>> download for a few seconds. >>>>> I am anxious to read how toad will integrate this into his paranoid >>>>> imagination that my patches are responsible for everything evil wrt >>>>> freenet and the universe. >> >> Okay how about this: >> >> There are reports of downloads taking a lot longer than people had hoped. >> >> Some have suggested this is "stalling". > > No, that is not meant with my (and most likely tgs3's) observation. > When the node is in that state (call it stalled or whatever) > > - Not any activity can be seen for downloads, neither in fproxy nor Fuqid, > for many hours or even days. > > - The downloads are not dead in the sense of not being retrievable > anymore. > > - If you pick any of those downloads and change its priority *all* > downloads start to proceed normally. > > - After ~1 hour all downloads get 'stalled' again. > > - This problem is somewhat persistent. It happens again after a node > restart and it does not help to remove node.db4o and persistent-temp/* > > - This problem vanishes eventually after ~1 to 3 weeks. I have seen it > several times here. > >> "Stalling" in the most literal sense, as I have previously debugged, >> would mean that particular requests are not getting selected AT ALL. >> >> The patch by oo's sole function is to increase the frequency of *trying >> to send requests*. It does not affect the client layer, where such stall >> supposedly would be. It does not affect the cooldown or RecentlyFailed >> logic. All it does is run the same code more frequently; this still has >> to get past e.g. local load limiting, and it still won't send requests >> for keys that are in cooldown. > > Then something at RecentlyFailed or CoolDown queue seems to get broken. > >> Yet this patch is universally praised as solving the problem. >> >> Conclusion? Requests are not stalling. Requests are being sent "too >> slowly" compared to some arbitrary expectation - maybe even compared to >> past performance. But they are still being sent. > > No. As said above these downloads are not dead, *if* requests would be > sent then most of them would succeed, there would be some observable > activity as it is after changing a download's priority.
Okay, so the patch does not solve the problem, it just causes requests to move faster when they are *NOT* stalled? (Please confirm) In that case, thanks for the bug report. How to fix it is difficult though ... I have debugged stalling problems in the client layer before, I doubt that that is the problem. It could be the same mechanism as bogus RecentlyFailed's. Looks like I'll spend the whole four months from the end of my exams to the start of uni fixing annoying bugs, to no visible benefit. :( > >> Having said that, there does seem to be an issue with bogus >> RecentlyFailed's, which needs looking into. >>>>> >>>>>> Perhaps there is a problem with the AIMD's. Your solution is to >>>>>> ignore the AIMD's. [...] > > Yes, there is a problem with what you call AIMD and no, my patch does not > ignore them completely. > > IMHO the misconception with 'your' AIMDs: > > - For TCP there is a somewhat fixed route of hops where AIMD can adapt to > the performance of that route. If there occurs a re-routing during > transmission then TCP AIMDs adapt to that changed route too. > > - If in TCP the performance (bw limit or whatever you want to call it) of > a route from request to request would be as unrelated as in fred then > AIMDs wouldn't work in TCP neither. > AIMDs in TCP only work because they can adapt to a more or less fixed > route with more or less stable properties. Fred is estimating the capacity of the entire network. This is somewhat vague, which is part of the reason why it adapts relatively slowly. On the other hand, it *should* be reasonably uniform, since nodes should be evenly distributed across the keyspace. This is the direct consequence of the basic metaphor for load management: Backoff is loosely inspired by Ethernet, AIMDs come from TCP. If we have backoff but not AIMDs, backoff becomes useless, because too many nodes get backed off (all nodes except the very fast ones in fact). If we have no backoff but have AIMDs, then one slow node will get far more requests routed to it than are appropriate, and thus slow down everything. If we have no backoff and no AIMDs, one slow node will still get routed to far too much, resulting in performance problems (either by timing out requests, or rejecting them, costing HTL and time). Of course this sucks. We need something better. But pulling out AIMDs without addressing backoffs, or even pulling out backoffs without addressing slow nodes, is insufficient. > > - If you watch the calculated delay value of fred you will notice that it > shows chaotic behaviour from request to request. Calculated values jump by > a factor of 20 and more. There is no adaptive behaviour possible, IMHO > this is true for the whole network. That doesn't make sense. Have you seen the code? src/freenet/node/ThrottleWindowManager.java: static final float PACKET_DROP_DECREASE_MULTIPLE = 0.97f; static final float PACKET_TRANSMIT_INCREMENT = (4 * (1 - (PACKET_DROP_DECREASE_MULTIPLE * PACKET_DROP_DECREASE_MULTIPLE))) / 3; (The latter is from a paper on TCP compatible AIMDs) This should give fairly slow adaptation. Of course I'd be very happy to talk about changing the parameter above, e.g. making it adapt even more slowly. IMHO it should change more quickly for realtime than for bulk, of course. > > - This is the negative feedback loop for all nodes: A negative feedback is exactly what we want: A mechanism that converges to a reasonable, non-zero, stable level. Is it what we have? Maybe not. > As no node has a way > to calculate a somewhat stable delay value where it would *slowly* adapt > to your node doesn't have either. The AIMD "request window" is increased when a request succeeds without a timeout, and without being rejected by any node (the RejectedOverload message with isLocal=true means our direct peer has rejected the request; with isLocal=false, it means a downstream peer rejected it; this is a "slow down" signal). "Load limiting" operates at a higher level: - Determine how many requests are running. - Assume they all succeed. How much data will we need to send? - How long will it take to send it? - If this is over some level, reject the request. This applies to all requests, including locally originated requests. It does not take into account AIMDs at all. There is also a concept of fairness between peers: Every peer is entitled to 1/2n'th of our capacity, but if we're below 1/2 overall, we allow every peer to do as many requests as it wants. Unfortunately *this isn't smooth*, and needs to be got rid of. IMHO replacing this with "every peer has 1/nth of our capacity" is dramatically simpler, allows us to make things MUCH more smooth (by implementing ian's idea of sending some slow-down messages before we actually reach the rejection point), and it also makes NLM much easier (and simpler) when we try that again. Unfortunately increasing the peers limit seems to have resulted in lots of nodes reaching thread limits, which is much less "smooth". See other mails... Also, "negative feedback loop" is a good thing usually: It means that it converges to some non-zero stable level, which is what we want. > > I agree that fred needs some figures about the networks state to locally > calculate the equivalent of an AIMD value but it must be somewhat stable > to be usefull at all. > More than 3 years ago (when fred was performing well) it took about 2 > hours after the release of a new build to reach such a stable state again. > Nowadays it doesn't matter, the network is unstable always. There is always a mythical point somewhere in the past when Freenet worked better. As I have explained, very little has happened code-wise in that period that would explain performance degradation. Which leaves the obvious possibilities: Either your memory is buggy, or there has been a deliberate attack, or there has been an accidental attack. IMHO all three possibilities are worth serious consideration. > > What my RequestStarter patch tries to improve: > > - Average the calculated delay value. My current patch uses the last 400 > samples, maybe this value should be greater like 5000, 10000? > This actually determines how fast or slow the local delay value adapts to > the rest of the network. > If all nodes would use that scheme then this constant should be chosen > such that a stable state is reached again in ~2 to 3 hours after a release > of a new build. This averaging is *the most critical point in what needs > to be changed*. If you don't agree on that then you can just stop reading > as well. This is pointless. The AIMD generates a "window", i.e. a target number of running requests. We then multiply that by the average time taken to complete a request of the type we are starting. See RequestStarterGroup.MyRequestThrottle. If the average RTT is too noisy, change the parameters. It probably is; at the moment it's a Klein filter with a parameter of 0.1 (i.e. avg = avg*0.9 + newvalue*0.1), i.e. similar to averaging the last 10 values. > > - After start of the node it starts with a rather large value of 4000 ms > to adapt to the networks delay from the slow side (to make sure it doesn't > overload its peers in the beginning). AIMDs actually store the last used value IIRC. > > - If your calculated delay value would have been meaningful there wouldn't > be a fixed minDelay and constant a where my delay value eventually > calculates a weighted delay value from (with largest weight on the > averaged value). You could just do something like > lastDelay = (400*lastDelay + networkDelay)/(400 + 1); > delay = lastDelay; > This however *requires* that networkDelay is a meaningful and reasonable > value (which isn't the case currently). Please give me some numbers to demonstrate this. How high and how low have you seen the delay from the AIMDs? I'm not opposed to a lower limit, but there's a difference between imposing a lower limit and averaging it in with a weight. I'm unclear about an upper limit... Clearly if it's completely crazy we should just wait for some period and then try again (with the new delay relative to the time when we started waiting originally). > > - As one doesn't want to let this networkDelay flow freely and eventually > adapt to an arbitrary value I think you need to specify some target value > that is taken into account for delay calculation. This is what is done > with my minDelay value (currently 0 ms, but could be something else like > 100 ms before). IMHO this value isn't critical but should be smaller than > the networks capability, a reasonable small weighting factor can make sure > that the network can reach an optimum and stable networkDelay value. No. This does not make sense. It is not necessary, because you are seeing feedback loops where they don't exist: AIMDs on one node have very little impact on AIMDs on another node. It is of course true that if one node reduces its send rate, other nodes' requests will be more likely to be accepted (at more than 1 hop away). But the decision on whether to accept a request does not depend on the state of the local AIMDs *IN ANY WAY*. > > I do not say that my current values of RequestStarter patch are the values > to go with (constant a is far to great tearing the result down to 0 ms to > be reasonable if the network behaves reasonable already) but we could > start with a safe set of parameters and then see what works best. IMHO you can achieve most of what you are aiming for simply by tweaking a few constants: Make the AIMDs and the roundTripTime RunningAverage a bit less sensitive. Such a patch may be accepted, especially if you figure out some way to evaluate its performance. Of course it won't be as fast as your current patch because your current patch effectively disables sender-side load management completely; I'd like to be able to do that, although even with NLM we will probably need some sender-side throttling (to limit queue timeouts), but it isn't possible with the current system, where we route requests immediately, and backoff if a node rejects requests. > > I know that you don't read me but hopefully some other devs get the ideas > of my comment above. Sorry about that. I have unblocked you. (Note that I have to block anyone who is obviously engaged in copyright infringement, even if they are contributing patches, as under Grokster vs MGM we can't support pirates. But IIRC I just blocked oo because I got annoyed with him.)
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Devl mailing list [email protected] https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
