[freenet-support] Freenet 0.7.5 build 1339
Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. Thanks! signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1339
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 01/29/2011 04:54 AM, Matthew Toseland wrote: Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. Thanks! ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe - From [freenet] board. - - fry@a3pdIbytnvYVKOz_Qa8SZLRKu0o - 2011.01.29 - 14:57:29GMT - Build 1338 was great. Would connect to 40 peers, download speed was fast. Build 1339 will connect to 16 out of 40 peers, with half backed off. :-( I downgraded back to 1338, and it immediately picked back up my 40 peers. My network is properly forwarding my Opennet port. - -- http://freedom.libsyn.com/ Echo of Freedom, Radical Podcast None of us are free until all of us are free.~ Mihail Bakunin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJNRFktAAoJENW9VI+wmYasqkkH/1ngO/256kpu7/GK0whGUpcT n6iSO1cyqpxT0kOS2HoJT7aULZGpcTQ+WfjeFUjCMsNLnOQaMvd4FzxQlqtTdqW+ h+MNsSmE3HQd95r6nSVMSfMLMh658hpj31Yv8U4ro3UwsoNG5jgHyapu3hoTldUd ql5XDXSGG3gKN4k70eLU6laOOIZv3J0kdUmIB/g5Fb+Z88ygzERHsxNrxJigIMoZ rU/C6JYq+WEkICZOjNzae29co3mJxQVPTiqOLpyyk7BNPlHu7bSKk9+oCYoSs0B+ m9PkoYYiajM1PZpCO/lUFHcZu2Za/w665sz8XPQndQW+z7zxuTC6AzoJ4pshVWA= =Yt4Q -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1339
On Saturday 29 January 2011 18:15:19 Volodya wrote: On 01/29/2011 04:54 AM, Matthew Toseland wrote: Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. Thanks! From [freenet] board. - fry@a3pdIbytnvYVKOz_Qa8SZLRKu0o - 2011.01.29 - 14:57:29GMT - Build 1338 was great. Would connect to 40 peers, download speed was fast. Build 1339 will connect to 16 out of 40 peers, with half backed off. :-( I downgraded back to 1338, and it immediately picked back up my 40 peers. My network is properly forwarding my Opennet port. Bizarre. Sometimes these things are not build-related at all, they just look like it. Was he using it in the exact same way with each? No fproxy requests? What were the backoff counts for realtime vs bulk? (On the strangers page in advanced mode)? signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1339
On Sat, 29 Jan 2011 01:54:47 +, Matthew Toseland wrote: Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. I'm getting lots and lots of Timeouts and Overloads in my stranger-status details, although not too many BackOffs. After over an hour of uptime, my bandwidth usage hasn't really stabilized. Even the upload speeds, which used to be quite stable around my 15KB/s limit, are very bumpy, quite often near 2KB/s. Same with my download speeds. (Although, at least they aren't flooding :p.) In general, whenever I look at my strangers, most of them will be: FatalTimeout/SENDER_DIED FatalTimeout/TransferFailedInsert FatalTimeout/AfterInsertAcceptedTimeout FatalTimeout/ForwardRejectedOverload2 I've seen a bunch of InsertTimeoutNoFinalAck, TransferFailedRequest (5,13). Is this abnormal, or normal congenstion control? I also see that the ping times to my strangers are very high, 300ms-1000 +ms, even though I can ping google under 20ms. I don't know if this is abnormal, or perhaps all my peers are on the other side of the planet, or perhaps the value is calculated differently. Also, perhaps unrelated, why was I being connected to 11 peers, with a 15KB/s connection? Isn't that too high? Although, even after I halved this, nothing much changed. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1336
On Friday 28 January 2011 23:04:27 Phillip Hutchings wrote: I was referring to Freenet's custom congestion control. There is no resending of UDP packets, unless Freenet pro-actively resends it. Right, and what we do is we resend packets if they are not acknowledged after a few round trips. Which is pretty much what TCP does. I'm not entirely sure how Freenet does it, but it doesn't sound quite the same as TCP. Disclaimer: I'm not an expert on TCP In TCP congestion control is handled by the window size, the exponential backoff algorithm and estimated round-trip time. The window size controls how many bytes can by 'in-flight', that is sent without an ACK received. This is advertised by the receiver as part of the handshake. If an ACK isn't received after a given delay the packet is resent and the window is decreased, say by a power of two. When the ACKs are received in a timely fashion the window size is increased linearly. This stabilises the transmission rate fairly well. Right. We do exactly this, in the link-level AIMD's. However, the congestion window at the moment operates at a relatively high level, not directly controlling the packets in flight, but only controlling the queued-plus-in-flight block transfer messages (not all packets). This is one of several things that needs to be fixed. TCP is more complex than this brief summary, as it also implements a slow-start algorithm and makes an effort to avoid hitting backoff by the linear increase. We also have slow start, and the RFC-specified hack to avoid increasing the window size when we're not filling it. But we need the window to be a true window that applies to all packets, and at the transfer level, not to block transfer messages only. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1336
On Friday 28 January 2011 18:25:09 Dennis Nezic wrote: On Fri, 28 Jan 2011 18:14:14 +, Matthew Toseland wrote: (My last flood occurred for over 10 minutes, and then managed to stop. I believe all 5 of my connected strangers were listed as BackedOff during the flood. I will try to provide more details and more testing.) Was your upstream saturated at the time due to e.g. external pressure? Nope. It might have started the flood -- I'll do some more testing, but during the flood the connection was only minimally being used. (I really don't know why my peers will still dumping so much data onto my node -- perhaps it was in their to-send queues? Perhaps there is a bug, and they didn't get my slow-down traffic messages?) What slow down messages? Are you saying that the peers are actually doing exactly what the node is asking of them, i.e. sending useful data? I.e. it's not a problem with constant resends? How does one differentiate between a resend and useful data? Generally you don't on the receiver side. On the sender side it is obvious from the stats - both the overall bandwidth stats and the per-peer in/out/resent stats. Does the value in the /stats page, or on the peer-list page indicate useful data? If so, it is steadily rising, even when the peer is backed off, even when my downstream is peaked at FIVE TIMES my downstream limit :p completely saturating my downstream (upstream is always generally low-traffic), for many (~5+) minutes. IMHO it is likely that it is resends because we do *try* to respect the downstream limits in our acceptance of requests. I.e. we only accept as many requests as we can transfer the data for, within 60 seconds (realtime) or 120 seconds (bulk); add a bit on for overhead. Plus we have some rarely used token buckets; your reject reasons might be interesting. However, it is still possible for it to max it out for a while even without resending; it just seems more likely that it is resending. The difficulty with the resending theory is that if your upstream is okay, and you don't have the sort of connection where saturating your downstream also makes your upstream break (I believe some forms of DSL have this problem or used to), there is no real reason to expect mass resend, except perhaps for hard to identify bugs. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1339
On Sat, 29 Jan 2011 13:25:48 -0500, Dennis Nezic wrote: On Sat, 29 Jan 2011 01:54:47 +, Matthew Toseland wrote: Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. I'm getting lots and lots of Timeouts and Overloads in my stranger-status details, although not too many BackOffs. After over an hour of uptime, my bandwidth usage hasn't really stabilized. Even the upload speeds, which used to be quite stable around my 15KB/s limit, are very bumpy, quite often near 2KB/s. Same with my download speeds. (Although, at least they aren't flooding :p.) In general, whenever I look at my strangers, most of them will be: FatalTimeout/SENDER_DIED FatalTimeout/TransferFailedInsert FatalTimeout/AfterInsertAcceptedTimeout FatalTimeout/ForwardRejectedOverload2 I've seen a bunch of InsertTimeoutNoFinalAck, TransferFailedRequest (5,13). Is this abnormal, or normal congenstion control? I also see that the ping times to my strangers are very high, 300ms-1000 +ms, even though I can ping google under 20ms. I don't know if this is abnormal, or perhaps all my peers are on the other side of the planet, or perhaps the value is calculated differently. Also, perhaps unrelated, why was I being connected to 11 peers, with a 15KB/s connection? Isn't that too high? Although, even after I halved this, nothing much changed. (Cool stats btw :b) 1h 28min uptime: nodeAveragePingTime: 883ms # backedOffPercent: 4.9% # pInstantReject: 27.6% Input/Output rates well under my limits. (Average rates under half my limit.) Routing Backoff Reason Count Avg. Time Total Time Timeout 1 7.955s 7.955s TransferFailedInsert10 7.082s 1m10s FatalTimeout22 6.158s 2m15s AcceptedTimeout 123 4.990s 10m13s ForwardRejectedOverload41 3.016s 3.016s ForwardRejectedOverload53 1.614s 4.843s ForwardRejectedOverload 152 1.490s 3m46s TransferFailedRequest5 37 1.459s 54.004s TransferFailedRequest13 2 1.314s 2.629s InsertTimeoutNoFinalAck 5 1.219s 6.096s TransferFailedRequest7 2 1.217s 2.435s AfterInsertAcceptedTimeout 2 1.196s 2.392s ForwardRejectedOverload220 0.975s 19.519s ForwardRejectedOverload31 0.180s 0.180s Transfer Backoff Reason Count Avg. Time Total Time SENDER_DIED 24 1m17s 30m54s ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Freenet 0.7.5 build 1336
On Friday 28 January 2011 18:39:53 Dennis Nezic wrote: On Fri, 28 Jan 2011 18:11:29 +, Matthew Toseland wrote: CC'ing Martin in case he has any ideas. On Thursday 27 January 2011 19:01:28 Dennis Nezic wrote: On Thu, 27 Jan 2011 18:49:57 +, Matthew Toseland wrote: On Thursday 27 January 2011 17:17:15 Dennis Nezic wrote: On Thu, 27 Jan 2011 16:55:58 +, Matthew Toseland wrote: On Wednesday 26 January 2011 20:14:52 Dennis Nezic wrote: On Wed, 26 Jan 2011 19:59:43 +, Matthew Toseland wrote: On Wednesday 26 January 2011 19:42:38 Dennis Nezic wrote: On Wed, 26 Jan 2011 19:38:55 +, Matthew Toseland wrote: On Monday 24 January 2011 22:28:36 you wrote: On Mon, 24 Jan 2011 18:29:14 +, Matthew Toseland wrote: Freenet 0.7.5 build 1336 is now available. Please upgrade, it will be mandatory on Friday. Much of this build is intended to try to improve network performance and particularly to prevent transfer failures especially on realtime requests. Details: - Fix some block transfer bugs related to transfer coalescing, disconnects and reassigning to self after a request has taken too long. - Keep on transferring the data if we need it for a local request, and explain why this is safe in comments. However it will go away when we get rid of receiver-side transfer cancels anyway. - Fix even more bugs related to request forking on timeout. - Always drop the queued messages when we disconnect due to a timeout. - Eliminate turtle transfers, they are no longer necessary and involve unnecessary complexity and transfer cancels. We will soon eliminate receiver cancels too which will further simplify matters. - Remove timed out filters more often. - Show totals for backoff times. - Fixes to auto-testing code. This build still can't manage to handle input bandwidth sanely, on a congested connection -- it entirely consumes my already busy connection. (Aka. my freenet is still unusable here.) Limiting input bandwidth usage accurately is extraordinarily difficult and most people have fat pipes downstream. Probably your peers are resending packets constantly. So, umm, make them stop, after say a minute of flooding? And disconnect completely? That kind of defeats the point doesn't it? No, that doesn't defeat the point. The point is to have a running Freenet, which is simply not possible at the moment, since it will flood my internet connection to an unusable state. A running Freenet that disconnects from all your peers and constantly announces because your connection is broken? How is that useful? My connection isn't broken. I actually took much pain to somewhat guarantee that my Freenet gets the bandwidth I allocate it. Moreover, what's the problem with completely disconnecting from a peer, after it continues to flood us for many minutes? You are flooding it, not the other way around. It's entirely your fault for not having the bandwidth to handle the data. Freenet is tested and developed on typical connections which have lots of downstream bandwidth and not much upstream bandwidth. And Freenet is only doing exactly what TCP would do! Admittedly TCP has a different algorithm for estimating round trip times, and a more precise congestion control mechanism. But I have yet to be convinced that this is a serious issue for any other person than you and therefore worth spending significant amounts of developer time on. I'm also not sure exactly what would fix it... I really think you haven't understood the problem yet. This isn't just a bumpy averaging of bursting input -- this is a *sustained 5+ minute flood that completely uses up 100KB/s downstream connection, and 5x the bandwidth I allocate it*. You're not sure what would fix it? Like I mentioned a couple times here already, *how about after a minute or two, telling my connected peers to SLOW DOWN*? The reason they are sending data is that they are NOT receiving your acknowledgements. So sending yet more control data won't help. Well that sure sounds like a recipe for disaster. How long are peers supposed to keep pushing data to un-acknowledging peers, before they do something? We disconnect from a node if we haven't received anything from it in 60 seconds.
Re: [freenet-support] Freenet 0.7.5 build 1339
On Saturday 29 January 2011 18:25:48 Dennis Nezic wrote: On Sat, 29 Jan 2011 01:54:47 +, Matthew Toseland wrote: Build 1339 is out. Please upgrade asap, it will be mandatory on Monday. The main change in this build is that backoff is now separate for realtime versus bulk requests. This means, hopefully, that if the performance problems recently have been caused by realtime requests causing lots of backoff, this will only affect realtime requests. It is investigating a theory, one of several, regarding the recent problems. I am sorry that the network has behaved so badly recently, I am working on it, but it is not easy. Please upgrade, and please report any and all problems you find. There is a thread on FMS where I am trying to get a better understanding of what problems people are seeing. So far the main reported issues seem to be: - Realtime requests (e.g. fproxy) are slow, and cause all or most peers to get backed off. - Downloads are very slow. - Bootstrapping onto the network is slow. I'm getting lots and lots of Timeouts and Overloads in my stranger-status details, although not too many BackOffs. After over an hour of uptime, my bandwidth usage hasn't really stabilized. Even the upload speeds, which used to be quite stable around my 15KB/s limit, are very bumpy, quite often near 2KB/s. Same with my download speeds. (Although, at least they aren't flooding :p.) This is different to 1338? I don't know what is normal for your node, you have a rather odd setup. Testing so far suggests once it settles fproxy performs almost acceptable, dunno about bulk downloads. In general, whenever I look at my strangers, most of them will be: FatalTimeout/SENDER_DIED FatalTimeout/TransferFailedInsert FatalTimeout/AfterInsertAcceptedTimeout FatalTimeout/ForwardRejectedOverload2 See that here too, but mostly for realtime. You doing lots of uploads? I see more TransferFailedRequest. I've seen a bunch of InsertTimeoutNoFinalAck, TransferFailedRequest (5,13). Is this abnormal, or normal congenstion control? This is a different layer and it's not normal but it's pretty common. I also see that the ping times to my strangers are very high, 300ms-1000 +ms, even though I can ping google under 20ms. I don't know if this is abnormal, or perhaps all my peers are on the other side of the planet, or perhaps the value is calculated differently. This is common. UDP and ICMP are routed differently, and there are significant delays imposed by Freenet for bandwidth efficiency reasons. Also, perhaps unrelated, why was I being connected to 11 peers, with a 15KB/s connection? Isn't that too high? Although, even after I halved this, nothing much changed. There is a minimum. IIRC it is 10 peers. Beyond that it's a square-root relationship because each peer brings more traffic incoming as well as outgoing. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Please test the new load management branch
Please get the snapshot (update.cmd testing / update.sh testing), and test it. I want to know if it causes serious problems, and also any other bugs you run into. I know there will be various errors, but I am still interested in the more severe ones. For people building from source: The tag is: testing-build-1340-maybe-merge-new-load-management-pre1 The branch is: merge-new-load-management The git rev is 47edfe611a1f088b51fccb36d3e7710e28c3d2b8 (that commit does matter despite it appearing to be a logging fix). THANKS! signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] [freenet-dev] Please test the new load management branch
On Sunday 30 January 2011 00:30:09 Matthew Toseland wrote: Please get the snapshot (update.cmd testing / update.sh testing), and test it. I want to know if it causes serious problems, and also any other bugs you run into. I know there will be various errors, but I am still interested in the more severe ones. For people building from source: The tag is: testing-build-1340-maybe-merge-new-load-management-pre1 The branch is: merge-new-load-management The git rev is 47edfe611a1f088b51fccb36d3e7710e28c3d2b8 (that commit does matter despite it appearing to be a logging fix). THANKS! More detail on the Frost post, basically if you can run both nodes on the main network and maybe some parallel darknet test-network as well, that'd be incredibly awesome (turn on all the cache-absolutely-everything options to make a small network work). Feel free to exchange noderefs over IRC, email, and FMS, in complete violation of normal rules about such things; if we don't release new load management soon, and possibly even if we do, I will probably reinstate the testnet network. [00:35:21] * toad_ posted appeal for new load management testers on FMS [00:35:39] * toad_ needs both test-network testing and real-network testing, feel free to get involved in both forms [00:35:51] TheSeeker o.รด [16:33.57] Flatron_55 Pong reply to 33.20 from TheSeeker at xx:33:43 [00:35:54] toad_ and to exchange noderefs over FMS and do other silly foolish things to get the network up [00:36:05] toad_ it's a test network remember? [00:36:05] toad_ :) [00:36:17] toad_ my test nodes are running on the main network [00:36:40] toad_ i suggest testers with reasonable bandwidth run one or more main network nodes and also one or more test network nodes [00:37:20] * toad_ is aiming to release new load management *EARLY* next week if possible, so any efforts over the next 48 hours could be very helpful signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
