On Sat, Jul 7, 2012 at 3:52 PM, Juliusz Chroboczek <[email protected]> wrote: >> I will be changing the hardware wifi VO queue in cerowrt 3.3.8-10 to >> be sane (2 packets rather than 128). > > That's probably going to kill your throughput for high-RTT links, unless > you also fix the lack of backpressure in Linux.
As I explained in brussels there are multiple queues in play here. The wifi hardware has a copy of the aggregates it is attempting to ship, 1 or more per hardware queue. Multicast usually ends up in a different bit of the hardware than VO. Linux presently has two or more queues for udp and three or more for tcp. The lowest level is embedded in the driver itself, and this was the queue I was suggesting reducing to saner values. The default value of 128 per driver queue is treble what is needed for a single aggregate, even if all streams were perfectly ordered. Since "sane disorder" is really want you want from "packets"... (eg fair queuing)... All the 4*128 queues in the present ath9k driver does in most cases is add latency. Tons of it. And once you got it stuck in the driver, there's no getting it out. "TCP's reaction time is quadratic to the amount of overbuffering." - van jacobson Codel's effects are nearly unnoticible at driver buffer sizes about 64 packets. I'm well aware that reducing the driver queue length for VI, BE, BK queues reduces the possibilities for aggregation, under linux, especially at the moment. We are evolving towards a plan to revise the linux wireless stack to have one (and only one!) fair queued and aqm'd aggregate accumulation per destination station, rather than the single FIFO which may or may not aggregate. http://www.bufferbloat.net/projects/cerowrt/wiki/Fq_Codel_on_Wireless In the interim it seems sane to reduce all extra forms of latency, work with the fq_codel implementation at the layer we have it at and measure what goes better and/or breaks. > It might break Babel on > networks with more than 120 nodes (the maximum number of packets that > Babel will send in a burst is roughly floor(n/60) where n is the number > of routes). The software "qdisc" queue above the driver queues (where fq_codel resides), in cerowrt is presently set for a limit of 100-600 packets depending on the queue type. This is based on some measurements (not enough!) and the fact that I ran out of memory on the box at fq_codel's default 10000 packets under certain conditions... Judging from the context of your statement here I think you were reading into my mail the idea of reducing all buffering between the software and device to 2. That would be crazy. However the present situation in wireless, with 1000s of unmanaged buffers by default, is also crazy, and contributing to the near-collapse of large wifi networks worldwide. No... the whole point of codel is to have sufficient buffering to absorb bursts, "good queue" but to evolve to a minimum queue length over time, by measuring the delay between entering the queue and exiting it, and starting to discard when above a target delay. http://queue.acm.org/detail.cfm?id=2209336 I appreciate your estimate as to babel's overhead! however I need to note that a buffer size of 400 packets translates out to 24000 routes. Secondly, as I've also noted, multicast packets are (sometimes) accumulated and only broadcast on wifi after some delay after a CAB, and usually at 6Mbits or less. It is my hope that with sane AQM strategies that meshes of much larger size than what we see today can be effectively constructed. >> the same test also reduced throughput by about 40% (from >10Mbit to >> ~6.5Mbit). > > Please try the same on a high-RTT path. Planned. I have setup a 12 node mesh on a campground here (it would be cool to have a battlemesh in california one day), but I was sane about the design, in that it is multichannel, mostly ethernet between radios, and signal strength is universally good. Before I finished getting it up yesterday several prototype locations were bad - and that kicked in all sorts of terrible things, including 3 hop RTT paths in excess of 2 seconds with the default (128) hardware queue size. That's probably not the kind of high RTT path you were thinking of!? Now that I have the good locations up, well, I can get anywhere in about 4.4ms - and under heavy load (thx to fq_codel) usually well under 25. Now that the connections are good - I'll be doing some CDF plots of ping latency under load of hardware queue size of 128 vs 3 in this setup. I suspect these will be interesting. I certainly plan to add the bad nodes back in but not on the backbone. > >> with *your permission* I'd like to be marking babel packets in cerowrt >> as CS6 + ECN capable. > > I tried to explain to you in Brussels (over a year ago) why that's > a horrible kludge. You're applying AQM to locally-originated data in > order to compensate for the lack of backpressure, and then falsely > asserting ECN in order to bypass the AQM policy. I understood your reasoning then and I understand it now, and agree with it, only partially. However there is no "separate, magic queue for routing and system critical packets", aside from the proposed move (in cerowrt at least) of CS6 into the underutilized qdisc VI queue. Where would you supply backpressure without a signaling mechanism of some sort? There are several interesting things that could be done. Have a sys call that told you packets outstanding in the driver? And then what would you do? One of the few negative aspects of fair queueing is that that a stream of packets destined for the same destination will be interspersed with as many other streams as possible. Now this is ok, up to a point, due to the aformentioned separate multicast queue's delays in accumulating what will be sent, but I do note that sending all multicast traffic in a burst will affect all other traffic and it might make sense for a routing protocol like babel to intentionally smear a routing table update across it's update window, rather than temporarily slow the network down so hugely. (I have tried to be very careful about my language in describing the behavior of each queue here) Another approach is to have some sort of congestion indicator that is different from mere reachability. e.g. ECN. which when used end-to-end is very helpful given all the work a wifi network has had to do to get from points A-Z. As for points A-B - link local multicast - I was not suggesting back in brussels that babel ignore ECN, I was suggesting that ecn be leveraged (somehow) in making routing decisions in case of congestion. I think we do agree that better metrics for mesh networks are needed. But, since babel shares a queue with other traffic, when ECT(0) supplied by an AQM it would be an early warning sign. (pretty unreliable given how sparse babel is compared to other flows). ECN enabling babel packets distinguishes between packet drop due to unreachability vs congestion, and should lead to a more reliable mesh network. **IMHO** I can think of several other more silent metrics (like leveraging information from deep within minstrel and mac80211's states regarding transmission rates - or from within codel as to it's drop states) that would probably be more useful to use a base for metrics. > -- Juliusz -- Dave Täht http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-6 is out with fq_codel!" _______________________________________________ Babel-users mailing list [email protected] http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users
