Re: [OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
Thx for the numbers! Could you do a TCP_RR while under load from UDP_STREAM? On Mon, May 21, 2012 at 1:31 AM, Tobias Diedrich wrote: > Tobias Diedrich wrote: >> Dave Taht wrote: >> > In looking over the enormous stack of boards and drivers that openwrt >> > supports, I see that many of the ethernet drivers don't yet support >> > Linux 3.3's "Byte Queue Limits", which are discussed here: >> > >> > http://lwn.net/Articles/454390/ >> > >> > It would be good if more did. They improve network performance in the >> > general case enormously, particularly when a link is not connected at >> > it's peak wire speed. >> > >> > *Adding* support for BQL to an ethernet driver is trivial, here's an >> > example of how. >> >> I tried adding BQL to the ramips ethernet driver, however I found >> some interesting behaviour while doing >> "root@OpenWrt:~# netperf -l 120 -t UDP_STREAM -H myserver" >> >> It looks like the briding code still needs to implement this as well? >> >> netperf UDP_STREAM: >> iface limit_min inflight tx mbps remote mbps ping ms >> eth0 0 ~15000 95.71 95.71 ~10ms >> eth0 100 ~30 177.98 23.28(*) ~30ms >> br0 0 ~15000 154.88 33.94(*) ~120ms >> br0 100 ~30 170.92 25.57(*) ~30ms >> >> (*) bwm-ng on the server showed ~100mbps incoming... > [...] >> Haven't tried codel yet... > > Turns out, it works nicely with codel, even with the bridge: > > netperf: netperf -l 120 -t UDP_STREAM -H myserver > fq_codel: tc qdisc add dev eth0 handle 1: root fq_codel target 5ms > > iface eth0 qdisc bql inflight tx mbps sys time ping ms > eth0 pfifo_fast no n/a 182.98(*) 96.43s ~30ms > eth0 fq_codel no n/a 177.98(*) 96.09s ~30ms > eth0 pfifo_fast yes ~15000 95.71 42.73s ~10ms > eth0 fq_codel yes ~15000 95.19 51.52s ~4ms > br0 pfifo_fast yes ~15000 155.19(*) 94.24s ~120ms > br0 fq_codel yes ~15000 90.92 65.52s ~4ms > > (*) 100mbit link after the switch, ifconfig eth0 shows no drops, > so I'm assuming they are getting dropped by the switch. > > -- > Tobias PGP: http://8ef7ddba.uguu.de -- Dave Täht SKYPE: davetaht US Tel: 1-239-829-5608 http://www.bufferbloat.net ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
Tobias Diedrich wrote: > Dave Taht wrote: > > In looking over the enormous stack of boards and drivers that openwrt > > supports, I see that many of the ethernet drivers don't yet support > > Linux 3.3's "Byte Queue Limits", which are discussed here: > > > > http://lwn.net/Articles/454390/ > > > > It would be good if more did. They improve network performance in the > > general case enormously, particularly when a link is not connected at > > it's peak wire speed. > > > > *Adding* support for BQL to an ethernet driver is trivial, here's an > > example of how. > > I tried adding BQL to the ramips ethernet driver, however I found > some interesting behaviour while doing > "root@OpenWrt:~# netperf -l 120 -t UDP_STREAM -H myserver" > > It looks like the briding code still needs to implement this as well? > > netperf UDP_STREAM: > iface limit_min inflight tx mbps remote mbps ping ms > eth0 0 ~1500095.7195.71~10ms > eth0 100 ~30 177.98 23.28(*) ~30ms > br00 ~15000154.88 33.94(*) ~120ms > br0100 ~30 170.92 25.57(*) ~30ms > > (*) bwm-ng on the server showed ~100mbps incoming... [...] > Haven't tried codel yet... Turns out, it works nicely with codel, even with the bridge: netperf: netperf -l 120 -t UDP_STREAM -H myserver fq_codel: tc qdisc add dev eth0 handle 1: root fq_codel target 5ms iface eth0 qdisc bql inflight tx mbpssys time ping ms eth0 pfifo_fast no n/a 182.98(*) 96.43s~30ms eth0 fq_codelno n/a 177.98(*) 96.09s~30ms eth0 pfifo_fast yes ~1500095.71 42.73s~10ms eth0 fq_codelyes ~1500095.19 51.52s~4ms br0pfifo_fast yes ~15000155.19(*) 94.24s~120ms br0fq_codelyes ~1500090.92 65.52s~4ms (*) 100mbit link after the switch, ifconfig eth0 shows no drops, so I'm assuming they are getting dropped by the switch. -- Tobias PGP: http://8ef7ddba.uguu.de ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
Dave Taht wrote: > In looking over the enormous stack of boards and drivers that openwrt > supports, I see that many of the ethernet drivers don't yet support > Linux 3.3's "Byte Queue Limits", which are discussed here: > > http://lwn.net/Articles/454390/ > > It would be good if more did. They improve network performance in the > general case enormously, particularly when a link is not connected at > it's peak wire speed. > > *Adding* support for BQL to an ethernet driver is trivial, here's an > example of how. I tried adding BQL to the ramips ethernet driver, however I found some interesting behaviour while doing "root@OpenWrt:~# netperf -l 120 -t UDP_STREAM -H myserver" It looks like the briding code still needs to implement this as well? netperf UDP_STREAM: iface limit_min inflight tx mbps remote mbps ping ms eth0 0 ~1500095.7195.71~10ms eth0 100 ~30 177.98 23.28(*) ~30ms br00 ~15000154.88 33.94(*) ~120ms br0100 ~30 170.92 25.57(*) ~30ms (*) bwm-ng on the server showed ~100mbps incoming... It is notable that the internal interface to the cpu is 1000Mbit, while the external interface after the built-in switch is only 100Mbit on this device and the switch can drop packets if the incoming rate is too high. The switch defaults are: Back pressure mode: JAM all until BP condition is released BP_JAM_CNT: 10 (10 packet jam, then one no-jam) It also supports 'carrier insertion' instead of jamming. And there are some flow control threshold values that could be tweaked. Haven't tried codel yet... -- Tobias PGP: http://8ef7ddba.uguu.de ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
On Thu, May 10, 2012 at 9:13 AM, John Crispin wrote: > On 10/05/12 17:57, Dave Taht wrote: >> In looking over the enormous stack of boards and drivers that openwrt >> supports, I see that many of the ethernet drivers don't yet support >> Linux 3.3's "Byte Queue Limits", which are discussed here: >> >> http://lwn.net/Articles/454390/ >> >> It would be good if more did. They improve network performance in the >> general case enormously, particularly when a link is not connected at >> it's peak wire speed. >> >> *Adding* support for BQL to an ethernet driver is trivial, here's an >> example of how. >> >> http://huchra.bufferbloat.net/~cero1/openwrt-bql-patches/0103-BQL-support-added-to-ag71xx-driver.patch >> >> * testing BQL* is also trivial - but will require the device to do. >> I've been beat up by memory barrier issues and stuff like that, so it >> pays to do the patch and then run serious tests through it, rather >> than try it blind. > > Hi, > > do you run specific tests or just hammer the unit ? Hammering via any means is good - netperf, iperf, apache benchmarks, etc. We have been using (and improving) netperf to exercise things like classification and alternate tcp algorithms. You can get that from netperf.org's svn, or There is a version of that already packaged up in the openwrt compatible ceropackages repository here: https://github.com/dtaht/ceropackages-3.3 the new (to be 2.6) version of netperf is not backward compatible with version 2.5, so all sides of the connection need to be running it. useful tests are TCP_STREAM, TCP_MAERTS, and TCP_RR, run multiple times simultaneously, for long periods. (memory barriers are hard to hit). Codel shows up well in a simultaneous TCP_STREAM and TCP_RR test vs pfifo_fast, too. Iperf is a simpler substitute. > > john > > > ___ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel -- Dave Täht SKYPE: davetaht US Tel: 1-239-829-5608 http://www.bufferbloat.net ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
On 10/05/12 17:57, Dave Taht wrote: > In looking over the enormous stack of boards and drivers that openwrt > supports, I see that many of the ethernet drivers don't yet support > Linux 3.3's "Byte Queue Limits", which are discussed here: > > http://lwn.net/Articles/454390/ > > It would be good if more did. They improve network performance in the > general case enormously, particularly when a link is not connected at > it's peak wire speed. > > *Adding* support for BQL to an ethernet driver is trivial, here's an > example of how. > > http://huchra.bufferbloat.net/~cero1/openwrt-bql-patches/0103-BQL-support-added-to-ag71xx-driver.patch > > * testing BQL* is also trivial - but will require the device to do. > I've been beat up by memory barrier issues and stuff like that, so it > pays to do the patch and then run serious tests through it, rather > than try it blind. Hi, do you run specific tests or just hammer the unit ? john ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] BQL support in Ethernet drivers (and Kathie Nichols and Van Jacobson's new AQM, codel)
In looking over the enormous stack of boards and drivers that openwrt supports, I see that many of the ethernet drivers don't yet support Linux 3.3's "Byte Queue Limits", which are discussed here: http://lwn.net/Articles/454390/ It would be good if more did. They improve network performance in the general case enormously, particularly when a link is not connected at it's peak wire speed. *Adding* support for BQL to an ethernet driver is trivial, here's an example of how. http://huchra.bufferbloat.net/~cero1/openwrt-bql-patches/0103-BQL-support-added-to-ag71xx-driver.patch * testing BQL* is also trivial - but will require the device to do. I've been beat up by memory barrier issues and stuff like that, so it pays to do the patch and then run serious tests through it, rather than try it blind. So, getting more BQL enabled ethernet drivers working would be good. There is BQL support in many of the mainstream ethernet cards already, see those for more examples of how to do it. ... Anyway, as folk are hopefully aware by now, Kathie Nichols and Van Jacobson have released a new "no knobs" AQM algorithm which can be used by default on all interfaces - not just the uplink - in order to manage bandwidth and latency far better than has ever been done before. If you haven't heard about it yet, please see http://queue.acm.org/detail.cfm?id=2209336 for the paper. Figure 7 shows the enormous differences in latency and buffering behavior between drop tail (the current linux default), RED, and Codel. There's also been good coverage in lwn, jim gettys blog, arstechnica, etc. I'd ignore slashdot. Codel is a brand new algorithm, something truly new under the sun, which doesn't happen every day. Kathie and Van been working on it for over a decade and the bufferbloat/cerowrt team for over a year. We took the wraps off the algorithm a few days ago. We've made the code freely available under a dual BSD/GPL license. AQM is not just for routers anymore, either, but it helps a lot there! Versions of Codel for linux 3.3 and 3.4 (and ns2 and ns3 simulations) are here: http://www.bufferbloat.net/projects/codel/wiki Codel will work best on a BQL-enabled ethernet driver. It will also work on devices with small numbers of ring buffers. I look forward to hearing of early successes (and failures!) on various bits of hardware. Certainly while codel + bql works well, htb and hfsc in combination with various other technologies is not well tested yet. Codel is a RED replacement, and will work best when paired up with QFQ, or the upcoming sfq-enabled version. However by itself it's pretty darn good, and can be used on all interfaces of a router, not just the uplink. Work on characterizing wireless behavior is on-going. (some help on that would be nice) There is still a great deal of work ahead for the cerowrt project, it's my hope that by sharing this stuff as soon as I felt the patches were baked enough, that all those frustrated by network delays and bufferbloat on supposedly fast hardware, would jump on it, on their own hardware. Our intent from day one was to get Codel into OpenWrt (and indeed, all OSes) as fast as possible. Unfortunately I kind of forgot about getting BQL on everything, everywhere, too! -- Dave Täht SKYPE: davetaht US Tel: 1-239-829-5608 http://www.bufferbloat.net ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
