Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
It looks like the 1900v2 and the 1200 have the same chipset, i saw that the 1900v2 got more memory and a faster cpu to bring it up to match the 1200. My understanding is that the only difference between the two is 2x2 vs 3x3 and the cost. David Lang On Thu, 2 Jul 2015, dpr...@reed.com wrote: Having not bought a 1200ac yet, I was wondering if I should splurge for the 1900ac v2 (which has lots of memory unlike the 1900ac v1). Any thoughts on the compatibility of this with the 1200ac? Current plans are to deploy Supermicro Mini ITX A1SRI-2558F-O Quad Core (Rangely) as my externally facing router and services platform, and either one of the above as my experimental wireless solution. On Thursday, July 2, 2015 11:47am, Toke Høiland-Jørgensen t...@toke.dk said: Mikael Abrahamsson swm...@swm.pp.se writes: Do you have a link to your .config for your builds somewhere? http://swm.pp.se/aqm/wrt1200ac.config Cool, thanks! BUT! I have had problems getting WPA2 to work properly with this .config. I must have missed something that is needed that has to do with the password/crypto handling. There already is a profile for the WRT1200AC (caiman) in Chaos Calmer RC and trunk, so it's actually not that hard to get working. The biggest problem is finding all those utilities one wants and making sure they're compiled into the image so one doesn't have to add them later. Yeah, realise that. Still have my old .config from when I used to build cerowrt for the WNDR lying around somewhere, so will take a look at that and make sure everything is in there :) -Toke ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi, for the sake of spreading this caveat, it does seem like the open source driver for wifi in the WRT1200AC isn't in very good shape. It works, but it's very slow, speeds vary depending on what kind of client I connect with, and the latency is horrible even with fq_codel or cake enabled on it. So if you were considering buying it and using it as a daily driver for wifi in your home, wait until I have better news regarding the wifi driver. I've already ping:ed Marvell about it and also spoken to one developer working for Belkin/Linksys, and they're aware of the problem. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Mikael Abrahamsson swm...@swm.pp.se writes: So if you were considering buying it and using it as a daily driver for wifi in your home, wait until I have better news regarding the wifi driver. I've already ping:ed Marvell about it and also spoken to one developer working for Belkin/Linksys, and they're aware of the problem. Do keep us up to date on this. I have one on my desk ready to be flashed once I get a bit of free time. Don't have anyone else to annoy with possible poor performance, though, so can live with experimental drivers :) Oh, and of course I volunteer to test any such new drivers. Hmm, maybe I should try to get my hands on a 802.11ac *client* as well... :P Do you have a link to your .config for your builds somewhere? -Toke ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Folks, ... The biggest problem is finding all those utilities one wants and making sure they're compiled into the image so one doesn't have to add them later. Yeah, realise that. Still have my old .config from when I used to build cerowrt for the WNDR lying around somewhere, so will take a look at that and make sure everything is in there :) The CeroWrtScripts repo also has config-cerowrt.sh, a shell script that I run immediately after flashing to configure a flock of things. The second step (after configuring and bringing up ge00) is to pull in all the packages I want to use. This might save a bunch of wrasslin' with the build scripts/configuration. Check the script at: https://github.com/richb-hanover/CeroWrtScripts#config-cerowrtsh Best, Rich ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Having not bought a 1200ac yet, I was wondering if I should splurge for the 1900ac v2 (which has lots of memory unlike the 1900ac v1). Any thoughts on the compatibility of this with the 1200ac? Current plans are to deploy Supermicro Mini ITX A1SRI-2558F-O Quad Core (Rangely) as my externally facing router and services platform, and either one of the above as my experimental wireless solution. On Thursday, July 2, 2015 11:47am, Toke Høiland-Jørgensen t...@toke.dk said: Mikael Abrahamsson swm...@swm.pp.se writes: Do you have a link to your .config for your builds somewhere? http://swm.pp.se/aqm/wrt1200ac.config Cool, thanks! BUT! I have had problems getting WPA2 to work properly with this .config. I must have missed something that is needed that has to do with the password/crypto handling. There already is a profile for the WRT1200AC (caiman) in Chaos Calmer RC and trunk, so it's actually not that hard to get working. The biggest problem is finding all those utilities one wants and making sure they're compiled into the image so one doesn't have to add them later. Yeah, realise that. Still have my old .config from when I used to build cerowrt for the WNDR lying around somewhere, so will take a look at that and make sure everything is in there :) -Toke ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Thu, 2 Jul 2015, dpr...@reed.com wrote: Having not bought a 1200ac yet, I was wondering if I should splurge for the 1900ac v2 (which has lots of memory unlike the 1900ac v1). From what I can tell, the only thing that differs from the WRT1200AC is the radio. It still uses marvell radio but with more everything, apart from RAM that (for some reason) the WRT1200AC has 512MB RAM but according to http://wiki.openwrt.org/toh/linksys/wrt1900ac the WRT1900ACv2 only has 256MB RAM. I don't have any information on the performance of the radio in the WRT1900ACv2. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Dave Taht wrote:
attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
I compiled openwrt trunk with linux kernel v4.0 and this patch, and the
results are here http://swm.pp.se/aqm/rrul_150630-cake-l4.0-1.tar.
As far as I can tell sirq load is higher rather than lower so it doesn't
seem like kernel 4.0 has any significant performance benefits, rather the
opposite.
--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Tue, 30 Jun 2015, Mikael Abrahamsson wrote:
On Mon, 29 Jun 2015, Dave Taht wrote:
attached is a patch for that, put it in your
feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
I compiled openwrt trunk with linux kernel v4.0 and this patch, and the
results are here http://swm.pp.se/aqm/rrul_150630-cake-l4.0-1.tar.
As far as I can tell sirq load is higher rather than lower so it doesn't seem
like kernel 4.0 has any significant performance benefits, rather the
opposite.
So it seems I was mistaken. I now did some tests, with 4.0 with the
cakepatch, with 3.18 with the cakepatch (that Dave sent the other day),
and then 3.18 with regular cake as it is in the ceropackages repo
yesterday.
The columns are mss size, -R or not means reverse, so with -R main packet
sizes are going in the server-client direction, ie flowing into eth0
which hosts the htb. Then it's the megabit/s as measured by iperf and then
the sirq load as seen by top. This jumps around a bit so don't read too
much into it. However, it looks like 4.0 is actually a slight improvement
especially for smaller packet sizes and in the server-client direction.
4.0 cakepatch
-M 200 -R 167 M 94%
-M 300 -R 188 M 71%
-M 600 -R 362 M 77%
-R 861 M 88%
-M 200 350 M 88%
-M 300 380 M 80%
-M 600 680 M 63%
860 M 55%
3.18 - cakepatch
-M 200 -R 140M 83%
-M 300 -R 167M 72%
-M 600 -R 308M 69%
-R 750M 82%
-M 200 289M 74%
-M 300 406M 73%
-M 600 780M 80%
860M 57%
3.18 vanilla
-M 200 -R 150M 90%
-M 300 -R 166M 72%
-M 600 -R 305M 68%
-M -R 740M 82%
-M 200 304M 80%
-M 300 440M 80%
-M 600 800M 81%
863M 56%
--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, On Jun 29, 2015, at 15:00 , Mikael Abrahamsson swm...@swm.pp.se wrote: [...] Hi, Ok, yes, this worked, I must have forgotten do to update after I moved ceropackages to the top of the list before. Thanks! So now I have a sysupgrade image for the wrt1200ac that out of the box comes with CeroPackages and working bidirectional shaping for cake (don't know why it didn't work before, it might have to do with my modifications. This time I didn't modify anything on-disk, this is purely from the CeroPackages feed). Erm, I committed the fix for the $DEV $IFACE confusion to the ceropackages repository, so you got the fixed version you helped fix ;) (that or you deselected ingress 3-tier classification). I did try to get Kernel 4.1 to compile but that didn't work even though I removed some packages that didn't compile, I ended up with no .dts file and nothing to me obvious in scrollback to fix. So this is with 3.18. Here are the cake 50M and 500M results and output from the router: oot@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option enabled '1' option script 'simple.qos' option qdisc 'cake' option linklayer_adaptation_mechanism 'cake' option download '50' option upload '50' root@OpenWrt:~# tc -d qdisc qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows noatm overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4 flows noatm overhead 42 I love that passing per packet overheads to cake works, but for your testing it is going to reduce the maximum achievable speed from: TCP/IPv4 Payload at 500Mbps shaping: 1500 - 20 - 20 = 1460 Byte payload to OTWS ratio 1460/1518 = 0.961791831357 Payload Bandwidth 500*1460/1518 = 480.90 Mbps to: TCP/IPv4 Payload at 500Mbps shaping: with overhead 42 MTU: 1500 MSS: 1500 - 20 - 20 = 1460 Byte PerPacketOverhead: 42 configured On-The-Wire Size: 1500 + 42 = 1542 Byte payload to OTWS ratio (due to the explicitly configured overhead42) 1460/1542 = 0.961791831357 Payload Bandwidth 500*1460/1542 = 473.411154345 Mbps I really ned to get a new router to take part in all the fun… Best Regards Sebastian These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script. http://swm.pp.se/aqm/rrul_150629-cake-4.tar Then I re-did the test that Dave asked before, I set the wan port to 100 megabit/s in my switch, and removed the SQM. It resulted in the following config: root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option script 'simple.qos' option qdisc 'cake' option linklayer_adaptation_mechanism 'cake' option download '5' option upload '5' option enabled '0' root@OpenWrt:~# tc -d qdisc qdisc mq 0: dev eth0 root qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
HI Mikael,, hi Jonathan, [...] These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script. http://swm.pp.se/aqm/rrul_150629-cake-4.tar Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before… Best Regards Sebastian ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
I would love to try out cake in my environment. However, as a non-combatant, it would be nice to have an instruction sheet on how to set the latest version up, and what hardware it works best on (WRT1200AC?). Obviously this is a work in progress, so that will change, but it would be nice to have a summarized wiki page. ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Toke Høiland-Jørgensen wrote: Mikael Abrahamsson swm...@swm.pp.se writes: How can I tell which one of these actually is included? Tried moving the feed statement so ceropackages was first but that doesn't seem to have helped, I still get OpenWrt regular sqm-scripts (no cake in luci-app-sqm). You need to have the cero feed first in feeds.conf, then do ./script/feeds uninstall sqm-scripts; ./script/feeds install sqm-scripts -- that should pull it from the cero feed. If it doesn't work, try doing ./scripts/feeds update first That seemed to work for me when I just tested it now, at least :) Hi, Ok, yes, this worked, I must have forgotten do to update after I moved ceropackages to the top of the list before. Thanks! So now I have a sysupgrade image for the wrt1200ac that out of the box comes with CeroPackages and working bidirectional shaping for cake (don't know why it didn't work before, it might have to do with my modifications. This time I didn't modify anything on-disk, this is purely from the CeroPackages feed). I did try to get Kernel 4.1 to compile but that didn't work even though I removed some packages that didn't compile, I ended up with no .dts file and nothing to me obvious in scrollback to fix. So this is with 3.18. Here are the cake 50M and 500M results and output from the router: oot@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option enabled '1' option script 'simple.qos' option qdisc 'cake' option linklayer_adaptation_mechanism 'cake' option download '50' option upload '50' root@OpenWrt:~# tc -d qdisc qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows noatm overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 500Mbit diffserv4 flows noatm overhead 42 These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script. http://swm.pp.se/aqm/rrul_150629-cake-4.tar Then I re-did the test that Dave asked before, I set the wan port to 100 megabit/s in my switch, and removed the SQM. It resulted in the following config: root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option script 'simple.qos' option qdisc 'cake' option linklayer_adaptation_mechanism 'cake' option download '5' option upload '5' option enabled '0' root@OpenWrt:~# tc -d qdisc qdisc mq 0: dev eth0 root qdisc fq_codel 0: dev eth0 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth0 parent :7 limit 1024p flows 1024 quantum 300
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
I'd also like to be able to try it out on CPE hardware. However, what I've got is a Buffalo H300N, so I'll need build instructions (preferably starting from an existing stock build) as well as setup. The Buffalo isn't as powerful as some others, being based around a 34K core. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller moell...@gmx.de wrote:
HI Mikael,, hi Jonathan,
[...]
These are the results from 50M and 500M, also including 50up and 50down that
I added to my test suite script.
http://swm.pp.se/aqm/rrul_150629-cake-4.tar
Now both ingress and egress are up to roughly 455Mbps from roughly 360 with
cake just playing leaf qdisc for HTB. This looks even better than before…
350 *usec* induced delay on the 50mbit rrul_be test. w00t!
Most of the tests compare well with the reference rangeley data now. I
would like a 900mbit soft shaped result.
1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
on that path (fq or fq_codel on the server and client?)
cake currently peels at 1ms / flows (more or less)... NAPI is an
issue... hw mq an issue...
There are a half dozen things in the mvneta driver I would try to
reduce it's latency more. The easy ones:
reduce this to 16:
netif_napi_add(dev, pp-napi, mvneta_poll, NAPI_POLL_WEIGHT);
Reduce this to 24: (this will also reduce the max outstanding stuff in
the driver by a LOT, but is still not BQL!)
/* Max number of allowed TCP segments for software TSO */
#define MVNETA_MAX_TSO_SEGS 100
Both of the will improve read side latency at the cost of more sirqs.
I do not know what reducing these will do, and would test both of the
above separately.
/* Coalescing */
#define MVNETA_TXDONE_COAL_PKTS 1
#define MVNETA_RX_COAL_PKTS 32
#define MVNETA_RX_COAL_USEC 100
As for cake itself, eric dumazet told us we dont need atomic ops in it,
and peeling at at even lower threshold has some appeal (to me, anyway)
attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
(bump up the makefile rel number also, if you want)
Best Regards
Sebastian
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast
From 46be609e95474e9db856b5e12756d4a7568adf42 Mon Sep 17 00:00:00 2001
From: Dave Taht dave.t...@bufferbloat.net
Date: Mon, 29 Jun 2015 09:38:00 -0700
Subject: [PATCH] Rid unneeded atomic ops and reduce peeling threshold
---
sch_cake.c | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/sch_cake.c b/sch_cake.c
index 80e1cb2..9a358b9 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -121,7 +121,7 @@ struct cake_fqcd_sched_data {
struct codel_params cparams;
u32 drop_overlimit;
-atomic_t flow_count;
+u32 flow_count;
struct list_head new_flows; /* list of new flows */
struct list_head old_flows; /* list of old flows */
@@ -427,7 +427,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
* Split GSO aggregates if they're likely to impair flow isolation
* or if we need to know individual packet sizes for framing overhead.
*/
- if(unlikely((len * max(atomic_read(fqcd-flow_count), 1)) q-peel_threshold skb_is_gso(skb)))
+ if(unlikely((len * max(fqcd-flow_count, 1)) q-peel_threshold skb_is_gso(skb)))
{
struct sk_buff *segs, *nskb;
netdev_features_t features = netif_skb_features(skb);
@@ -477,7 +477,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
/* flowchain */
if(list_empty(flow-flowchain)) {
list_add_tail(flow-flowchain, fqcd-new_flows);
- atomic_inc(fqcd-flow_count);
+ fqcd-flow_count+=1;
flow-deficit = fqcd-quantum;
flow-dropped = 0;
}
@@ -615,7 +615,7 @@ retry:
list_move_tail(flow-flowchain, fqcd-old_flows);
} else {
list_del_init(flow-flowchain);
- atomic_dec(fqcd-flow_count);
+ fqcd-flow_count-=1;
}
goto begin;
}
@@ -966,7 +966,7 @@ static void cake_reconfigure(struct Qdisc *sch)
if(q-buffer_limit 65536)
q-buffer_limit = 65536;
- q-peel_threshold = (q-rate_flags CAKE_FLAG_ATM) ? 0 : min(65535U, q-rate_bps 10);
+ q-peel_threshold = (q-rate_flags CAKE_FLAG_ATM) ? 0 : min(65535U, q-rate_bps 12);
} else {
q-buffer_limit = 1 20;
q-peel_threshold = 0;
@@ -1083,7 +1083,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt)
fqcd-perturbation = prandom_u32();
INIT_LIST_HEAD(fqcd-new_flows);
INIT_LIST_HEAD(fqcd-old_flows);
- atomic_set(fqcd-flow_count, 0);
+ fqcd-flow_count = 0;
/* codel_params_init(fqcd-cparams); */
fqcd-flows= cake_zalloc(fqcd-flows_cnt * sizeof(struct cake_fqcd_flow));
--
1.9.1
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi List,
On Jun 29, 2015, at 18:44 , Dave Taht dave.t...@gmail.com wrote:
On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller moell...@gmx.de wrote:
HI Mikael,, hi Jonathan,
[...]
These are the results from 50M and 500M, also including 50up and 50down
that I added to my test suite script.
http://swm.pp.se/aqm/rrul_150629-cake-4.tar
Now both ingress and egress are up to roughly 455Mbps from roughly 360 with
cake just playing leaf qdisc for HTB. This looks even better than before…
350 *usec* induced delay on the 50mbit rrul_be test. w00t!
Most of the tests compare well with the reference rangeley data now. I
would like a 900mbit soft shaped result.
Make sure to use the correct per packet overhead of 18 + 20, on gigabit
ethernet inter-frame gap and preamble cost 20 Bytes worth of data. So with a
MTU of 1500 thee is no issue (900 * (1538/1518) = 911.85770751 Mbps) but the
smaller the packets get:
900 * (MTU+38/MTU+18) = 1000
(MTU+38/MTU+18) = (1000 / 900)
MTU+38 = (10 / 9) * (MTU + 18)
MTU + 38 = (10/9) * MTU + (10/9)*18
38 - (10/9)*18 = (10/9) * MTU - (9/9)MTU
38 - (10/9)*18 = (1/9) MTU
MTU = (38 - ((10/9)*18))*9 = 162
So for TCP/IPv4 MSS 122 the shaper will not keep the ethernet hardware queues
empty…
On the other hand shaping at
1000/(88/64) = 727.272727273 Mbps
should make sure that even at minimal packet size of 64byte shaping would still
be keeping the ethernet queues “empty-ish”. If the 1200ac can shape at 900 I
would rather specify the correct overhead though.
To make things a bit trickier, depending on the interface used the kernel will
already account for the standards ethernet header without the frame check
sequence, so I would guess in the 900Mbps soft shaper on ethN scenario one
would need to add a per packet overhead of 24 bytes. If someone in the know
could double check that reasoning I would be much obliged…
Best Regards
Sebastian
1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
on that path (fq or fq_codel on the server and client?)
cake currently peels at 1ms / flows (more or less)... NAPI is an
issue... hw mq an issue...
There are a half dozen things in the mvneta driver I would try to
reduce it's latency more. The easy ones:
reduce this to 16:
netif_napi_add(dev, pp-napi, mvneta_poll, NAPI_POLL_WEIGHT);
Reduce this to 24: (this will also reduce the max outstanding stuff in
the driver by a LOT, but is still not BQL!)
/* Max number of allowed TCP segments for software TSO */
#define MVNETA_MAX_TSO_SEGS 100
Both of the will improve read side latency at the cost of more sirqs.
I do not know what reducing these will do, and would test both of the
above separately.
/* Coalescing */
#define MVNETA_TXDONE_COAL_PKTS 1
#define MVNETA_RX_COAL_PKTS 32
#define MVNETA_RX_COAL_USEC 100
As for cake itself, eric dumazet told us we dont need atomic ops in it,
and peeling at at even lower threshold has some appeal (to me, anyway)
attached is a patch for that, put it in your
feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
(bump up the makefile rel number also, if you want)
Best Regards
Sebastian
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
--
Dave Täht
worldwide bufferbloat report:
http://www.dslreports.com/speedtest/results/bufferbloat
And:
What will it take to vastly improve wifi for everyone?
https://plus.google.com/u/0/explore/makewififast
From 46be609e95474e9db856b5e12756d4a7568adf42 Mon Sep 17 00:00:00 2001
From: Dave Taht dave.t...@bufferbloat.net
Date: Mon, 29 Jun 2015 09:38:00 -0700
Subject: [PATCH] Rid unneeded atomic ops and reduce peeling threshold
---
sch_cake.c | 12 ++--
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/sch_cake.c b/sch_cake.c
index 80e1cb2..9a358b9 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -121,7 +121,7 @@ struct cake_fqcd_sched_data {
struct codel_params cparams;
u32 drop_overlimit;
-atomic_t flow_count;
+u32 flow_count;
struct list_head new_flows; /* list of new flows */
struct list_head old_flows; /* list of old flows */
@@ -427,7 +427,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
* Split GSO aggregates if they're likely to impair flow isolation
* or if we need to know individual packet sizes for framing overhead.
*/
- if(unlikely((len * max(atomic_read(fqcd-flow_count), 1)) q-peel_threshold skb_is_gso(skb)))
+ if(unlikely((len * max(fqcd-flow_count, 1)) q-peel_threshold skb_is_gso(skb)))
{
struct sk_buff *segs, *nskb;
netdev_features_t features = netif_skb_features(skb);
@@ -477,7 +477,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
/* flowchain */
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Dave Taht wrote: Most of the tests compare well with the reference rangeley data now. I would like a 900mbit soft shaped result. http://swm.pp.se/aqm/rrul_150629-cake-9.tar Above is with cake as link layer adaptation and pie, fq_codel and cake as 900 meg bidirectional shaped. Then I removed the ll adaptation and changed to simplest.qos and ran another test with cake: http://swm.pp.se/aqm/rrul_150629-cake-10.tar Now I'm going to try out if my new version with your cake patch will boot and work. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Dave Taht wrote:
attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
Test results: http://swm.pp.se/aqm/rrul_150629-cake-11.tar
--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Sebastian Moeller wrote: Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism). Ok, so when I changed stab to cake, I get the following, but it still only shapes one way. If I change to discipline to fq_codel or pie I get bidirectonal shaping with otherwise same settings. root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option download '5' option upload '5' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option linklayer_adaptation_mechanism 'cake' option enabled '1' option qdisc 'cake' option script 'simple.qos' root@OpenWrt:~# tc -d qdisc qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn root@OpenWrt:~# tc -d class show dev eth0 class cake 8002:4e1 parent 8002: class cake 8002:b89 parent 8002: root@OpenWrt:~# tc -d class show dev ifb4eth0 class fq_codel :12c parent none class fq_codel :222 parent none root@OpenWrt:~# /etc/init.d/sqm restart SQM: Trying to start/stop SQM on all interfaces. SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0 SQM: ifb associated with interface eth0: SQM: trying to create new IFB: ifb4eth0 RTNETLINK answers: File exists SQM: /usr/lib/sqm/stop.sh: Stopping eth0 SQM: ifb associated with interface eth0: SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos + . /usr/lib/sqm/functions.sh + [ -z 5 ] + [ -z 5 ] + [ -z eth0 ] + [ -z cake ] + [ -z cake ] + [ -z ethernet ] + [ -z 42 ] + [ -z 2047 ] + [ -z 0 ] + [ -z 128 ] + [ -z ] + AUTOFLOW=0 + [ -z ] + LIMIT=1001 + [ -z ] + ILIMIT= + [ -z ] + ELIMIT= + [ -z ] + ITARGET= + [ -z ] + ETARGET= + [ -z ECN ] + [ -z ECN ] + [ -z 0 ] + [ -z 0 ] + [ -z ] + IQDISC_OPTS= + [ -z ] + EQDISC_OPTS= + [ -z ] + which tc + TC=/usr/sbin/tc + [ -z ] + which ip + IP=/usr/sbin/ip + [ -z ] + which insmod + INSMOD=/usr/sbin/insmod + [ -z ] + TARGET=5ms + [ -z ] + IPT_MASK=0xff + [ -z ] + IPT_MASK_STRING=/0xff + [ -z ] + get_ifb_for_if eth0 + CUR_IF=eth0 + get_ifb_associated_with_if eth0 + CUR_IF=eth0 + tc+ -p filtergrep -o -e ifb[^)]\+ show parent : dev eth0 + CUR_IFB= + sqm_logger ifb associated with interface eth0: + logger -t SQM -s ifb associated with interface eth0: SQM: ifb associated with interface eth0: + echo + CUR_IFB= + [ -z ] + create_new_ifb_for_if eth0 + CUR_IF=eth0 + MAX_IF_NAME_LENGTH=15 + IFB_PREFIX=ifb4 + NEW_IFB=ifb4eth0 + IFB_NAME_LENGTH=8 + [ 8 -gt 15 ] + sqm_logger trying to create new IFB: ifb4eth0 + logger -t SQM -s trying to create new IFB: ifb4eth0 SQM: trying to create new IFB: ifb4eth0 + /usr/sbin/ip link add name ifb4eth0 type ifb RTNETLINK answers: File exists + echo ifb4eth0 + CUR_IFB=ifb4eth0 + [ -z ifb4eth0 ] + echo ifb4eth0 + DEV=ifb4eth0 + do_modules + insmod act_ipt + lsmod+ grep -q ^act_ipt + insmod sch_cake + + grep -q ^sch_cake lsmod + insmod sch_ingress + + greplsmod -q ^sch_ingress + insmod act_mirred + + greplsmod -q ^act_mirred + insmod cls_fw + + grep -q ^cls_fw lsmod + insmod sch_htb + + lsmodgrep -q ^sch_htb + ipt_setup + ipt -t mangle -N QOS_MARK_eth0 + echo -t mangle -N QOS_MARK_eth0 + sed s/-A/-D/g + d=-t mangle -N QOS_MARK_eth0 + [ -t mangle -N QOS_MARK_eth0 != -t mangle -N QOS_MARK_eth0 ] + echo -t mangle -N QOS_MARK_eth0 + sed s/-I/-D/g + d=-t mangle -N QOS_MARK_eth0 + [ -t
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
HI Mikael,
On Jun 29, 2015, at 09:54 , Mikael Abrahamsson swm...@swm.pp.se wrote:
On Mon, 29 Jun 2015, Sebastian Moeller wrote:
Good work-around; not a real solution as sqm_logger() will also add
SQM:”. I think the solution most likely is to remove line 155 completely.
But weirdly this seems to work with cerowrt…
Correction, calling logger with an empty string works on cero (as well
as on openwrt), but passing an empty string to sqm_logger results in the
observed bug, so I can reproduce this locally.
May I suggest a wrapper around the logger part that handles this case, ie if
the string is empty? Because when the string is empty and it gets stuck, luci
just waits and waits and waits. I mean, this specific case can be fixed, but
for the future it would be more robust if this error case can't happen at all.
--
Mikael Abrahamssonemail: swm...@swm.pp.se
I had a look at sqm_logger():
sqm_logger() {
logger -t SQM -s ${1}
}
and hope that
sqm_logger() {
logger -t SQM -s ${1}”
}
Fixes the issue for me and is more concise than trying to introduce real error
handling in sqm_logger. Does this also work for you?
Best Regards
Sebastian
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Sebastian Moeller wrote: Good work-around; not a real solution as sqm_logger() will also add SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt… May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all. -- Mikael Abrahamssonemail: swm...@swm.pp.se___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
HI Mikael, On Jun 29, 2015, at 10:09 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Mon, 29 Jun 2015, Sebastian Moeller wrote: Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism). Ok, so when I changed stab to cake, I get the following, but it still only shapes one way. If I change to discipline to fq_codel or pie I get bidirectonal shaping with otherwise same settings. Probabkr breakage I introduced... root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option download '5' option upload '5' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU ‘0' we can ignore the last 3 savely. option linklayer_adaptation_mechanism 'cake' option enabled '1' option qdisc 'cake' option script 'simple.qos’ Okay, it really is trying cake’s LLA now. root@OpenWrt:~# tc -d qdisc qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42 And that works on egress, excellent so bot sch_cake and tc are recent enough. qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn Oops, ingress is not right, yet. root@OpenWrt:~# tc -d class show dev eth0 class cake 8002:4e1 parent 8002: class cake 8002:b89 parent 8002: root@OpenWrt:~# tc -d class show dev ifb4eth0 class fq_codel :12c parent none class fq_codel :222 parent none root@OpenWrt:~# /etc/init.d/sqm restart SQM: Trying to start/stop SQM on all interfaces. SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0 SQM: ifb associated with interface eth0: SQM: trying to create new IFB: ifb4eth0 RTNETLINK answers: File exists SQM: /usr/lib/sqm/stop.sh: Stopping eth0 SQM: ifb associated with interface eth0: SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos + . /usr/lib/sqm/functions.sh + [ -z 5 ] + [ -z 5 ] + [ -z eth0 ] + [ -z cake ] + [ -z cake ] + [ -z ethernet ] + [ -z 42 ] + [ -z 2047 ] + [ -z 0 ] + [ -z 128 ] + [ -z ] + AUTOFLOW=0 + [ -z ] + LIMIT=1001 + [ -z ] + ILIMIT= + [ -z ] + ELIMIT= + [ -z ] + ITARGET= + [ -z ] + ETARGET= + [ -z ECN ] + [ -z ECN ] + [ -z 0 ] + [ -z 0 ] + [ -z ] + IQDISC_OPTS= + [ -z ] + EQDISC_OPTS= + [ -z ] + which tc + TC=/usr/sbin/tc + [ -z ] + which ip + IP=/usr/sbin/ip + [ -z ] + which insmod + INSMOD=/usr/sbin/insmod + [ -z ] + TARGET=5ms + [ -z ] + IPT_MASK=0xff + [ -z ] + IPT_MASK_STRING=/0xff + [ -z ] + get_ifb_for_if eth0 + CUR_IF=eth0 + get_ifb_associated_with_if eth0 + CUR_IF=eth0 + tc+ -p filtergrep -o -e ifb[^)]\+ show parent : dev eth0 + CUR_IFB= + sqm_logger ifb associated with interface eth0: + logger -t SQM -s ifb associated with interface eth0: SQM: ifb associated with interface eth0: + echo + CUR_IFB= + [ -z ] + create_new_ifb_for_if eth0 + CUR_IF=eth0 + MAX_IF_NAME_LENGTH=15 + IFB_PREFIX=ifb4 + NEW_IFB=ifb4eth0 + IFB_NAME_LENGTH=8 + [ 8 -gt 15 ] + sqm_logger trying to create new IFB: ifb4eth0 + logger -t SQM -s trying to create new IFB: ifb4eth0 SQM: trying to create new IFB: ifb4eth0 + /usr/sbin/ip link add name ifb4eth0 type ifb RTNETLINK answers: File exists + echo ifb4eth0 + CUR_IFB=ifb4eth0 + [ -z ifb4eth0 ] + echo ifb4eth0 + DEV=ifb4eth0 + do_modules + insmod act_ipt + lsmod+ grep -q ^act_ipt + insmod sch_cake + + grep -q ^sch_cake
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Mon, 29 Jun 2015, Sebastian Moeller wrote:
and hope that
sqm_logger() {
logger -t SQM -s ${1}”
}
Fixes the issue for me and is more concise than trying to introduce real error
handling in sqm_logger. Does this also work for you?
Yes, I reverted my previous change, reproduced the hang, then changed the
above, which does not hang. Seems like the most elegant way to solve this!
Thanks!
--
Mikael Abrahamssonemail: swm...@swm.pp.se___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Jun 29, 2015, at 10:17 , Mikael Abrahamsson swm...@swm.pp.se wrote:
On Mon, 29 Jun 2015, Sebastian Moeller wrote:
and hope that
sqm_logger() {
logger -t SQM -s ${1}”
}
Fixes the issue for me and is more concise than trying to introduce real
error handling in sqm_logger. Does this also work for you?
Yes, I reverted my previous change, reproduced the hang, then changed the
above, which does not hang. Seems like the most elegant way to solve this!
Thanks for finding, reporting, debugging, fixing, and testing this.
This change is committed to the ceropackages 3.10 repository.
Best Regards
Sebastian
Thanks!
--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Sun, 28 Jun 2015, Sebastian Moeller wrote: After the modification, the installed policy looks better, I am now not getting any drops in iperf3 (=ECN is working). Please see link to test results. From my bw monitoring script, it looks like I am receiving more than 500 megabit/s though, even though sending is spot on at 500 megabit/s. Yeah, the receiving limiter isn't working at all, I see now in rrul_be that I am getting full gige speed download. root@OpenWrt:~# tc -d qdisc qdisc cake 8005: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows raw linklayer ethernet overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn root@OpenWrt:~# tc -d class show dev eth0 class cake 8005:6 parent 8005: class cake 8005:44b parent 8005: class cake 8005:50c parent 8005: class cake 8005:5fc parent 8005: class cake 8005:716 parent 8005: class cake 8005:7b3 parent 8005: class cake 8005:b43 parent 8005: class cake 8005:c7f parent 8005: class cake 8005:c87 parent 8005: class cake 8005:d8d parent 8005: root@OpenWrt:~# tc -d class show dev ifb4eth0 class fq_codel :32c parent none I have put the results at http://swm.pp.se/aqm/rrul_150629-cake-1.tar I also have a small shell script that prints out pps and bw once per second that I run on my ubuntu machine, just to see approx how much traffic is being used. Might be useful for someone. http://swm.pp.se/aqm/measurebw.sh -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
HI Mikael, On Jun 29, 2015, at 09:54 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Mon, 29 Jun 2015, Sebastian Moeller wrote: Good work-around; not a real solution as sqm_logger() will also add SQM:”. I think the solution most likely is to remove line 155 completely. But weirdly this seems to work with cerowrt… May I suggest a wrapper around the logger part that handles this case, ie if the string is empty? Because when the string is empty and it gets stuck, luci just waits and waits and waits. I mean, this specific case can be fixed, but for the future it would be more robust if this error case can't happen at all. Sounds reasonably defensive; what irk me is that in my testbed that specific error does not crop up; one more reason to code defensive I guess… Best Regards Sebastian -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, since coke overhead seems to work, you can switch back to link layer none or set the overhead to 0 again. Thanks for testing this, which helps getting sqm-scripts into better shape, but will not be helpful fur your tests. In case you want to try fancy options for cake the advances qdisc option strings in the queue discipline tab should work (caveat no error checking so one typo and the qdisc will most likely not be set-up properly). Best Regards Sebastian On Jun 29, 2015, at 10:09 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Mon, 29 Jun 2015, Sebastian Moeller wrote: Ah, I see, you are still using tc’s stab mechanism for account of per packet overhead and link layer adjustments instead of cake’s (you need to check the advanced options check box in the link layer adjustments tab, and then select “cake” as the lnk layer adjustment mechanism). Ok, so when I changed stab to cake, I get the following, but it still only shapes one way. If I change to discipline to fq_codel or pie I get bidirectonal shaping with otherwise same settings. root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option download '5' option upload '5' option linklayer 'ethernet' option overhead '42' option linklayer_advanced '1' option tcMTU '2047' option tcTSIZE '128' option tcMPU '0' option linklayer_adaptation_mechanism 'cake' option enabled '1' option qdisc 'cake' option script 'simple.qos' root@OpenWrt:~# tc -d qdisc qdisc cake 8002: dev eth0 root refcnt 9 bandwidth 50Mbit diffserv4 flows noatm overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn root@OpenWrt:~# tc -d class show dev eth0 class cake 8002:4e1 parent 8002: class cake 8002:b89 parent 8002: root@OpenWrt:~# tc -d class show dev ifb4eth0 class fq_codel :12c parent none class fq_codel :222 parent none root@OpenWrt:~# /etc/init.d/sqm restart SQM: Trying to start/stop SQM on all interfaces. SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0 SQM: ifb associated with interface eth0: SQM: trying to create new IFB: ifb4eth0 RTNETLINK answers: File exists SQM: /usr/lib/sqm/stop.sh: Stopping eth0 SQM: ifb associated with interface eth0: SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos + . /usr/lib/sqm/functions.sh + [ -z 5 ] + [ -z 5 ] + [ -z eth0 ] + [ -z cake ] + [ -z cake ] + [ -z ethernet ] + [ -z 42 ] + [ -z 2047 ] + [ -z 0 ] + [ -z 128 ] + [ -z ] + AUTOFLOW=0 + [ -z ] + LIMIT=1001 + [ -z ] + ILIMIT= + [ -z ] + ELIMIT= + [ -z ] + ITARGET= + [ -z ] + ETARGET= + [ -z ECN ] + [ -z ECN ] + [ -z 0 ] + [ -z 0 ] + [ -z ] + IQDISC_OPTS= + [ -z ] + EQDISC_OPTS= + [ -z ] + which tc + TC=/usr/sbin/tc + [ -z ] + which ip + IP=/usr/sbin/ip + [ -z ] + which insmod + INSMOD=/usr/sbin/insmod + [ -z ] + TARGET=5ms + [ -z ] + IPT_MASK=0xff + [ -z ] + IPT_MASK_STRING=/0xff + [ -z ] + get_ifb_for_if eth0 + CUR_IF=eth0 + get_ifb_associated_with_if eth0 + CUR_IF=eth0 + tc+ -p filtergrep -o -e ifb[^)]\+ show parent : dev eth0 + CUR_IFB= + sqm_logger ifb associated with interface eth0: + logger -t SQM -s ifb associated with interface eth0: SQM: ifb associated with interface eth0: + echo + CUR_IFB= + [ -z ] + create_new_ifb_for_if eth0 + CUR_IF=eth0 + MAX_IF_NAME_LENGTH=15 + IFB_PREFIX=ifb4 + NEW_IFB=ifb4eth0 + IFB_NAME_LENGTH=8 + [ 8 -gt 15 ] + sqm_logger trying to create new IFB: ifb4eth0 + logger -t SQM -s trying to create new IFB: ifb4eth0 SQM: trying to create new IFB: ifb4eth0 + /usr/sbin/ip link add name ifb4eth0 type ifb RTNETLINK
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, On Jun 28, 2015, at 09:06 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: src-git https://github.com/dtaht/ceropackages-3.10.git ./scripts feeds update ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie edit the .config file to make them modules or installed by default make menuconfig then save make I have now done this. I have sch_cake, and I have a tc I can add an qdisc cake with to for instance eth1 and see stats from it with tc -s qdisc I however do not have anything cake in luci-app-sqm. I see luci-app-sqm in my cero feed, but it seems to install luci-app-sqm from regular OpenWrt instead. Ah, I see the latest changes have not yet made it into the openwrt repository (due to lack of testing). As I do not have permissions/authority to push changes into the openwrt repository, there is nothing I can do to expedite the process. You could replace /usr/lib/lua/luci/model/cbi/sqm.lua on your router with the following (attached file) sqm.lua Description: Binary data that should do the trick and get you to the most recent version that needs testers badly. The minimal change required is to add c:value(“cake”) to the following section of sqm.lua: c = s:taboption(tab_qdisc, ListValue, qdisc, translate(Queueing discipline)) c:value(fq_codel, fq_codel (..translate(default)..)) c:value(efq_codel) c:value(nfq_codel) c:value(sfq) c:value(codel) c:value(ns2_codel) c:value(pie) c:value(sfq”) c:value(“cake) c.default = fq_codel c.rmempty = false @Dave, which of these do we want to keep carrying and which can we safely retire? Especially in the openwrt context most of the experimental ones are not available anyways. (If anybody has a way to make tc or the kernel enumerate the qdiscs that are either compiled in or available as modules that would be much appreciated.) Best Regards Sebastian So either if someone has any tips on how to fix this, or can just give me how to configure qdiscs manually, I will be able to do cake testing on the WRT1200AC. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: src-git https://github.com/dtaht/ceropackages-3.10.git ./scripts feeds update ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie edit the .config file to make them modules or installed by default make menuconfig then save make I have now done this. I have sch_cake, and I have a tc I can add an qdisc cake with to for instance eth1 and see stats from it with tc -s qdisc I however do not have anything cake in luci-app-sqm. I see luci-app-sqm in my cero feed, but it seems to install luci-app-sqm from regular OpenWrt instead. So either if someone has any tips on how to fix this, or can just give me how to configure qdiscs manually, I will be able to do cake testing on the WRT1200AC. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Sun, 28 Jun 2015, Sebastian Moeller wrote: Hi Mikael, root@OpenWrt:~# tc -d qdisc qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532 qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32 qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024 quantum 500 target 5.0ms interval 100.0ms ecn qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn then I change to cake and add 42 as overhead: root@OpenWrt:~# tc -d qdisc qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows raw linklayer ethernet overhead 42 qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn root@OpenWrt:~# tc -d class show dev eth0 class cake 8003:508 parent 8003: class cake 8003:944 parent 8003: root@OpenWrt:~# tc -d class show dev ifb4eth0 root@OpenWrt:~# -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, On Jun 28, 2015, at 22:48 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Sun, 28 Jun 2015, Sebastian Moeller wrote: Hi Mikael, root@OpenWrt:~# tc -d qdisc qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532 qdisc fq_codel 110: dev eth0 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 120: dev eth0 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 130: dev eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32 qdisc fq_codel 110: dev ifb4eth0 parent 1:11 limit 1001p flows 1024 quantum 500 target 5.0ms interval 100.0ms ecn qdisc fq_codel 120: dev ifb4eth0 parent 1:12 limit 1001p flows 1024 quantum 1500 target 5.0ms interval 100.0ms ecn qdisc fq_codel 130: dev ifb4eth0 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn then I change to cake and add 42 as overhead: root@OpenWrt:~# tc -d qdisc qdisc cake 8003: dev eth0 root refcnt 9 bandwidth 500Mbit diffserv4 flows raw Cake still did not take the overhead argument (or raw would have gone from the output) linklayer ethernet overhead 42 This looks a bit like the output I would expect from using tic’s stab mechanism to define link layer adjustments and per packet overhead. Could you post the result of: cat /etc/config/sqm from the router, please? And the output of /etc/init.d/sqm stop ; /etc/init.d/sqm start this might give me a clue where to look... qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev ifb4eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn The ingress setup did not go through, which most likely means tc got unhappy about one of the options. Thanks for your time in testing this, I fear sqm-scripts is not really in a good shape for cake testing yet. Best Regars Sebastian root@OpenWrt:~# tc -d class show dev eth0 class cake 8003:508 parent 8003: class cake 8003:944 parent 8003: root@OpenWrt:~# tc -d class show dev ifb4eth0 root@OpenWrt:~# -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Sun, 28 Jun 2015, Sebastian Moeller wrote: So either I mixed things up later or Mikael’s simple.qos somehow got stale. Anyway, there is work thee for me to do to fix this up properly. I believe this is simple.qos from whatever nightly OpenWRT build there is. It's quite likely it didn't pull any of the sqm-scripts from the CeroWrt feed at all. -- Mikael Abrahamssonemail: swm...@swm.pp.se___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Dave, hi List, On Jun 28, 2015, at 20:04 , Dave Taht dave.t...@gmail.com wrote: htb + cake is the wrong configuration. :) “Wrong” might be a bit hard, but certainly not the preferred solution. During my tests on your box simple.qos resulted in the following: root@davedesk2:/usr/lib/sqm# tc -d qdisc qdisc fq_codel 0: dev eth0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc cake 8030: dev eth1 root refcnt 2 bandwidth 4500Kbit diffserv4 flows raw qdisc ingress : dev eth1 parent :fff1 qdisc cake 8031: dev ifb4eth1 root refcnt 2 bandwidth 45Mbit besteffort flows atm overhead 0 qdisc bfifo 800b: dev wlan1 root refcnt 5 limit 16Kb qdisc cake 8008: dev wlan0 root refcnt 5 unlimited diffserv4 flows raw So either I mixed things up later or Mikael’s simple.qos somehow got stale. Anyway, there is work thee for me to do to fix this up properly. Best Regards Sebastian cake has an integral bandwidth limiter and diffserv, there should be no htb here and it looked like their was. I have only been using the simplest qdisc, possibly the script for the simple qdisc is wrong? On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton chromati...@gmail.com wrote: To be honest, HTB + cake isn't really the preferred configuration. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
To be honest, HTB + cake isn't really the preferred configuration. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
htb + cake is the wrong configuration. :) cake has an integral bandwidth limiter and diffserv, there should be no htb here and it looked like their was. I have only been using the simplest qdisc, possibly the script for the simple qdisc is wrong? On Sun, Jun 28, 2015 at 10:58 AM, Jonathan Morton chromati...@gmail.com wrote: To be honest, HTB + cake isn't really the preferred configuration. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, thanks a lot. On Jun 28, 2015, at 19:32 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Sun, 28 Jun 2015, Sebastian Moeller wrote: This looks great, could you by any chance confirm that the GUI does allow to configure cake and that you can or can not set the overhead for cake in the link layer adjustments (LLA) tab? (select cake as link layer adjustment method, and put 42 into the overhead field and report the output of “tc -d qdisc” before and after selecting cake as LLA). @Toke: If that works, I think we can safely push these changes into the openwrt repositories... Here is the output. What I don't see is both ingress and egress ECN markings even though I have selected this in the advanced configuration under Queue Discipline. Good point, I have not connected these fields with cake yet, will do in the near future. I believe cake defaults to ECN, but if no packets are marked during a test, but rather dropped, something is fishy... Before changing LLA: root@OpenWrt:~# tc -d qdisc qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532 qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32 qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw After changing LLA: root@OpenWrt:~# tc -d qdisc qdisc htb 1: dev eth0 root refcnt 9 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 532 linklayer ethernet overhead 42 qdisc cake 110: dev eth0 parent 1:11 unlimited diffserv4 flows raw qdisc cake 120: dev eth0 parent 1:12 unlimited diffserv4 flows raw qdisc cake 130: dev eth0 parent 1:13 unlimited diffserv4 flows raw qdisc ingress : dev eth0 parent :fff1 qdisc mq 0: dev eth1 root qdisc fq_codel 0: dev eth1 parent :1 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :3 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :4 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :5 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :6 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :7 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc fq_codel 0: dev eth1 parent :8 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn qdisc htb 1: dev ifb4eth0 root refcnt 2 r2q 10 default 12 direct_packets_stat 0 ver 3.17 direct_qlen 32 linklayer ethernet overhead 42 qdisc cake 110: dev ifb4eth0 parent 1:11 unlimited diffserv4 flows raw qdisc cake 120: dev ifb4eth0 parent 1:12 unlimited diffserv4 flows raw qdisc cake 130: dev ifb4eth0 parent 1:13 unlimited diffserv4 flows raw This is showing tha the cake setup went wrong. It is using HTB as shaper instead of cake. There should be no cake line. Could you post your /usr/lib/sqm/simple.qos file please. Best Regards Sebastian -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Sun, 28 Jun 2015, Sebastian Moeller wrote: Could you post the result of: cat /etc/config/sqm from the router, please? And the output of /etc/init.d/sqm stop ; /etc/init.d/sqm start this might give me a clue where to look... root@OpenWrt:~# cat /etc/config/sqm config queue 'eth1' option script 'simple.qos' option interface 'eth0' option qdisc_advanced '1' option squash_dscp '0' option squash_ingress '0' option ingress_ecn 'ECN' option egress_ecn 'ECN' option qdisc_really_really_advanced '0' option enabled '1' option download '50' option upload '50' option qdisc 'cake' option linklayer 'ethernet' option overhead '42' root@OpenWrt:~# /etc/init.d/sqm stop SQM: Trying to start/stop SQM on all interfaces. SQM: run.sh stop SQM: /usr/lib/sqm/run.sh Stopping SQM on interface: eth0 SQM: ifb associated with interface eth0: SQM: trying to create new IFB: ifb4eth0 RTNETLINK answers: File exists SQM: /usr/lib/sqm/stop.sh: Stopping eth0 SQM: ifb associated with interface eth0: SQM: /usr/lib/sqm/run.sh SQM qdiscs on eth0 removed root@OpenWrt:~# /etc/init.d/sqm start SQM: Trying to start/stop SQM on all interfaces. SQM: /usr/lib/sqm/run.sh Queue Setup Script: /usr/lib/sqm/simple.qos SQM: ifb associated with interface eth0: SQM: trying to create new IFB: ifb4eth0 RTNETLINK answers: File exists SQM: cake SQM: Keeping differentiated services code points (DSCP) from ingress. SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet This command never completes. I added -x to simple.qos and the RTNETLINK is from: + /usr/sbin/ip link add name ifb4eth0 type ifb RTNETLINK answers: File exists and the last commands that execute are: SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet + echo stab mtu 2047 tsize 512 mpu 0 overhead 42 linklayer ethernet + get_cake_lla_string + STABSTRING= + [ tc_stab = cake -a ethernet != none ] + sqm_logger + logger -t SQM -s -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 10:50 PM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: your results are showing basically tail drop behavior. Although I would have expected intrinsic delay on the link to crack 100mbits on the rrul test, not 20ms (which is still high), and you only hit 7 on the single threaded tcp up test, based on what I saw in the driver. turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up test. Also do a capture to see if CE was ever exerted. http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz I have a capture of this as well, it's 1.5gigs. It has lots of mentions of ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no such packets. All the packets are 0x2. http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2 Maybe you are dropping at the internal switch? A lot of manufacturers ran everything through the switch, even the uplink, in recent years. This will make things hard to fix. I went back to gig and set SQM to 50 megabit/s bidirectional and looking at tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the others have non-zero ecn_mark. Yea, well, you need to use a diffserv enabled test to see marks or drops in other queues. Sort of my hope is that cake can run at 940mbit with software rate limiting. But we probably need to find ways of parallizing the needed softirq. I do not know why my linksys ac1200 build does not work. I generally suspect it is because my big build server is getting ancient. Can you send me your .config file for openwrt? http://swm.pp.se/aqm/wrt1200ac.config This was my first try, I haven't fine tuned it yet, but it works (boots and moves packets anyway :P ) Must be nice to win on the first try. It would be VERY helpful if you pulled from ceropackages-3.14, and added and built kmod-sched-cake and tc-adv and switched to using the ceropackages feed also for luci-app-sqm and sqm-scripts. There are a couple people here that would probably leap on that! :) I will give it a go in the next few days. -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Sat, 27 Jun 2015, Dave Taht wrote: Maybe you are dropping at the internal switch? A lot of manufacturers ran everything through the switch, even the uplink, in recent years. This will make things hard to fix. Looking at the vlans and pvids it looks like they've connected the SoC to port 5 and 6 on the switch, port 4 is the Internet port, and 0-3 is LAN1-4 port. Sigh, I had hoped at least eth0 (WAN/Internet) was a physically dedicated port. root@OpenWrt:~# swconfig dev switch0 show Global attributes: enable_vlan: 0 Port 0: mask: 0x004e: (0) 1 2 3 6 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 1: mask: 0x004d: 0 (1) 2 3 6 qmode: 0 status: link: down link: 0 pvid: 0 Port 2: mask: 0x004b: 0 1 (2) 3 6 qmode: 0 status: link: down link: 0 pvid: 0 Port 3: mask: 0x0047: 0 1 2 (3) 6 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 4: mask: 0x0020: (4) 5 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 5: mask: 0x0010: 4 (5) qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 6: mask: 0x000f: 0 1 2 3 (6) qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Sigh. It is possible, maybe, to build something using this chipset that does not connect the wan port through the switch, built by someone else. Maybe some other manufacturer did that. I had first evaluated this chipset on the dual port mirabox, which as best as I recall had no switch, two genuine ethernet ports (but it ran WAY too hot and at the time, the kernel was ancient). On Sat, Jun 27, 2015 at 11:23 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Sat, 27 Jun 2015, Dave Taht wrote: Maybe you are dropping at the internal switch? A lot of manufacturers ran everything through the switch, even the uplink, in recent years. This will make things hard to fix. Looking at the vlans and pvids it looks like they've connected the SoC to port 5 and 6 on the switch, port 4 is the Internet port, and 0-3 is LAN1-4 port. Sigh, I had hoped at least eth0 (WAN/Internet) was a physically dedicated port. root@OpenWrt:~# swconfig dev switch0 show Global attributes: enable_vlan: 0 Port 0: mask: 0x004e: (0) 1 2 3 6 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 1: mask: 0x004d: 0 (1) 2 3 6 qmode: 0 status: link: down link: 0 pvid: 0 Port 2: mask: 0x004b: 0 1 (2) 3 6 qmode: 0 status: link: down link: 0 pvid: 0 Port 3: mask: 0x0047: 0 1 2 (3) 6 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 4: mask: 0x0020: (4) 5 qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 5: mask: 0x0010: 4 (5) qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 Port 6: mask: 0x000f: 0 1 2 3 (6) qmode: 0 status: link: up, speed: 1000 Mbps, duplex: full link: 1000 pvid: 0 -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Dave, hi Mikael, On Jun 27, 2015, at 19:59 , Dave Taht dave.t...@gmail.com wrote: [...] Yea, well, you need to use a diffserv enabled test to see marks or drops in other queues. Sort of my hope is that cake can run at 940mbit with software rate limiting. But we probably need to find ways of parallizing the needed softirq. […] If we go that route we need to remember adding the 20 bytes worth of inter frame gap and preamble to the per packet overhead (in addition to the 18 bytes of “normal” ethernet header with frame check sequence), otherwise the shaper will not do what we expect it to at small packet sizes. Best Regards Sebastian ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Thu, 25 Jun 2015, Toke Høiland-Jørgensen wrote: Dave Taht dave.t...@gmail.com writes: Unfortunately the core piece of metadata I wanted from the router was the qdisc statistics. Didnt parse. Will file bug. My guess is that in this case it's due to the openwrt box missing the tc and ip binaries - those are not installed by default... I did have tc, but I didn't have ip. Do you need ip or ip-full? I installed ip. Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses the ip binary to get the ip and route information to know from which interfaces to pull the qdisc information from... We'll see if my next test run includes the correct information then. -- Mikael Abrahamssonemail: swm...@swm.pp.se___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Thu, 25 Jun 2015, Dave Taht wrote: on your host, server, or switch. (you have pfifo_fast on your host) Try fq_codel on host and server (and/or sch_fq) and see what happens. Disable tso/gro/gso on your server/host also. That leaves the switch which I have no insight into. What switch chip is it? (see /etc/config/network) - on the cerowrt project we got less buffering out of the switch by enabling jumbo frames. So the complete physical setup is: Linux laptop cable WRT1200AC cable TP-Link TL-SG3424 L2 switch cable macbook pro thunderbolt port This means there are multiple places buffering can occur, that doesn't have fq_codel. I can't tell what switch chip is being used, it's not on http://techinfodepot.info/wiki/Linksys_WRT1200AC and I don't know where to look to find it. In /etc/config/network there is just br-lan and eth0 and eth1. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 9:36 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Sebastian Moeller wrote: Tangent: What is the shaper rate the wdr4900 can push with sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked like the finest little router platform in the last years, too bad it was ignored by the mass market...) Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70 megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150 megabit/s at 25k PPS through it. This is at 100% sirq. Does this help, or do you want me to do any other tests. I have the WDR4900 powered up and on my desk right now. I am never allergic to somene running a comprehensive flent suite through something, and sticking the results up somewhere. :) There are always more surprises and more things we seem to have to wackamole. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Jonathan Morton wrote: Hypothesis: this might have to do with the receive path. Some devices might have more capacity than others to buffer inbound packets until the CPU can get around to servicing them. Is there a way to tell? I am better at diagnosing Cisco CPU based routers than Linux ones. I looked in /sys/class/net/ethX/statistics but these drops are not recorded there. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 9:18 AM, Jonathan Morton chromati...@gmail.com wrote: Hypothesis: this might have to do with the receive path. Some devices might have more capacity than others to buffer inbound packets until the CPU can get around to servicing them. *Good* hypothesis. I am certain I have seen this on multiple occasions on other hardware. Hard to confirm. Wet paint... so I finally got off my arse and looked at the driver this morning. given that this is a multi-core box, I would lean towards a smaller napi_poll_weight, which unfortunately is a constant (64) in the code. 4 cores can take interrupts faster. (and I hate napi on routers) I have sometimes longed for an IQL (ingress queue limits) to also handle differences in packet size, dynamically changing the poll weight based on load - increasing it for loads with lots of small packets, decreasing it for lots of big packets. Furthermore this thing is doing software gro (up to 64 packets at a time) which is a LOT of processing at this layer in the stack. Its a two line patch to cut the weight to 16, but I have never managed to get a working build for this platform. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, On Jun 26, 2015, at 14:26 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Sebastian Moeller wrote: Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link). I did some more tests and it seems with SQM at 500 megabit/s I start to lose packets at around 250-300k PPS. At these speeds, the rrul_be test is only 85k PPS at 500 megabit/s bidirectional with large packets. Ah, that sounds like there 1200ac is a practical solution today; at 500Mbps there are 500^3/(64*8) = 244 Kpps at the smallest size (since the wire is at 1000 Mbps I think we should and can ignore inter frame gap and preamble, SQM certainly ignores them) so it looks like the router is really close to the theoretical maximum, and with larger packets things get way more relaxed (500^3/(1518*8) = 10 Kpps). Pretty decent for a router using no proprietary offload features. Also, I have an Edgerouter ER-5, but as soon as it does CPU based forwarding it's really weak, easily under 100 megabit/s even with large packets. OpenVPN without encryption is less than 20 megabit/s. Ah, interesting, so neither the affordable edge router lite nor the (similar) ER-5 will cut it unless the offloads are used; I think the newer edgerouterX is rated for slightly higher speeds without offloads, but not nearly close to what your 1200ac does... Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR incl 25% VAT and shipping here in Sweden now. In Germany it still retails for around 200EUR with the cheapest offer at 180EUR (but that is an outlier); guess I should visit Sweden then ;) Btw, I tried WNDR3800 setting it to 100/100 SQM. Yeah, not pretty, huh? It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets? Interesting observation, no idea, but intrigued ;) Best Regards Sebastian -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Mikael Abrahamsson wrote: Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets? I checked again, and my WDR4900 with same setup doesn't lose packets either. Even at 99% sirq, no packets are lost. WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above to avoid packet loss. Does flent check for packet loss at all? Perhaps it's something to look into, because with ECN we really don't want to see any packets lost and this might be good to include test results for. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: your results are showing basically tail drop behavior. Although I would have expected intrinsic delay on the link to crack 100mbits on the rrul test, not 20ms (which is still high), and you only hit 7 on the single threaded tcp up test, based on what I saw in the driver. turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up test. Also do a capture to see if CE was ever exerted. http://swm.pp.se/aqm/rrul_50_up-2015-06-27T072819.166452.iperf3_150626_9_100M.flent.gz I have a capture of this as well, it's 1.5gigs. It has lots of mentions of ECT(0) but 0 mention of ECT(1). TOS should be 0x3 when the EC=1 ? I see no such packets. All the packets are 0x2. http://swm.pp.se/aqm/rrul_50up-100M.pcap.bz2 I went back to gig and set SQM to 50 megabit/s bidirectional and looking at tc qdisc -s eth0 I see ecn_mark 10339 on a single queue, none of the others have non-zero ecn_mark. I do not know why my linksys ac1200 build does not work. I generally suspect it is because my big build server is getting ancient. Can you send me your .config file for openwrt? http://swm.pp.se/aqm/wrt1200ac.config This was my first try, I haven't fine tuned it yet, but it works (boots and moves packets anyway :P ) It would be VERY helpful if you pulled from ceropackages-3.14, and added and built kmod-sched-cake and tc-adv and switched to using the ceropackages feed also for luci-app-sqm and sqm-scripts. There are a couple people here that would probably leap on that! :) I will give it a go in the next few days. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: Mikael, a simple test of the analysis I just did would be to use ethtool to set your server to 100mbits (ethtool -s your_ethernet_device advertise 0x008 and turn on fq_codel on both the client and server. Hm what do you mean by client and server? Where do you want the queueing to happen? Egress from the WRT1200AC towards the server? Then setting the WAN port of the WRT1200AC to 100 megabit/s would work, yes. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 9:35 AM, Jonathan Morton chromati...@gmail.com wrote: These would be hardware tail drops - there might not be a physical counter recording them. But you could instrument three driver to see whether the receive buffer is full when serviced. from drivers/net/ethernet/marvel/mvneta.c: /* Max number of Rx descriptors */ #define MVNETA_MAX_RXD 128 this is probably too small, especially given the 64 it is willing to wait for. At the same time, it is too large, as there are 8 hardware queues in play here. So you get a huge burst from one flow, it gros it all together aggghh... /* Max number of Tx descriptors */ #define MVNETA_MAX_TXD 532 this really needs BQL. Same problem(s). Only worse. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: I am never allergic to somene running a comprehensive flent suite through something, and sticking the results up somewhere. http://swm.pp.se/aqm/wdr4900-150626-9.tar Happy no sneezing! -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
HI Mikael, On Jun 26, 2015, at 16:49 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Mikael Abrahamsson wrote: Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets? I checked again, and my WDR4900 with same setup doesn't lose packets either. Even at 99% sirq, no packets are lost. Tangent: What is the shaper rate the wdr4900 can push with sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked like the finest little router platform in the last years, too bad it was ignored by the mass market...) WRT1200AC starts to lose packets at 500 megabit/s SQM around MSS 300 and lower. If I turn off gso, tso and gro, I have to go to MSS 600 and above to avoid packet loss. So the offloads buy roughly a doubling of the achievable packet rate, not bad, but as your results show also knot essential. Best Regards Sebastian Does flent check for packet loss at all? Perhaps it's something to look into, because with ECN we really don't want to see any packets lost and this might be good to include test results for. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Sebastian Moeller wrote: Tangent: What is the shaper rate the wdr4900 can push with sqm-scripts? (Before your 1200ac results the ppc-soc in the wdr4900 looked like the finest little router platform in the last years, too bad it was ignored by the mass market...) Well, if I set SQM to 500 megabit/s and MSS to 300 I am able to get 70 megabit/s of iperf3 through it at 42k PPS. At default MSS I get 150 megabit/s at 25k PPS through it. This is at 100% sirq. Does this help, or do you want me to do any other tests. I have the WDR4900 powered up and on my desk right now. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
These would be hardware tail drops - there might not be a physical counter recording them. But you could instrument three driver to see whether the receive buffer is full when serviced. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: Mikael, a simple test of the analysis I just did would be to use ethtool to set your server to 100mbits (ethtool -s your_ethernet_device advertise 0x008 and turn on fq_codel on both the client and server. Hm what do you mean by client and server? Where do you want the queueing to happen? Egress from the WRT1200AC towards the server? Yes. Then setting the WAN port of the WRT1200AC to 100 megabit/s would work, yes. Yes, but I am unsure from looking at the driver that using ethtool on the egress on the wrt1200ac will actually work, but pretty sure it will work if you set it on the server. feel free to try both. :) -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht dave.t...@gmail.com wrote: On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: Mikael, a simple test of the analysis I just did would be to use ethtool to set your server to 100mbits (ethtool -s your_ethernet_device advertise 0x008 and turn on fq_codel on both the client and server. Hm what do you mean by client and server? your topology is: client box - router - server forcing the router - server link to 100mbit will push the egress buffering into the router for the rrul_50_up test in particular. No, my topology is client - router - switch - server, that is what made me confused. So if I forced the eth0 router-switch link into 100M I will break the server-client direction (it'll be shallow buffer fifo) but at least client-server direction will exercise fq_codel on the router. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht dave.t...@gmail.com wrote: On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: Mikael, a simple test of the analysis I just did would be to use ethtool to set your server to 100mbits (ethtool -s your_ethernet_device advertise 0x008 and turn on fq_codel on both the client and server. Hm what do you mean by client and server? your topology is: client box - router - server forcing the router - server link to 100mbit will push the egress buffering into the router for the rrul_50_up test in particular. Where do you want the queueing to happen? Egress from the WRT1200AC towards the server? Yes. Then setting the WAN port of the WRT1200AC to 100 megabit/s would work, yes. Yes, but I am unsure from looking at the driver that using ethtool on the egress on the wrt1200ac will actually work, but pretty sure it will work if you set it on the server. feel free to try both. :) -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, Jun 26, 2015 at 12:11 PM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: On Fri, Jun 26, 2015 at 11:58 AM, Dave Taht dave.t...@gmail.com wrote: On Fri, Jun 26, 2015 at 11:38 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: Mikael, a simple test of the analysis I just did would be to use ethtool to set your server to 100mbits (ethtool -s your_ethernet_device advertise 0x008 and turn on fq_codel on both the client and server. Hm what do you mean by client and server? your topology is: client box - router - server forcing the router - server link to 100mbit will push the egress buffering into the router for the rrul_50_up test in particular. No, my topology is client - router - switch - server, that is what made me confused. So if I forced the eth0 router-switch link into 100M I will break the server-client direction (it'll be shallow buffer fifo) but at least client-server direction will exercise fq_codel on the router. well, maybe the driver will take ethtool on the mvneta. try it. :) -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, thanks a lot. On Jun 24, 2015, at 13:31 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Tue, 23 Jun 2015, Sebastian Moeller wrote: As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?) What kind of tests should I run? I have rrul results now, both at the same time as running iperf3. This is interesting. I also like the way iperf3 reports retransmits (BTW way how does it know this, I thought the kernel does the retransmits under the hood out of sight of applications). Since RRUL does not have such a nice report as iperf3 I will not be able to calculate the difference to the theoretical maximum transfer rates (also I lack insight on the ACK traffic). I set iperf3 to run with 10 parallel streams, different MSS per test, and let it run for 30 seconds into the rrul test. The plots are interesting, even though I have a hard time with the logarithmic scale. So in all the tests, iperf3 session stops running half way into the rrul test. I like how the RRUL flows soak up the newly available bandwidth. I set sqm to 500M up and down on eth0, ECN up and down, and not to squash DSCP in any direction. I will have a look at the flint files, but it looks like the 1200ac does 500Mbps bidirectionally with SQM, so this looks like a decent machine for the present and (near?) future. Now I just need to wait until prices come down a tad (I hope that the 1900AC v2 release should drop the 1200ac’s process a bit). Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link). Best Regards Sebastian http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf Tell me if there is more information I can provide or tests to run. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Dave Taht wrote: Yes, but I am unsure from looking at the driver that using ethtool on the egress on the wrt1200ac will actually work, but pretty sure it will work if you set it on the server. feel free to try both. :) I set speed 100 on my switch and did some new tests, I left SQM at 500M, don't know what happens then, thought it was worthwile to test? Remember, now I don't have fq_codel in the server-client direction, but there it's an low buffered switchport that is doing the rate adaptation. Btw, now I am running a nightly build that I compiled for myself for the WRT1200AC, so if there is anything you would like me to try to change in the mvneta driver, I can certainly test that. I can do serial console access to the WRT1200AC if needed as well, I already unbricked it once. Btw, disregard the title in the flent files, I just realised I forgot to change the title argument. This is without iperf3, with SQM set to 500M, but eth0 at 100megabit/full duplex. http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
your results are showing basically tail drop behavior. Although I would have expected intrinsic delay on the link to crack 100mbits on the rrul test, not 20ms (which is still high), and you only hit 7 on the single threaded tcp up test, based on what I saw in the driver. turn off sqm, stay at 100mbit, let fq_codel ride, try the rrul_50_up test. Also do a capture to see if CE was ever exerted. I do not know why my linksys ac1200 build does not work. I generally suspect it is because my big build server is getting ancient. Can you send me your .config file for openwrt? It would be VERY helpful if you pulled from ceropackages-3.14, and added and built kmod-sched-cake and tc-adv and switched to using the ceropackages feed also for luci-app-sqm and sqm-scripts. There are a couple people here that would probably leap on that! :) add to your feeds.conf src-git https://github.com/dtaht/ceropackages-3.10.git ./scripts feeds update ./scripts feeds install kmod-sched-cake tc-adv kmod-sched-fq_pie edit the .config file to make them modules or installed by default make menuconfig then save make I went through hell trying to to figure out how to switch over to the cero versions of these. If yer doing the builds, I could try hacking the BQL. Maybe. In sweden. On Fri, Jun 26, 2015 at 10:03 PM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Fri, 26 Jun 2015, Dave Taht wrote: Yes, but I am unsure from looking at the driver that using ethtool on the egress on the wrt1200ac will actually work, but pretty sure it will work if you set it on the server. feel free to try both. :) I set speed 100 on my switch and did some new tests, I left SQM at 500M, don't know what happens then, thought it was worthwile to test? Remember, now I don't have fq_codel in the server-client direction, but there it's an low buffered switchport that is doing the rate adaptation. Btw, now I am running a nightly build that I compiled for myself for the WRT1200AC, so if there is anything you would like me to try to change in the mvneta driver, I can certainly test that. I can do serial console access to the WRT1200AC if needed as well, I already unbricked it once. Btw, disregard the title in the flent files, I just realised I forgot to change the title argument. This is without iperf3, with SQM set to 500M, but eth0 at 100megabit/full duplex. http://swm.pp.se/aqm/wrt1200ac-150627-100m-sqm.tar -- Mikael Abrahamssonemail: swm...@swm.pp.se -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Fri, 26 Jun 2015, Sebastian Moeller wrote: Thanks for the tests, now I know what router to try next (the edgerouterX, which I had eyed as a replacement for the shaper in the wndr3700 tops out at 130K packets per second and hence will not really work that well for a 100/40 Mbps link). I did some more tests and it seems with SQM at 500 megabit/s I start to lose packets at around 250-300k PPS. At these speeds, the rrul_be test is only 85k PPS at 500 megabit/s bidirectional with large packets. Also, I have an Edgerouter ER-5, but as soon as it does CPU based forwarding it's really weak, easily under 100 megabit/s even with large packets. OpenVPN without encryption is less than 20 megabit/s. Btw, the WRT1200AC is now becoming more widely available and it's 150 EUR incl 25% VAT and shipping here in Sweden now. Btw, I tried WNDR3800 setting it to 100/100 SQM. It seems to max out around 25-30k PPS, but the difference is that when the CPU is full, it seems to delay/ECN-mark packets because there are no packets lost. When the WRT1200AC runs out of CPU it starts dropping packets. I always have 0 packets lost with the WNDR3800 when doing iperf3 testing. I found this difference interesting, wonder where in the forwarding path the WRT1200AC loses packets? -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Did you restart sqm after turning off offloads? maxpacket is accumulative and wont change back otherwise. ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Thu, Jun 25, 2015 at 2:12 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Wed, 24 Jun 2015, Dave Taht wrote: From what I see here you are rarely, if ever, engaging fq_codel properly. Latencies are pretty high. In particular, I would suspect you are hitting offloads hard, and the current (fixed in linux 4.1) codel drop algorithm stops dropping below maxpacket, which was meant in the ns2 code to be a MTU, but in the linux code ended up being a TSO sized (64k!) packet. tc -s qdisc show # will show the maxpacket. http://swm.pp.se/aqm/qdisc-show.txt Ugh. 64k maxpacket. I did not even know that was possible until I saw it (GRO is usually for tcp acks, and peaks at 24k or so). I will check to see if openwrt backported that crucial codel patch. Or we can switch sqm simplest to use tbf, or we can do cake´s peeling I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the linux box. All the tests done in http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN working, without iperf3 running, and with and without SQM, but with default offload setting. Also, now iperf3 says 0 packet lost when I use that. cool. :) Your router has it off, too, and if you plan to use it for other things than routing, you might want to turn it on (gleaned from your metadata, thx!) 2) Please run your flent tests with -x --disable-log Done. Use -t title to differentiate between variables under test. Done. 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox to get the stats on that box into the metadata. You have to add your local .ssh/id_rsa.pub key to your_openwrtbox:/etc/dropbear/authorized_keys file to do this. Done. Unfortunately the core piece of metadata I wanted from the router was the qdisc statistics. Didnt parse. Will file bug. 4) With all that in hand, sticking up a tarball of the results makes for easy plotting of various other graphs, and using the flent-gui, you can combine results from each run easily, also. See above. 5) try disabling offloads on all interfaces on the router (or running cake) This is my next thing to test, I have some other things I need to try first. My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay. Done. and rtt_fair (if you have more than one target server available)... all without the iperf stuff I do not have another machine readily available here at home. 6) I am pretty interested as to what happens *without* sqm at the max forwarding rate with fq_codel engaged on all these tests. This is included. Why did you want to do this test? Just to see if the WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the setup I have, I don't see how there can be any buffering going on because it's a single gig port both in and out. Well, it was good to see fq_codel actually engaging on a few of your hardware queues. There are plenty of reasons why your egress might not match your ingress periodically and vice versa. To quote from the BQL commit: ( https://lwn.net/Articles/454378/ ) Hardware queuing limits are typically specified in terms of a number hardware descriptors, each of which has a variable size. The variability of the size of individual queued items can have a very wide range. For instance with the e1000 NIC the size could range from 64 bytes to 4K (with TSO enabled). This variability makes it next to impossible to choose a single queue limit that prevents starvation and provides lowest possible latency. The objective of byte queue limits is to set the limit to be the minimum needed to prevent starvation between successive transmissions to the hardware. The latency between two transmissions can be variable in a system. It is dependent on interrupt frequency, NAPI polling latencies, scheduling of the queuing discipline, lock contention, etc. Therefore we propose that byte queue limits should be dynamic and change in iaccordance with networking stack latencies a system encounters. In particular, aggressive NAPI settings bug me, in routers. And I am unfond of hardware multiqueue as presently implemented. Sometimes I think it would be better to use them for QoS rather than fq with birthday problems. Another test idea for you would be to enable fq_codel on just the main ethernet devices on the router, and not use hw mq on it at all. (tc qdisc add dev each_ethernet_device_on_router root fq_codel) 2) You still have 15ms of delay at various rates. That is quite a lot more than what I see on a rangeley (where we get below 5ms on sparse traffic (partially due to local cpu scheduling delays), and 200*us* or so when measured at the router with cake) I also see packet loss of the measurement flows in most tests, which are possibly due to gro forcing out smaller packets, or some other factor. It is possible however, that the observed buffering here, is actually on your host, server, or switch. (you have pfifo_fast on your host)
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Dave Taht dave.t...@gmail.com writes: Unfortunately the core piece of metadata I wanted from the router was the qdisc statistics. Didnt parse. Will file bug. My guess is that in this case it's due to the openwrt box missing the tc and ip binaries - those are not installed by default... Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses the ip binary to get the ip and route information to know from which interfaces to pull the qdisc information from... -Toke ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Thu, Jun 25, 2015 at 1:24 PM, Toke Høiland-Jørgensen t...@toke.dk wrote: Dave Taht dave.t...@gmail.com writes: Unfortunately the core piece of metadata I wanted from the router was the qdisc statistics. Didnt parse. Will file bug. My guess is that in this case it's due to the openwrt box missing the tc and ip binaries - those are not installed by default... Or rather, sqm-scripts pulls in the tc binary, I think, but Flent uses the ip binary to get the ip and route information to know from which interfaces to pull the qdisc information from... -Toke I keep wanting to have a way to have the metadata occupy a bottom panel, and be pre-expanded, rather than the side. Metadata is becoming increasingly useful... In other news: Thank you for this commit! I still keep wanting a rewrite in something faster than python, or to buy myself a faster pc when I load up 100 datasets, or find some parallization opportunities... but this was noticibly faster. Author: Toke Høiland-Jørgensen t...@toke.dk Date: Tue Jun 9 13:04:58 2015 +0200 Don't copy raw_values when creating ResultSet. ~3.5x speed-up of loading data files with raw values. Not bad for a two-line patch... :) -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Wed, 24 Jun 2015, Dave Taht wrote: From what I see here you are rarely, if ever, engaging fq_codel properly. Latencies are pretty high. In particular, I would suspect you are hitting offloads hard, and the current (fixed in linux 4.1) codel drop algorithm stops dropping below maxpacket, which was meant in the ns2 code to be a MTU, but in the linux code ended up being a TSO sized (64k!) packet. tc -s qdisc show # will show the maxpacket. http://swm.pp.se/aqm/qdisc-show.txt I discovered also that I wasn't running ECN, I had set tcp_ecn = 2 on the linux box. All the tests done in http://swm.pp.se/aqm/flent-mikabr-150625-1.tar are now done with ECN working, without iperf3 running, and with and without SQM, but with default offload setting. Also, now iperf3 says 0 packet lost when I use that. 2) Please run your flent tests with -x --disable-log Done. Use -t title to differentiate between variables under test. Done. 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox to get the stats on that box into the metadata. You have to add your local .ssh/id_rsa.pub key to your_openwrtbox:/etc/dropbear/authorized_keys file to do this. Done. 4) With all that in hand, sticking up a tarball of the results makes for easy plotting of various other graphs, and using the flent-gui, you can combine results from each run easily, also. See above. 5) try disabling offloads on all interfaces on the router (or running cake) This is my next thing to test, I have some other things I need to try first. My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay. Done. and rtt_fair (if you have more than one target server available)... all without the iperf stuff I do not have another machine readily available here at home. 6) I am pretty interested as to what happens *without* sqm at the max forwarding rate with fq_codel engaged on all these tests. This is included. Why did you want to do this test? Just to see if the WRT1200AC can do wirespeed forwarding with fq_codel on? Because with the setup I have, I don't see how there can be any buffering going on because it's a single gig port both in and out. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Tue, 23 Jun 2015, Sebastian Moeller wrote: As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?) What kind of tests should I run? I have rrul results now, both at the same time as running iperf3. I set iperf3 to run with 10 parallel streams, different MSS per test, and let it run for 30 seconds into the rrul test. So in all the tests, iperf3 session stops running half way into the rrul test. I set sqm to 500M up and down on eth0, ECN up and down, and not to squash DSCP in any direction. http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf Tell me if there is more information I can provide or tests to run. -- Mikael Abrahamssonemail: swm...@swm.pp.se___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Here's a long-range (indoors) test result with the 1900AC. I have about 9dB snr at the client (all the way across my house, which has plaster walls): http://www.dslreports.com/speedtest/737736 download buffering is the 1900ac, upload buffering is the mac laptop's driver. Both are trying too hard to deliver those packets, I think. The driver has crazy over-buffering in the face of poor signal conditions. OTOH, I'm amazed at the throughput numbers. But this is an 80Mhz wide 5Ghz channel, in a quiet environment, so it has LOTS of airspace to work in. -Aaron On Wed, Jun 24, 2015 at 9:32 AM, Dave Taht dave.t...@gmail.com wrote: On Wed, Jun 24, 2015 at 4:31 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Tue, 23 Jun 2015, Sebastian Moeller wrote: As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?) What kind of tests should I run? I have rrul results now, both at the same time as running iperf3. I set iperf3 to run with 10 parallel streams, different MSS per test, and let it run for 30 seconds into the rrul test. So in all the tests, iperf3 session stops running half way into the rrul test. I set sqm to 500M up and down on eth0, ECN up and down, and not to squash DSCP in any direction. http://swm.pp.se/aqm/rrul-wrt1200ac-150624.pdf From what I see here you are rarely, if ever, engaging fq_codel properly. Latencies are pretty high. In particular, I would suspect you are hitting offloads hard, and the current (fixed in linux 4.1) codel drop algorithm stops dropping below maxpacket, which was meant in the ns2 code to be a MTU, but in the linux code ended up being a TSO sized (64k!) packet. tc -s qdisc show # will show the maxpacket. Tell me if there is more information I can provide or tests to run. 0) I tend to have a script for everything... I can give you an example script in another email. 1) tc -s qdisc show dev whatever # will show actuall drop/mark statistics 2) Please run your flent tests with -x --disable-log -x collects more metadata, --disable-log disables the automatic log scaling which is so hard to discern on these plots. Use -t title to differentiate between variables under test. 3) I also tend to use flent's --remote-metadata=root@your_openwrtbox to get the stats on that box into the metadata. You have to add your local .ssh/id_rsa.pub key to your_openwrtbox:/etc/dropbear/authorized_keys file to do this. 4) With all that in hand, sticking up a tarball of the results makes for easy plotting of various other graphs, and using the flent-gui, you can combine results from each run easily, also. 5) try disabling offloads on all interfaces on the router (or running cake) My usual suite of tests is rrul, rrul_be, tcp_1up, tcp_1down, and tcp_2up_delay. and rtt_fair (if you have more than one target server available)... all without the iperf stuff Your mss changing idea is interesting, and I would like to do a mod to tcp_2up_delay to sort of match your iperf combination + flent to totally capture all data. 6) I am pretty interested as to what happens *without* sqm at the max forwarding rate with fq_codel engaged on all these tests. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Wed, 24 Jun 2015, Aaron Wood wrote: Here's a long-range (indoors) test result with the 1900AC. I have about 9dB snr at the client (all the way across my house, which has plaster walls): http://www.dslreports.com/speedtest/737736 download buffering is the 1900ac, upload buffering is the mac laptop's driver. Both are trying too hard to deliver those packets, I think. The driver has crazy over-buffering in the face of poor signal conditions. OTOH, I'm amazed at the throughput numbers. But this is an 80Mhz wide 5Ghz channel, in a quiet environment, so it has LOTS of airspace to work in. Is this over a 25/5 Internet connection? Because I wouldn't imagine that you would get any significant buffering on the wifi with that kind of Internet access speed. I regularily get hundreds of megabit/s over 802.11ac wifi. Also, I guess this is WRT1900ACv1? Running what OS? -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Wed, Jun 24, 2015 at 8:07 PM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Wed, 24 Jun 2015, Aaron Wood wrote: Here's a long-range (indoors) test result with the 1900AC. I have about 9dB snr at the client (all the way across my house, which has plaster walls): Is this over a 25/5 Internet connection? Because I wouldn't imagine that you would get any significant buffering on the wifi with that kind of Internet access speed. I regularily get hundreds of megabit/s over 802.11ac wifi. Comcast cable, with ~150Mbps down and 12Mbps up. here's a close-range result for comparison: http://www.dslreports.com/speedtest/675012 Also, I guess this is WRT1900ACv1? Running what OS? Yep, running OpenWRT CC RC1, with sqm-scripts installed, setup as in: http://burntchrome.blogspot.com/2015/06/sqmscripts-before-and-after-at-160mbps.html -Aaron ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Tue, Jun 23, 2015 at 5:55 AM, Mikael Abrahamsson swm...@swm.pp.se wrote: On Tue, 23 Jun 2015, Sebastian Moeller wrote: Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper. I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from Linux box behind WRT1200AC to an OSX macbook connected on a switch on the same L2 subnet as the WAN port. Linux -WRT1200AC-switch-OSX I get 765 megabit/s of throughput using single session, at sirq load of around 25%. If I lower the mss to 300 (to generate higher pps) I get around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq. If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single session at 80% sirq and 930 megabit/s at 26% sirq with default MSS. Yer missing the more important figure. What is the induced latency in all these cases? With the system being software limited, I would imagine that other oddities arise. Running out of cpu, additional oddities. When going at hardware line rate, is fq_codel enabled? Does it ever engage? (My limited testing showed that lacking BQL, delay accrued big time in the drivers themselves on this platform) Good idea on using a reduced MSS size! I would really like to get to the point where cake ran on this platform, but thus far we have not managed to get a working build, nor push cake into mainline openwrt my observation is that a lot of the overhead of sqm comes from tc filters, iptables rules and NAT. So if you want high performing device that is OpenWRT compatible and still does forwarding using CPU so you can test queuing algorithms, the WRT1200AC and WRT1900ACv2 is the best I have been able to find currently (unless you go for x86 platform). yes, x86 is fastest. I have not tried the reduced mss idea on my rangeley box, I will check that out! -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Hi Mikael, On Jun 23, 2015, at 14:55 , Mikael Abrahamsson swm...@swm.pp.se wrote: On Tue, 23 Jun 2015, Sebastian Moeller wrote: Most likely not. Check http://wiki.openwrt.org/doc/howto/sqm . Rich published a great set of instructions for setting up sqm-scripts under openwrt proper. I tried it on Linksys WRT1200AC with OpenWrt CC RC2. I configured sqm to have 800 megabit/s each direction, and ran iperf3 over IPv4 with NAT44 from Linux box behind WRT1200AC to an OSX macbook connected on a switch on the same L2 subnet as the WAN port. Linux -WRT1200AC-switch-OSX Thanks a lot, interesting data! Was this test stressing both directions at the same time? (My guess is if the test was UDP i don’t know, for a TCP test I am quite confident that it was uni-directional as the @full MTU data does not show enough loss to accommodate the roughly 2% reverse ACK traffic for the opposite direction). I get 765 megabit/s of throughput using single session, at sirq load of around 25%. If I lower the mss to 300 (to generate higher pps) I get around 560 megabit/s of throughput at 50% sirq. With 10 parallel TCP sessions, I get about the same. At MSS of 200 bytes, I get 400 megabit/s at 70% sirq. I assume iperf3 uses TCP or UDP streams and reports the payload rate, correct? Then we have a MSS of 1460 (with 20 bytes IPv4 header and 20 bytes for TCP or UDP). @full MTU MSS: 1500 - 20 - 20 = 1460 byte Number of packets at 765 Mbps goodput: (765 * 1000^2) / ((1500 - 20 - 20) * 8) = 65496.5753425 = 65K On-the-wire packet size (OTWPS) assuming ethernet with FCS and no VLAN tag)s): 1500 + 14 + 4 = 1518 bytes MSS to OTWPS ratio: (1500 - 20 - 20) / (1500 + 14 + 4) = 0.961791831357 raw bandwidth consumed by 765 Mbps good put: 765 / ((1500 - 20 - 20) / (1500 + 14 + 4)) = ((765 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4) * 8) = 795.390410959 Mbps So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad @MSS 300 MSS: 300 byte Number of packets at 560 Mbps goodput: (560 * 1000^2) / ((300) * 8) = 23.33 = 233K On-the-wire packet size (OTWPS) assuming ethernet with FCS: 300 + 20 + 20 + 14 + 4 = 358 bytes MSS to OTWPS ratio: (300) / (300 + 20 + 20 + 14 + 4) = 0.837988826816 raw bandwidth consumed by 560 Mbps good put: 560 / ((300) / (300 + 20 + 20 + 14 + 4 )) = ((560 * 1000^2) / ((300) * 8)) * ((300 + 20 + 20 + 14 + 4 ) * 8) = 668.26667 Mbps So basically (1 - (668.26667/800))*100 = 16.46 % unexplained loss, not pretty but bearable I guess @MSS 200 MSS: 200 byte Number of packets at 400 Mbps goodput: (400 * 1000^2) / ((200) * 8) = 25 = 250K On-the-wire packet size (OTWPS) assuming ethernet with FCS: 200 + 20 + 20 + 14 + 4 = 258 bytes MSS to OTWPS ratio: (200) / (200 + 20 + 20 + 14 + 4) = 0.77519379845 raw bandwidth consumed by 400 Mbps good put: 400 / ((200) / (200 + 20 + 20 + 14 + 4 )) = ((400 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 ) * 8) = 516 Mbps So basically (1 - (516/800))*100 = 35.5 % unexplained loss, that is sad. But the packet rate is still at 250K, I winder how this router does with 64 byte ethernet frames If I turn off SQM completely, I get 600 megabit/s at 200 byte MSS single session at 80% sirq and 930 megabit/s at 26% sirq with default MSS. Since no shaper was used, I think we need to include the inter-frame-gap and preamble to calculate the maximal payload rates for different packet sizes. @1Gbps MSS (1500 - 20 - 20) = 1460 byte Number of packets at 930 Mbps goodput: (930 * 1000^2) / ((1500 - 20 - 20) * 8) = 79623.2876712 = 80K To asses the maximum achievable at 1 GBE we need to take IFG and preamble into account I think On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble: 1500 + 14 + 4 + 12 + 8 = 1538 bytes MSS to OTWPS ratio: (1500 - 20 - 20) / (1500 + 14 + 4 + 12 + 8) = 0.949284785436 raw bandwidth consumed by 930 Mbps good put: 930 / ((1500 - 20 - 20) / (1500 + 14 + 4 + 12 + 8)) = ((930 * 1000^2) / ((1500 - 20 - 20) * 8)) * ((1500 + 14 + 4 + 12 + 8) * 8) = 979.684931507 Mbps So basically (1 - (979.684931507/1000))*100 = 2.0315068493 % unexplained loss, not bad. @1Gbps MSS 200 = 1460 byte Number of packets at 600 Mbps goodput: (600 * 1000^2) / ((200) * 8) = 375000 = 375K On-the-wire packet size (OTWPS) assuming ethernet with FCS plus inter frame gap and preamble: 200 + 20 + 20 + 14 + 4 + 12 + 8 = 278 bytes MSS to OTWPS ratio: 200 / (200 + 20 + 20 + 14 + 4 + 12 + 8) = 0.719424460432 raw bandwidth consumed by 600 Mbps good put: 600 / ((200) / (200 + 20 + 20 + 14 + 4 + 12 + 8)) = ((600 * 1000^2) / ((200) * 8)) * ((200 + 20 + 20 + 14 + 4 + 12 + 8) * 8) = 834 Mbps So basically (1 - (834/1000))*100 = 16.6 % unexplained loss, not bad. As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
Not so easy to find those in Finland, it seems, but I assume Amazon carry them. - Jonathan Morton ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Tue, 23 Jun 2015, Sebastian Moeller wrote: Thanks a lot, interesting data! Was this test stressing both directions at the same time? (My guess is if the test was UDP i don’t know, for a TCP test I am quite confident that it was uni-directional as the @full MTU data does not show enough loss to accommodate the roughly 2% reverse ACK traffic for the opposite direction). It was TCP using iperf3, so it was larger data packets one direction and ACKs going the other direction. So basically (1 - (795.390410959/800))*100 = 0.58 % unexplained loss, not bad Oh, iperf3 can tell you packets lost, so I can re-run the tests and tell you exactly how many packets were lost and within what 1 second interval they were lost. My guess is that it would be better to run something else. My numbers were not meant to be as exact as you have used for calculation, I'd say I aimed for approximate number. I'll do a test run and give you more exact data. As Dave said it would be nice see RRUL data from the same testbed. It would be so nice if flint had a way to send different sized TCP packets… (I guess this might be faked with MSS clamping in the router and relaying on path MTU discovery?) That is my next thing to test. The 1200AC retailed for around 200EUR in Germany the 1900ACv2 will be closer to 300EUR I guess, not too expensive but certainly above my impulse buy limit ;) Yes, I paid 195 EUR for it (1790 SEK), which includes 25% VAT. It's around 150 USD (not including tax) so I'm hoping it'll drop 10-20% in price when it's more widely available in Europe. -- Mikael Abrahamssonemail: swm...@swm.pp.se___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; cake overhead 40 didn't)
On Tue, 23 Jun 2015, Jonathan Morton wrote: Not so easy to find those in Finland, it seems, but I assume Amazon carry them. www.webhallen.com is the only retailer in Sweden that currently have them in stock. They just now becoming available, so I would imagine in the next few weeks they'll be more widely available. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
