Hi Dave, I guess I found the culprit:
once I added $ADSLL to the ingress() in simple.qos: ingress() { CEIL=$DOWNLINK PRIO_RATE=`expr $CEIL / 3` # Ceiling for prioirty BE_RATE=`expr $CEIL / 6` # Min for best effort BK_RATE=`expr $CEIL / 6` # Min for background BE_CEIL=`expr $CEIL - 64` # A little slop at the top LQ="quantum `get_mtu $IFACE`" $TC qdisc del dev $IFACE handle ffff: ingress 2> /dev/null $TC qdisc add dev $IFACE handle ffff: ingress $TC qdisc del dev $DEV root 2> /dev/null $TC qdisc add dev $DEV root handle 1: ${STABSTRING} htb default 12 $TC class add dev $DEV parent 1: classid 1:1 htb $LQ rate ${CEIL}kbit ceil ${CEIL}kbit $ADSLL $TC class add dev $DEV parent 1:1 classid 1:10 htb $LQ rate ${CEIL}kbit ceil ${CEIL}kbit prio 0 $ADSLL $TC class add dev $DEV parent 1:1 classid 1:11 htb $LQ rate 32kbit ceil ${PRIO_RATE}kbit prio 1 $ADSLL $TC class add dev $DEV parent 1:1 classid 1:12 htb $LQ rate ${BE_RATE}kbit ceil ${BE_CEIL}kbit prio 2 $ADSLL $TC class add dev $DEV parent 1:1 classid 1:13 htb $LQ rate ${BK_RATE}kbit ceil ${BE_CEIL}kbit prio 3 $ADSLL # I'd prefer to use a pre-nat filter but that causes permutation... $TC qdisc add dev $DEV parent 1:11 handle 110: $QDISC limit 1000 $ECN `get_quantum 500` `get_flows ${PRIO_RATE}` $TC qdisc add dev $DEV parent 1:12 handle 120: $QDISC limit 1000 $ECN `get_quantum 1500` `get_flows ${BE_RATE}` $TC qdisc add dev $DEV parent 1:13 handle 130: $QDISC limit 1000 $ECN `get_quantum 1500` `get_flows ${BK_RATE}` diffserv $DEV ifconfig $DEV up # redirect all IP packets arriving in $IFACE to ifb0 $TC filter add dev $IFACE parent ffff: protocol all prio 10 u32 \ match u32 0 0 flowid 1:1 action mirred egress redirect dev $DEV } I get basically the same RRUL ping RTTs for htb_private as for tc_stab. So Jesper was right the patch seems to fix the issue. I guess I should send out my current version of yours and Toke's AQM scripts soon. Best Sebastian P.S.: I am not sure whether I want to tackle the PIE issue today... On Aug 23, 2013, at 21:47 , Dave Taht <dave.t...@gmail.com> wrote: > quick note: running this script requires that you > > ifconfig ifb0 up > > at some point. In my case on cerowrt you took care of that already... > > > On Fri, Aug 23, 2013 at 12:38 PM, Sebastian Moeller <moell...@gmx.de> wrote: > Hi Dave, > > On Aug 23, 2013, at 07:13 , Dave Taht <dave.t...@gmail.com> wrote: > > > > > > > > > On Thu, Aug 22, 2013 at 5:52 PM, Sebastian Moeller <moell...@gmx.de> wrote: > > Hi List, hi Jesper, > > > > So I tested 3.10.9-1 to assess the status of the HTB atm link layer > > adjustments to see whether the recent changes resurrected this feature. > > Unfortunately the htb_private link layer adjustments still is > > broken (RRUL ping RTT against Toke's netperf host in Germany of ~80ms, same > > as without link layer adjustments). On the bright side the tc_stab method > > still works as well as before (ping RTT around 40ms). > > I would like to humbly propose to use the tc stab method in cerowrt > > to perform ATM link layer adjustments as default. To repeat myself, simply > > telling the kernel a lie about the packet size seems more robust than > > fudging HTB's rate tables. Especially since the kernel already fudges the > > packet size to account for the ethernet header and then some, so this path > > should receive more scrutiny by virtue of having more users? > > > > It's my hope that the atm code works but is misconfigured. You can output > > the tc commands by overriding the TC variable with TC="echo tc" and paste > > here. > > So I went for TC="logger tc" and used log read to harvest as I could > not find the echo output, but I guess that should not matter. So here is the > result (slightly edited to get rid of the log timestamps and log level): > > tc qdisc del dev ge00 root > tc qdisc add dev ge00 root handle 1: htb default 12 > tc class add dev ge00 parent 1: classid 1:1 htb quantum 1500 rate 2430kbit > ceil 2430kbit mpu 0 linklayer adsl overhead 40 mtu 2047 > tc class add dev ge00 parent 1:1 classid 1:10 htb quantum 1500 rate > 2430kbit ceil 2430kbit prio 0 mpu 0 linklayer adsl overhead 40 mtu 2047 > tc class add dev ge00 parent 1:1 classid 1:11 htb quantum 1500 rate 128kbit > ceil 810kbit prio 1 mpu 0 linklayer adsl overhead 40 mtu 2047 > tc class add dev ge00 parent 1:1 classid 1:12 htb quantum 1500 rate 405kbit > ceil 2366kbit prio 2 mpu 0 linklayer adsl overhead 40 mtu 2047 > tc class add dev ge00 parent 1:1 classid 1:13 htb quantum 1500 rate 405kbit > ceil 2366kbit prio 3 mpu 0 linklayer adsl overhead 40 mtu 2047 > tc qdisc add dev ge00 parent 1:11 handle 110: fq_codel limit 600 noecn > quantum 300 > tc qdisc add dev ge00 parent 1:12 handle 120: fq_codel limit 600 noecn > quantum 300 > tc qdisc add dev ge00 parent 1:13 handle 130: fq_codel limit 600 noecn > quantum 300 > tc filter add dev ge00 parent 1:0 protocol all prio 999 u32 match ip > protocol 0 0x00 flowid 1:12 > tc filter add dev ge00 parent 1:0 protocol ip prio 1 handle 1 fw classid > 1:11 > tc filter add dev ge00 parent 1:0 protocol ip prio 2 handle 2 fw classid > 1:12 > tc filter add dev ge00 parent 1:0 protocol ip prio 3 handle 3 fw classid > 1:13 > tc filter add dev ge00 parent 1:0 protocol ipv6 prio 4 handle 1 fw classid > 1:11 > tc filter add dev ge00 parent 1:0 protocol ipv6 prio 5 handle 2 fw classid > 1:12 > tc filter add dev ge00 parent 1:0 protocol ipv6 prio 6 handle 3 fw classid > 1:13 > tc filter add dev ge00 parent 1:0 protocol arp prio 7 handle 1 fw classid > 1:11 > tc qdisc del dev ge00 handle ffff: ingress > tc qdisc add dev ge00 handle ffff: ingress > tc qdisc del dev ifb0 root > tc qdisc add dev ifb0 root handle 1: htb default 12 > tc class add dev ifb0 parent 1: classid 1:1 htb quantum 1500 rate 15494kbit > ceil 15494kbit > tc class add dev ifb0 parent 1:1 classid 1:10 htb quantum 1500 rate > 15494kbit ceil 15494kbit prio 0 > tc class add dev ifb0 parent 1:1 classid 1:11 htb quantum 1500 rate 32kbit > ceil 5164kbit prio 1 > tc class add dev ifb0 parent 1:1 classid 1:12 htb quantum 1500 rate > 2582kbit ceil 15430kbit prio 2 > tc class add dev ifb0 parent 1:1 classid 1:13 htb quantum 1500 rate > 2582kbit ceil 15430kbit prio 3 > tc qdisc add dev ifb0 parent 1:11 handle 110: fq_codel limit 1000 ecn > quantum 500 > tc qdisc add dev ifb0 parent 1:12 handle 120: fq_codel limit 1000 ecn > quantum 1500 > tc qdisc add dev ifb0 parent 1:13 handle 130: fq_codel limit 1000 ecn > quantum 1500 > tc filter add dev ifb0 parent 1:0 protocol all prio 999 u32 match ip > protocol 0 0x00 flowid 1:12 > tc filter add dev ifb0 protocol ip parent 1:0 prio 1 u32 match ip tos 0x00 > 0xfc classid 1:12 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 2 u32 match ip6 > priority 0x00 0xfc classid 1:12 > tc filter add dev ifb0 protocol ip parent 1:0 prio 3 u32 match ip tos 0x20 > 0xfc classid 1:13 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 4 u32 match ip6 > priority 0x20 0xfc classid 1:13 > tc filter add dev ifb0 protocol ip parent 1:0 prio 5 u32 match ip tos 0x10 > 0xfc classid 1:11 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 6 u32 match ip6 > priority 0x10 0xfc classid 1:11 > tc filter add dev ifb0 protocol ip parent 1:0 prio 7 u32 match ip tos 0xb8 > 0xfc classid 1:11 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 8 u32 match ip6 > priority 0xb8 0xfc classid 1:11 > tc filter add dev ifb0 protocol ip parent 1:0 prio 9 u32 match ip tos 0xc0 > 0xfc classid 1:11 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 10 u32 match ip6 > priority 0xc0 0xfc classid 1:11 > tc filter add dev ifb0 protocol ip parent 1:0 prio 11 u32 match ip tos 0xe0 > 0xfc classid 1:11 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 12 u32 match ip6 > priority 0xe0 0xfc classid 1:11 > tc filter add dev ifb0 protocol ip parent 1:0 prio 13 u32 match ip tos 0x90 > 0xfc classid 1:11 > tc filter add dev ifb0 protocol ipv6 parent 1:0 prio 14 u32 match ip6 > priority 0x90 0xfc classid 1:11 > tc filter add dev ifb0 parent 1:0 protocol arp prio 15 handle 1 fw classid > 1:11 > tc filter add dev ge00 parent ffff: protocol all prio 10 u32 match u32 0 0 > flowid 1:1 action mirred egress redirect dev ifb0 > > I notice it seem this only shows up for egress(), but looking at simple.qos > ingress() is not addend ${ADSLL} at all so that is to be expected. There is > nothing in dmesg at all. > > So I am off to add ADSLL to ingress() as well and then test RRUL again... > > > Jesper please let me know if this looks reasonable, at least to my eye it > seems to fit with what "tc disc add htb help" tells me. I tried your: > echo "func __detect_linklayer +p" /sys/kernel/debug/dynamic_debug/control > but got no output even though debugs was already mounted… > > Best > Sebastian > > > > > Now, I have been testing this using Dave's most recent cerowrt > > alpha version with a 3.10.9 kernel on mips hardware, I think this kernel > > should contain all htb fixes including commit 8a8e3d84b17 (net_sched: > > restore "linklayer atm" handling) but am not fully sure. > > > > It does. > > > > `@Dave is there an easy way to find which patches you applied to the > > kernels of the cerowrt (testing-)releases? > > > > Normally I DO commit stuff that is in testing, but my big push this time > > around was to get everything important into mainline 3.10, as it will be > > the "stable" release for a good long time. > > > > So I am still mostly working the x86 side at the moment. I WAS kind of > > hoping that everything I just landed would make it up to 3.10. But for your > > perusal: > > > > http://snapon.lab.bufferbloat.net/~cero2/patches/3.10.9-1/ has most of the > > kernel patches I used in it. 3.10.9-2 has the ipv6subtrees patch ripped out > > due to another weird bug I'm looking at. (It also has support for ipv6 nat > > thx to the ever prolific stephen walker heeding the call for patches...). > > 100% totally untested, I have this weird bug to figure out how to fix next: > > > > http://lists.alioth.debian.org/pipermail/babel-users/2013-August/001419.html > > > > I fear it's a comparison gone south, maybe in bradley's optimizations for > > not kernel trapping, don't know. > > > > 3.10.9-2 also disables dnsmasq's dhcpv6 in favor of 6relayd. I HATE losing > > the close naming integration, but, had to try this.... > > > > If you guys want me to start committing and pushing patches again, I'll do > > it, but most of that stuff will end up in 3.10.10, I think, in a couple > > days. The rest might make 3.12. Pie has to survive scrutiny on the netdev > > list in particular. > > > > While I have you r attention :) I also tested 3.10.9-1's pie and it is way > > better than 3.10.6-1's (RRUL ping RTTs around 110 ms instead of 3000ms) but > > still worse than fq_codel (ping RTTs around 40ms with proper atm link layer > > adjustments). > > > > This is with simple.qos I imagine? Simplest should do better than that with > > pie. Judging from how its estimator works I think it will do badly with > > multiple queues. But testing will tell... > > > > But, yea, this pie is actually usable, and the previous wasn't. Thank you > > for looking at it! > > > > It is different from cisco's last pie drop in that it can do ecn, does > > local congestion notification, has a better use of net_random, it's mostly > > KernelStyle, and I forget what else. > > > > There is still a major rounding error in the code, and I'd like cisco to > > fix the api so it uses identical syntax to codel. Right now you specify > > "target 8" to get "target 7", and the "ms" is implied. target 5 becomes > > target 3. The default target is a whopping 20 (rounded to 19), which is in > > part where your 70+ms of extra delay came from. > > > > Multiple parties have the delusion that 20ms is "good enough". > > > > Part of the remaining delay may also be rounding error. Cisco uses kernels > > with HZ=1000, cero uses HZ=250..... > > > > Anyway, to get more comparable tests... you can fiddle with the two $QDISC > > lines in simple*.qos to add a target 8 to get closer to a codel 5ms config, > > but that would break a codel config which treats target 8 as target 8us. > > > > I MIGHT, if I get energetic enough, fix the API, the time accounting, and a > > few other things in pie, the problem is, that ns2_codel seems still more > > effective on most workloads and *fq_codel smokes absolutely everything. > > There are a few places where pie is a win over straight codel, notably on > > packet floods. And it may well be easier to retrofit into existing hardware > > fast path designs. > > > > I worry about interactions between pie and other stuff. It seems inevitable > > at this point that some form of pie will be widely deployed, and I simply > > haven't tried enough traffic types and RTTs to draw a firm conclusion, > > period. Long RTTs are the last big place where codel and pie and fq_codel > > have to be seriously tested. > > > > ns2_codel is looking pretty good now, at the shorter RTTs I've tried. A big > > problem I have is getting decent long RTT emulation out of netem (some > > preliminary code is up at github) > > > > ... and getting cero stable enough for others to actually use - next up is > > fixing the userspace problems. > > > > ... and trying to make a small dent in the wifi problem along the way > > (couple commits coming up) > > > > ... and find funding to get through the winter. > > > > There's probably a few other things that are on that list but I forget. Oh, > > yea, since the aqm wg was voted on to be formed, I decided I could quit > > smoking. > > > > While I am not able to build kernels, it seems that I am able to quickly > > test whether link layer adjustments work or not. SO aim happy to help where > > I can :) > > > > Give pie target 8 and target 5 a shot, please? ns2_codel target 3ms and > > target 7ms, too. fq_codel, same.... > > > > tc -s qdisc show dev ge00 > > tc -s qdisc show dev ifb0 > > > > would be useful info to have in general after each test. > > > > TIA. > > > > There are also things like tcp_upload and tcp_download and > > tcp_bidirectional that are useful tests in the rrul suite. > > > > Thank you for your efforts on these early alpha releases. I hope things > > will stablize more soon, and I'll fold your aqm stuff into my next attempt > > this weekend. > > > > This is some of the stuff I know that needs fixing in userspace: > > > > * TODO readlink not found > > * TODO netdev user missing > > * TODO Wed Dec 5 17:14:46 2012 authpriv.error dnsmasq: found already > > running DHCP-server on interface 'se00' refusing to start, use 'option > > force 1' to override > > * TODO [ 18.480468] Mirror/redirect action on > > [ 18.539062] Failed to load ipt action > > * upload and download are reversed in aqm > > * BCP38 > > * Squash CS values > > * Replace ntp > > * Make ahcp client mode > > * Drop more privs for polipo > > * upnp > > * priv separation > > * Review FW rules > > * dhcpv6 support > > * uci-defaults/make-cert.sh uses a bad path for px5g > > * Doesn't configure the web browser either > > > > > > > > > > Best > > Sebastian > > > > > > > > > > -- > > Dave Täht > > > > Fixing bufferbloat with cerowrt: > > http://www.teklibre.com/cerowrt/subscribe.html > > > > > -- > Dave Täht > > Fixing bufferbloat with cerowrt: > http://www.teklibre.com/cerowrt/subscribe.html _______________________________________________ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel