Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 8, 2019, at 11:27 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> >>> Your patch in the latest kernels looks simpler. Bringing the patch >>> back to prior kernel versions would be appreciated, but I can >>> understand how 3.16 becomes less and less

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 8, 2019, at 11:33 PM, Dave Taht wrote: > > On Tue, Jan 8, 2019 at 2:01 PM Pete Heist > wrote: >> >> I should have done that: >> https://www.heistp.net/downloads/htb_split_gso_patched/ >> >> >> Note

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 8, 2019, at 11:27 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >> Your patch in the latest kernels looks simpler. Bringing the patch >> back to prior kernel versions would be appreciated, but I can >> understand how 3.16 becomes less and less relevant as time goes on,

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

On Tue, Jan 8, 2019 at 2:01 PM Pete Heist wrote: > > I should have done that: > https://www.heistp.net/downloads/htb_split_gso_patched/ > > Note that I changed the names in the plots to match the convention of my > first email, but it should be clear which is which and I left all plots in. >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > Your patch in the latest kernels looks simpler. Bringing the patch > back to prior kernel versions would be appreciated, but I can > understand how 3.16 becomes less and less relevant as time goes on, > although, it’s not at end of life yet. :) Could you try the latest git

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

I should have done that: https://www.heistp.net/downloads/htb_split_gso_patched/ Note that I changed the names in the plots to match the convention of my first email, but it should be clear which is which and I left all plots in. The text output is there too as I sometimes like to open several

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

On Tue, Jan 8, 2019 at 12:03 PM Pete Heist wrote: > > Here’s the re-test with the patched version and HTB. Looks like success, nice > work! I note that I'm big on having the flent.gz files around also. In this case, by eyeball, split-gso appears to have about 130us less latency, but a cdf

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Here’s the re-test with the patched version and HTB. Looks like success, nice work! Split GSO on: https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split1.svg https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split2.svg

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Sorry, that’s without the patch, will give that a try when I have a chance and post the results, probably tomorrow... > On Jan 7, 2019, at 12:30 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >>> On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen wrote: >>> >>> Pete Heist

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> >>> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve >>> been doing that for a while without any apparent issues, though I’m >>> hesitating now to try that in a

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

another answer is to disable gro/tso in the driver via ethtool. On Sun, Jan 6, 2019 at 4:30 PM Pete Heist wrote: > > > > On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen wrote: > > > > Pete Heist writes: > > > >> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve > >>

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve >> been doing that for a while without any apparent issues, though I’m >> hesitating now to try that in a production environment. > >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve > been doing that for a while without any apparent issues, though I’m > hesitating now to try that in a production environment. Hmm, that's a good question. I would expect so; but I would also expect the

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 5, 2019, at 11:27 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> On Jan 5, 2019, at 9:10 PM, Toke Høiland-Jørgensen wrote: Well, it's the same WARN_ON(), and if that patch had been applied, debugging our issue would have been

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 5, 2019, at 11:27 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >>> On Jan 5, 2019, at 9:10 PM, Toke Høiland-Jørgensen wrote: >>> >>> Well, it's the same WARN_ON(), and if that patch had been applied, >>> debugging our issue would have been a lot harder, I think. >>

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 5, 2019, at 9:10 PM, Toke Høiland-Jørgensen wrote: >> >> Well, it's the same WARN_ON(), and if that patch had been applied, >> debugging our issue would have been a lot harder, I think. > > Yikes, this is what I mean. I’d rather suffer the warning than be >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > That first bug report looks decidedly similar to mine, but Toke would > have to comment on the specifics. So far I see the patch to > sch_codel.c you mentioned and another two-liner to remove the warning > in hfsc.c (https://patchwork.ozlabs.org/patch/933611/). It would be >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

That first bug report looks decidedly similar to mine, but Toke would have to comment on the specifics. So far I see the patch to sch_codel.c you mentioned and another two-liner to remove the warning in hfsc.c (https://patchwork.ozlabs.org/patch/933611/). It would be really good to know that

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Dear all, I am most likely wrong, but did you have a look at https://bugs.openwrt.org/index.php?do=details_id=1136 yet? Especially https://bugzilla.kernel.org/show_bug.cgi?id=109581 and https://www.spinics.net/lists/netdev/msg450655.html might be related to Pete's bug. Then again, I might be

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 5, 2019, at 2:35 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> On Jan 5, 2019, at 2:10 PM, Toke Høiland-Jørgensen wrote: Hmm, that's odd. Could you try adding this debugging line in adjust_parent_qlen(), right before the

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On 5 Jan, 2019, at 5:34 pm, Pete Heist wrote: > > There’s nobody in MAINTAINERS for hfsc, so we may not get much of a response > to any bug submissions... Most likely it defaults to Eric Dumazet in that case. - Jonathan Morton ___ Cake mailing

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 5, 2019, at 2:35 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > >>> On Jan 5, 2019, at 2:10 PM, Toke Høiland-Jørgensen wrote: >>> >>> Hmm, that's odd. Could you try adding this debugging line in >>> adjust_parent_qlen(), right before the sch->q.qlen += n line: >>> >>>

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 5, 2019, at 2:10 PM, Toke Høiland-Jørgensen wrote: >> >> Hmm, that's odd. Could you try adding this debugging line in >> adjust_parent_qlen(), right before the sch->q.qlen += n line: >> >> net_info_ratelimited("Adjusting parent qdisc %d with pkt +=

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 5, 2019, at 2:10 PM, Toke Høiland-Jørgensen wrote: > > Hmm, that's odd. Could you try adding this debugging line in > adjust_parent_qlen(), right before the sch->q.qlen += n line: > > net_info_ratelimited("Adjusting parent qdisc %d with pkt += %d, > len += %d", >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > Ok, that fixes the compiler warnings, but I get this now. Same as > before it’s repeated until reboot, stack sometimes changes but I > always see sch_hfsc.c:1427 at the beginning: Hmm, that's odd. Could you try adding this debugging line in adjust_parent_qlen(), right

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > Quick update to the trace because I had to apply the patch manually > and missed one line to remove (qdisc_tree_reduce_backlog...), just so > it doesn’t through off the addresses for you, but it still does the > same thing: Ah, my bad; you need to replace

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Giving you this also, for how it’s set up... IFACE=eth0 RATE=100mbit tc qdisc add dev $IFACE root handle 1: hfsc default 1 tc class add dev $IFACE parent 1: classid 1:1 hfsc ls rate $RATE ul rate $RATE tc class add dev $IFACE parent 1: classid 1:2 hfsc ls rate $RATE ul rate $RATE tc qdisc add

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Nope, egress on both eth0 and eth0.3300. Dunce question, but I’m applying the patch manually because copying it from email didn’t seem to work- how to get patch to work? root@apu1a:~/src/sch_cake# patch sch_cake.c ../hfsc.patch patching file sch_cake.c patch: malformed patch at line 7:

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

On 5 January 2019 11:44:44 CET, Jonathan Morton wrote: >> On 5 Jan, 2019, at 12:34 am, Toke Høiland-Jørgensen >wrote: >> >> This basically means that we can't use CAKE as a leaf qdisc with GSO >> splitting as it stands currently. I *think* the solution is for CAKE >to >> notify its parents;

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Quick update to the trace because I had to apply the patch manually and missed one line to remove (qdisc_tree_reduce_backlog...), just so it doesn’t through off the addresses for you, but it still does the same thing: root@apu1a:~/src/sch_cake# [ 697.089814] BUG: unable to handle kernel NULL

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 5, 2019, at 11:06 AM, Toke Høiland-Jørgensen wrote: > > Hmm, try this version for 3.16 - probably doesn't work on later kernels. > I'll look into a proper backport once you've confirmed that it works :) Thanks! Quick reminder, I’ve only seen this happen with hfsc, not when cake is a

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On 5 Jan, 2019, at 12:34 am, Toke Høiland-Jørgensen wrote: > > This basically means that we can't use CAKE as a leaf qdisc with GSO > splitting as it stands currently. I *think* the solution is for CAKE to > notify its parents; could you try the patch below and see if it helps? Is this also a

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 4, 2019, at 11:34 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> >> This basically means that we can't use CAKE as a leaf qdisc with GSO >> splitting as it stands currently. I *think* the solution is for CAKE to >> notify its parents; could you try

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 4, 2019, at 11:34 PM, Toke Høiland-Jørgensen wrote: > > Pete Heist writes: > > This basically means that we can't use CAKE as a leaf qdisc with GSO > splitting as it stands currently. I *think* the solution is for CAKE to > notify its parents; could you try the patch below and see if

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: >> On Jan 4, 2019, at 10:34 PM, Toke Høiland-Jørgensen wrote: >> >> Pete Heist writes: >> >>> Ok, the lockup goes away if you use no-split-gso on the cake qdiscs for the >>> default traffic (noted below in the drr and hfsc cases with "!!! must use >>> no-split-gso here

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Jan 4, 2019, at 11:10 PM, Pete Heist wrote: > > Ok, fwiw one doesn’t actually need a one-armed router or VLANs to reproduce > this. Just do this: > > tc qdisc add dev $IFACE root handle 1: hfsc default 1 > tc class add dev $IFACE parent 1: classid 1:1 hfsc ls rate $RATE ul rate $RATE >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Pete Heist writes: > Ok, the lockup goes away if you use no-split-gso on the cake qdiscs for the > default traffic (noted below in the drr and hfsc cases with "!!! must use > no-split-gso here !!!"). Only I’d like my 600 μs back. :) > > This smells of a bug Toke fixed on Sep 12, 2018 in >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On Dec 28, 2018, at 11:07 PM, Jonathan Morton wrote: > >> On 28 Dec, 2018, at 11:22 pm, Pete Heist wrote: >> >> This smells of a bug Toke fixed on Sep 12, 2018 in >> 42e87f12ea5c390bf5eeb658c942bc810046160a, but then reverted in the next >> commit because it was fixed upstream. However,

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

> On 28 Dec, 2018, at 11:22 pm, Pete Heist wrote: > > This smells of a bug Toke fixed on Sep 12, 2018 in > 42e87f12ea5c390bf5eeb658c942bc810046160a, but then reverted in the next > commit because it was fixed upstream. However, if I re-apply that commit, it > still doesn’t fix it. > >

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

Note that this doesn’t happen when prio is used in place of hfsc and cake is used in the leafs to do the rate limiting, i.e.: tc qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth 100mbit

[Cake] cake infinite loop(?) with hfsc on one-armed router

I’m seeing what I think it an infinite loop when cake is used in a one-armed router configuration with hfsc as the rate limiter. Three APUs are connected to the same switch and the “middle” APU (apu1a) routes between the default VLAN and a tagged VLAN. apu2a <— default VLAN —> apu1a <—