Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Pete Heist 

> On Jan 8, 2019, at 11:33 PM, Dave Taht  wrote:
> 
> On Tue, Jan 8, 2019 at 2:01 PM Pete Heist  > wrote:
>> 
>> I should have done that: 
>> https://www.heistp.net/downloads/htb_split_gso_patched/ 
>> 
>> 
>> Note that I changed the names in the plots to match the convention of my 
>> first email, but it should be clear which is which and I left all plots in. 
>> The text output is there too as I sometimes like to open several up in 
>> different browser tabs and switch between tabs to compare values.
>> 
>> It looks like about 100 usec to me. Throughput also looks consistently about 
>> 0.3 Mbit higher (~1.3%) in the split results.
> 
> My guess is with ecn on would have the highest latency and the same
> throughput. ?

I see a slight increase in icmp/udp rtt (1.04ms-1.02ms = 20us) and a slight 
increase in throughput (180.12Mbit - 179.50Mbit = 620kbit).

https://www.heistp.net/downloads/htb_cakep2_ecnon/
https://www.heistp.net/downloads/htb_cakep2_ecnon2/

These results are not five sigma. :)

> Since we started this effort in an era when seconds of added latency
> was common, a mere 100us improvement seems insignificant, except that
> that's a 10% improvement over the present-day baseline, and *that's
> worth it*. ;) This is also a function of the number of flows, kernel

Who knows how many timeouts are avoided in the future, just because of this 
100us. :)

> I have noticed that BQL's values can get quite large with cake doing
> the shaping, btw, much larger than they do with htb.

Wonder why that is.

So far it’s harder to use cake’s shaper in the current setup I’m working on, 
because it appears “better" to have eth0 and eth0.3300 under one link sharing 
hierarchy for eth0:

tc qdisc add dev eth0 root handle 1: htb default 1
tc class add dev eth0 parent 1: classid 1:1 htb rate $RATE
tc class add dev eth0 parent 1: classid 1:2 htb rate $RATE
tc qdisc add dev eth0 parent 1:1 cake besteffort
tc qdisc add dev eth0 parent 1:2 cake besteffort
tc filter add dev eth0 parent 1:0 prio 1 protocol all \
basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 1:1
tc filter add dev eth0 parent 1:0 prio 2 protocol all \
basic match "meta(vlan mask 0xfff eq `printf \"0x%x\" $VLAN_TAG`)" 
flowid 1:2

but I can’t do that with cake’s shaper. It’s better in the sense that when 
you’re out of CPU, latency doesn’t increase suddenly. I _can_ use cake’s shaper 
by adding cake to the VLAN interface and filtering out vlan traffic from the 
main interface:

tc qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 
1 1 1 1 1
tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth $RATE
tc filter add dev eth0 parent 1:0 prio 1 protocol all \
basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 10:1
tc qdisc add dev eth0.3300 root cake besteffort bandwidth $RATE

but when you’re out of CPU, you get starvation and inter-flow latency 
increases. So so far, hfsc or htb is working better in this case. It might be 
something to think about for an “ISP Cake”…

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Pete Heist 

> On Jan 8, 2019, at 11:27 PM, Toke Høiland-Jørgensen  wrote:
> 
> Pete Heist  writes:
> 
>> Your patch in the latest kernels looks simpler. Bringing the patch
>> back to prior kernel versions would be appreciated, but I can
>> understand how 3.16 becomes less and less relevant as time goes on,
>> although, it’s not at end of life yet. :)
> 
> Could you try the latest git version of the out-of-tree version of cake;
> that has the same patch as upstream, and I *think* it will work on old
> kernels as well (for the HTB issue; can't fix HFSC without patching it).

Yes, it compiles fine on 3.16, and gives statistically identical results for 
HTB. :)

https://www.heistp.net/downloads/htb_split_gso_patched2/ 


___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Dave Taht 
On Tue, Jan 8, 2019 at 2:01 PM Pete Heist  wrote:
>
> I should have done that: 
> https://www.heistp.net/downloads/htb_split_gso_patched/
>
> Note that I changed the names in the plots to match the convention of my 
> first email, but it should be clear which is which and I left all plots in. 
> The text output is there too as I sometimes like to open several up in 
> different browser tabs and switch between tabs to compare values.
>
> It looks like about 100 usec to me. Throughput also looks consistently about 
> 0.3 Mbit higher (~1.3%) in the split results.

My guess is with ecn on would have the highest latency and the same
throughput. ?

Since we started this effort in an era when seconds of added latency
was common, a mere 100us improvement seems insignificant, except that
that's a 10% improvement over the present-day baseline, and *that's
worth it*. ;) This is also a function of the number of flows, kernel
scheduling time, etc, etc. Theoretically,
were there no other delays in the system, we seem to actually be at
the minimal RTT achievable (4*130ms in each direction for the fat
flows = 1040us) but given the fast/slow queue abstraction the best
possible result would be 10s of us for the measurement flows.

I have noticed that BQL's values can get quite large with cake doing
the shaping, btw, much larger than they do with htb.

> > On Jan 8, 2019, at 9:44 PM, Dave Taht  wrote:
> >
> > On Tue, Jan 8, 2019 at 12:03 PM Pete Heist  wrote:
> >>
> >> Here’s the re-test with the patched version and HTB. Looks like success, 
> >> nice work!
> >
> > I note that I'm big on having the flent.gz files around also. In this
> > case, by eyeball, split-gso appears to have about 130us less latency,
> > but a cdf comparison of split vs no-split woud show that more easily.
> >
> >> Split GSO on:
> >>
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split1.svg
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split2.svg
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split3.svg
> >>
> >> Split GSO off:
> >>
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split1.svg
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split2.svg
> >> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split3.svg
> >>
> >> Your patch in the latest kernels looks simpler. Bringing the patch back to 
> >> prior kernel versions would be appreciated, but I can understand how 3.16 
> >> becomes less and less relevant as time goes on, although, it’s not at end 
> >> of life yet. :)
> >>
> >> Interesting how download rate control in each of the graphs with GSO 
> >> splitting on looks accurate to the point where flent’s throughput graph 
> >> scale is at 0.02 Mbit per step, and one can see that values coming back 
> >> from netperf are probably quantized to 0.01 Mbit...
> >>
> >>> On Jan 7, 2019, at 4:07 PM, Pete Heist  wrote:
> >>>
> >>> Sorry, that’s without the patch, will give that a try when I have a 
> >>> chance and post the results, probably tomorrow...
> >>>
>  On Jan 7, 2019, at 12:30 PM, Toke Høiland-Jørgensen  wrote:
> 
>  Pete Heist  writes:
> 
> >> On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen  
> >> wrote:
> >>
> >> Pete Heist  writes:
> >>
> >>> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve
> >>> been doing that for a while without any apparent issues, though I’m
> >>> hesitating now to try that in a production environment.
> >>
> >> Hmm, that's a good question. I would expect so; but I would also expect
> >> the issue to show up pretty much straight away, so if you haven't hit 
> >> it
> >> yet, I may be wrong. I'll do some more digging... Should probably also
> >> try to replicate all this stuff on my own machine :)
> >
> >
> > Ok, after what I’m seeing on my APU1 tests on 3.16.7, I’m definitely
> > not putting split GSO into production. I just turned it on and off
> > three times and here’s what I got:
> >
> > Split GSO on:
> >
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso.svg
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso2.svg
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso3.svg
> >
> > Split GSO off:
> >
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso.svg
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso2.svg
> > https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso3.svg
> >
> > I’ve seen these square waves before with htb and wondered where they
> > came from, and I think we may finally have an answer! What manner of
> > thing causes this I don’t know, but there’s a chance you may end up
> > finding out… :)
> 
>  Is this without the patch to CAKE that adjusts the qlen? And have

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Toke Høiland-Jørgensen 
Pete Heist  writes:

> Your patch in the latest kernels looks simpler. Bringing the patch
> back to prior kernel versions would be appreciated, but I can
> understand how 3.16 becomes less and less relevant as time goes on,
> although, it’s not at end of life yet. :)

Could you try the latest git version of the out-of-tree version of cake;
that has the same patch as upstream, and I *think* it will work on old
kernels as well (for the HTB issue; can't fix HFSC without patching it).

-Toke
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Pete Heist 
I should have done that: https://www.heistp.net/downloads/htb_split_gso_patched/

Note that I changed the names in the plots to match the convention of my first 
email, but it should be clear which is which and I left all plots in. The text 
output is there too as I sometimes like to open several up in different browser 
tabs and switch between tabs to compare values.

It looks like about 100 usec to me. Throughput also looks consistently about 
0.3 Mbit higher (~1.3%) in the split results.

> On Jan 8, 2019, at 9:44 PM, Dave Taht  wrote:
> 
> On Tue, Jan 8, 2019 at 12:03 PM Pete Heist  wrote:
>> 
>> Here’s the re-test with the patched version and HTB. Looks like success, 
>> nice work!
> 
> I note that I'm big on having the flent.gz files around also. In this
> case, by eyeball, split-gso appears to have about 130us less latency,
> but a cdf comparison of split vs no-split woud show that more easily.
> 
>> Split GSO on:
>> 
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split1.svg
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split2.svg
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split3.svg
>> 
>> Split GSO off:
>> 
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split1.svg
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split2.svg
>> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split3.svg
>> 
>> Your patch in the latest kernels looks simpler. Bringing the patch back to 
>> prior kernel versions would be appreciated, but I can understand how 3.16 
>> becomes less and less relevant as time goes on, although, it’s not at end of 
>> life yet. :)
>> 
>> Interesting how download rate control in each of the graphs with GSO 
>> splitting on looks accurate to the point where flent’s throughput graph 
>> scale is at 0.02 Mbit per step, and one can see that values coming back from 
>> netperf are probably quantized to 0.01 Mbit...
>> 
>>> On Jan 7, 2019, at 4:07 PM, Pete Heist  wrote:
>>> 
>>> Sorry, that’s without the patch, will give that a try when I have a chance 
>>> and post the results, probably tomorrow...
>>> 
 On Jan 7, 2019, at 12:30 PM, Toke Høiland-Jørgensen  wrote:
 
 Pete Heist  writes:
 
>> On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen  wrote:
>> 
>> Pete Heist  writes:
>> 
>>> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve
>>> been doing that for a while without any apparent issues, though I’m
>>> hesitating now to try that in a production environment.
>> 
>> Hmm, that's a good question. I would expect so; but I would also expect
>> the issue to show up pretty much straight away, so if you haven't hit it
>> yet, I may be wrong. I'll do some more digging... Should probably also
>> try to replicate all this stuff on my own machine :)
> 
> 
> Ok, after what I’m seeing on my APU1 tests on 3.16.7, I’m definitely
> not putting split GSO into production. I just turned it on and off
> three times and here’s what I got:
> 
> Split GSO on:
> 
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso.svg
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso2.svg
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso3.svg
> 
> Split GSO off:
> 
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso.svg
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso2.svg
> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso3.svg
> 
> I’ve seen these square waves before with htb and wondered where they
> came from, and I think we may finally have an answer! What manner of
> thing causes this I don’t know, but there’s a chance you may end up
> finding out… :)
 
 Is this without the patch to CAKE that adjusts the qlen? And have you
 tried running with that patch (with HTB)?
 
 -Toke
>>> 
>> 
>> ___
>> Cake mailing list
>> Cake@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cake
> 
> 
> 
> -- 
> 
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [PATCH 0/4] sched: Fix qdisc interactions exposed by using sch_cake as a leaf qdisc

2019-01-08 Thread David Miller 
From: Toke Høiland-Jørgensen 
Date: Mon,  7 Jan 2019 20:47:29 +0100

> This series fixes a couple of issues exposed by running sch_cake as a
> leaf qdisc in an HFSC tree, which were discovered and reported by Pete
> Heist. The interaction between CAKE's GSO splitting and the parent
> qdisc's notion of its own queue length could cause queue stalls. While
> investigating the report, I also noticed that several qdiscs would
> dereference the skb pointer after dequeue, which is potentially
> problematic since the GSO splitting code also frees the original skb.
> 
> See the individual patches in the series for details.

Toke, can you please resubmit this without patch #3.

If you want to push for patch #3 still, it is much easier to submit
it separately.
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Dave Taht 
On Tue, Jan 8, 2019 at 12:03 PM Pete Heist  wrote:
>
> Here’s the re-test with the patched version and HTB. Looks like success, nice 
> work!

I note that I'm big on having the flent.gz files around also. In this
case, by eyeball, split-gso appears to have about 130us less latency,
but a cdf comparison of split vs no-split woud show that more easily.

> Split GSO on:
>
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split1.svg
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split2.svg
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split3.svg
>
> Split GSO off:
>
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split1.svg
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split2.svg
> https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split3.svg
>
> Your patch in the latest kernels looks simpler. Bringing the patch back to 
> prior kernel versions would be appreciated, but I can understand how 3.16 
> becomes less and less relevant as time goes on, although, it’s not at end of 
> life yet. :)
>
> Interesting how download rate control in each of the graphs with GSO 
> splitting on looks accurate to the point where flent’s throughput graph scale 
> is at 0.02 Mbit per step, and one can see that values coming back from 
> netperf are probably quantized to 0.01 Mbit...
>
> > On Jan 7, 2019, at 4:07 PM, Pete Heist  wrote:
> >
> > Sorry, that’s without the patch, will give that a try when I have a chance 
> > and post the results, probably tomorrow...
> >
> >> On Jan 7, 2019, at 12:30 PM, Toke Høiland-Jørgensen  wrote:
> >>
> >> Pete Heist  writes:
> >>
>  On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen  wrote:
> 
>  Pete Heist  writes:
> 
> > Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve
> > been doing that for a while without any apparent issues, though I’m
> > hesitating now to try that in a production environment.
> 
>  Hmm, that's a good question. I would expect so; but I would also expect
>  the issue to show up pretty much straight away, so if you haven't hit it
>  yet, I may be wrong. I'll do some more digging... Should probably also
>  try to replicate all this stuff on my own machine :)
> >>>
> >>>
> >>> Ok, after what I’m seeing on my APU1 tests on 3.16.7, I’m definitely
> >>> not putting split GSO into production. I just turned it on and off
> >>> three times and here’s what I got:
> >>>
> >>> Split GSO on:
> >>>
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso.svg
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso2.svg
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso3.svg
> >>>
> >>> Split GSO off:
> >>>
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso.svg
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso2.svg
> >>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso3.svg
> >>>
> >>> I’ve seen these square waves before with htb and wondered where they
> >>> came from, and I think we may finally have an answer! What manner of
> >>> thing causes this I don’t know, but there’s a chance you may end up
> >>> finding out… :)
> >>
> >> Is this without the patch to CAKE that adjusts the qlen? And have you
> >> tried running with that patch (with HTB)?
> >>
> >> -Toke
> >
>
> ___
> Cake mailing list
> Cake@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] cake infinite loop(?) with hfsc on one-armed router

2019-01-08 Thread Pete Heist 
Here’s the re-test with the patched version and HTB. Looks like success, nice 
work!

Split GSO on:

https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split1.svg
https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split2.svg
https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_split3.svg

Split GSO off:

https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split1.svg
https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split2.svg
https://www.heistp.net/downloads/htb_split_gso_patched/htb_cakep_no_split3.svg

Your patch in the latest kernels looks simpler. Bringing the patch back to 
prior kernel versions would be appreciated, but I can understand how 3.16 
becomes less and less relevant as time goes on, although, it’s not at end of 
life yet. :)

Interesting how download rate control in each of the graphs with GSO splitting 
on looks accurate to the point where flent’s throughput graph scale is at 0.02 
Mbit per step, and one can see that values coming back from netperf are 
probably quantized to 0.01 Mbit...

> On Jan 7, 2019, at 4:07 PM, Pete Heist  wrote:
> 
> Sorry, that’s without the patch, will give that a try when I have a chance 
> and post the results, probably tomorrow...
> 
>> On Jan 7, 2019, at 12:30 PM, Toke Høiland-Jørgensen  wrote:
>> 
>> Pete Heist  writes:
>> 
 On Jan 6, 2019, at 9:56 PM, Toke Høiland-Jørgensen  wrote:
 
 Pete Heist  writes:
 
> Lastly, is using cake as a leaf to htb risky until a fix is made? I’ve
> been doing that for a while without any apparent issues, though I’m
> hesitating now to try that in a production environment.
 
 Hmm, that's a good question. I would expect so; but I would also expect
 the issue to show up pretty much straight away, so if you haven't hit it
 yet, I may be wrong. I'll do some more digging... Should probably also
 try to replicate all this stuff on my own machine :)
>>> 
>>> 
>>> Ok, after what I’m seeing on my APU1 tests on 3.16.7, I’m definitely
>>> not putting split GSO into production. I just turned it on and off
>>> three times and here’s what I got:
>>> 
>>> Split GSO on:
>>> 
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso.svg
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso2.svg
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_split_gso3.svg
>>> 
>>> Split GSO off:
>>> 
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso.svg
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso2.svg
>>> https://www.heistp.net/downloads/htb_split_gso/htb_cake_no_split_gso3.svg
>>> 
>>> I’ve seen these square waves before with htb and wondered where they
>>> came from, and I think we may finally have an answer! What manner of
>>> thing causes this I don’t know, but there’s a chance you may end up
>>> finding out… :)
>> 
>> Is this without the patch to CAKE that adjusts the qlen? And have you
>> tried running with that patch (with HTB)?
>> 
>> -Toke
> 

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake