Re: [Cake] [Bloat] cake + mpls?
If you just want to use cake with priority tins based on the MPLS "Traffic Class" (TC) field (i.e. the renamed original "EXP" field, see RFC5462), I think you can use a tc flower filter ( https://man7.org/linux/man-pages/man8/tc-flower.8.html) matching on mpls_tc values. See here for some examples: https://www.redhat.com/sysadmin/mpls-tc-linux-kernel /Jonas On Mon, Jan 10, 2022 at 7:21 PM Dave Taht wrote: > I noticed that sometime in the past 8 years the flow_dissector gained > support for dissecting mpls packets. I don't know how deep that rabbit > hole > goes. > > Over here on this mikrotik thead > https://forum.mikrotik.com/viewtopic.php?p=904422#p904422 the question > was asked about cake, the exp bits, and mpls. > > In looking over this, would we slam cake onto the vrf? or? > > https://blog.swineson.me/en/use-linux-as-an-mpls-router/ > > I have precisely zero experience with mpls. Is there an mpls expert in > the house? > > -- > I tried to build a better future, a few times: > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org > > Dave Täht CEO, TekLibre, LLC > ___ > Bloat mailing list > bl...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] separate shaping of wifi
> Aside from fixing the end device so it isn't bloated in the first place, I think you have a reasonable solution there. You might only need to limit the ingress rate on wlan0, if your router's wifi stack is debloated. Right, only limit the ingress rate, that's what I'm doing and it seems to be working but I haven't seen this type of setup being discussed anywhere. The tricky part is choosing the shaping rate since the actual wifi rate varies. There's not so much I can do to fix bloat on windows and android devices. /Jonas On Sun, Feb 3, 2019 at 11:08 PM Jonathan Morton wrote: > > On 4 Feb, 2019, at 12:04 am, Jonas Mårtensson < > martensson.jo...@gmail.com> wrote: > > > > I'm running OpenWrt with sqm on my home router. Is there any potential > problem with enabling sqm with cake on both eth0 (wan) and wlan0? The > reason for doing this is that if I only do shaping on the wan interface I > still get bad uplink bufferbloat on wifi. I assume this is because the > bottleneck in this case is actually the wifi and the bloat is in the end > device (laptop, phone, etc.). By shaping to a rate lower than the wifi > rate, the bloat disappears. Is there any better way to accomplish this than > to enable a second sqm instance on wlan0 if I don't want to sacrifice speed > for wired devices? > > Aside from fixing the end device so it isn't bloated in the first place, I > think you have a reasonable solution there. You might only need to limit > the ingress rate on wlan0, if your router's wifi stack is debloated. > > - Jonathan Morton > > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
[Cake] separate shaping of wifi
I'm running OpenWrt with sqm on my home router. Is there any potential problem with enabling sqm with cake on both eth0 (wan) and wlan0? The reason for doing this is that if I only do shaping on the wan interface I still get bad uplink bufferbloat on wifi. I assume this is because the bottleneck in this case is actually the wifi and the bloat is in the end device (laptop, phone, etc.). By shaping to a rate lower than the wifi rate, the bloat disappears. Is there any better way to accomplish this than to enable a second sqm instance on wlan0 if I don't want to sacrifice speed for wired devices? /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] State of the Cake speech...
Maybe not a changelog exactly but you can see commits to sch_cake here: https://github.com/dtaht/sch_cake/commits/master And the history for the kmod-sched-cake package in OpenWrt is here: https://git.openwrt.org/?p=openwrt/openwrt.git;a=history;f=package/kernel/kmod-sched-cake/Makefile Each new version of kmod-sched-cake just points to a more recent commit for sch_cake. /Jonas On Thu, Jan 24, 2019 at 4:13 AM Jon Pike wrote: > Ok, sorry about that one... And I'm not asking that anyone actually make a > speech. > > I was wondering how to easily find what's changing in the different > versions, as well as what versions are available where... I appreciate the > great efforts of continuing development, and the ability to watch it, it's > just a bit hard for a non developer like me to follow what's changed and > what's still coming. I haven't been able to find any changelog or even > feature list in my searching. > > Poking around with my home setup/test bed where I have a few OpenWrt > versions available... I can see the following: > > TP-Link C7, 17.01.06 > kmod cake - 2018-7-16-f3..4-1 > luci-app-sqm - 1.1.3-1 > > CI327 x86 , 18.06.01 > kmod cake - 2018-07-16-f3..4-1 > luci-app-sqm - 1.2.4-1 > > TP-Link C7, 18 + Snapshot 1-16-19 > kmod cake - 2019-01-08 > luci-app-sqm - 1.2.4-1 > > So, I can see the sqm package changing between v17 stable and v18 > stable, but the kmod package the same. And, the recent v18 snapshot with > same sqm as v18 stable, but a much more recent kmod package. > > Would be a nice thing to know what each version brings to the table, as > well as what's still coming, for cake as well as the make wifi fast > stuff... but the last one is a speech request for another group > > ___ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] inbound cake or fq_codel shaping fails on cable on netflix reno
On Mon, Jul 23, 2018 at 4:56 PM Dave Taht wrote: > Great info, thx. Using this opportunity to rant about city-wid > networks, I'd have done something so different > than what the governments and ISPs have inflicted on us, substituting > redundancy for reliability. > > I'd have used bog standard ethernet over fiber instead of gpon. > Well, in Sweden 100% of ftth connections are standard active ethernet instead of gpon (and more than 60% of fixed connections are ftth). ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] lanman2018 cake talk ideas
On Thu, Jun 21, 2018 at 9:04 AM Pete Heist wrote: > > > On Jun 21, 2018, at 5:43 AM, Dave Taht wrote: > > > > I think your "megabit myth" idea (and language) would be a very > > powerful paper and/or talk to try and hammer home in multiple venues. > > > > I might spend a slide on it at this conference, but it deserves more > > focus than that. > > Ok, I’ve finally got a few freer days coming up, so I’ll see if I can’t > make progress on backed up tasks, then maybe write something up on this. > Might retreat from the lists meanwhile. > > On slide #4: Ubiquity -> Ubiquiti. And consider adding Verizon. My mother > has Verizon Fios (50Mbit symmetric fiber) near Philly and that was the > connection I referred to earlier- it just doesn’t feel like 50Mbit > symmetric fiber should. > > As a side note, Google as a whole seems responsible as far as the big four > go when it comes to bloat (and other things). BBR advanced the state of > things, and they financed it after all, so it deserves some appreciation, > to the devs as well. Worth mentioning in this preso? :) > > I like the Cake vs Sonic Fiber slide, and would like to hear the > commentary. In case they video the presentation, do post a link. Best of > luck! > I like the slide too but what exactly do you mean by "Cake gets inside the GPON request/grant loop"? Now, even more impressive would be a Cake vs Google Fiber plot, showing a reduction of latency under load from ~2000ms to a few ms. ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] CAKE upstreaming - testers wanted, ACK filtering rescuers needed
I thought the discussion was only about GSO/TSO. Also, isn't GRO/LRO incompatible with routing? Anyway, I was just supporting your interpretation of what Eric potentially means. /Jonas On Thu, Apr 26, 2018 at 10:55 AM, Toke Høiland-Jørgensen wrote: > Jonas Mårtensson writes: > > > "I *think* that what Eric means is that the GSO logic should > automatically > > size the GSO superpackets so the latency cost is negligible for the > actual > > link rate." > > > > Something like this? > > > > https://lwn.net/Articles/564979/ > > > > https://lwn.net/Articles/564978/ > > Yeah, that's for TCP on the local host. I'm not sure how things work for > GRO on a router (which is more relevant for the CAKE use case). > > -Toke > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] CAKE upstreaming - testers wanted, ACK filtering rescuers needed
"I *think* that what Eric means is that the GSO logic should automatically size the GSO superpackets so the latency cost is negligible for the actual link rate." Something like this? https://lwn.net/Articles/564979/ https://lwn.net/Articles/564978/ /Jonas On Thu, Apr 26, 2018 at 9:34 AM, Toke Høiland-Jørgensen wrote: > Kevin Darbyshire-Bryant writes: > > >> On 25 Apr 2018, at 21:45, Toke Høiland-Jørgensen wrote: > >> > >> For those who have not been following the discussion on the upstreaming > >> patches, here's an update: > >> > >> > >> > >> So please do test the current git version (cobalt branch, still). I'm > >> planning to resubmit on Friday. > > > > The two routers running that latest code survived the night which is a > > good sign. > > Awesome! > > > I’ve sort of half been following the ‘discussion’, as ever the > > reaction from the kernel people makes it a place I never wish to > > look/contribute, ….. this from Eric has me disturbed "If you keep > > saying this old urban legend, I will NACK your patch.I am tired of > > people pretending GSO/TSO are bad for latencies.” > > Heh, yeah, the tone on kernel lists can be a bit... abrasive... just > smile and wave and ignore the vitriol is my approach. But I can totally > understand why some people don't want to put up with it... :) > > > Genuine question: I have a superpacket circa 64K, this is a lump of > > data in a tcp flow. I have another small VOIP packet, it’s latency > > sensitive. If I split the super packet into individual 1.5K packets > > as they would be on the wire, I can insert my VOIP packet at suitable > > place in time such that jitter targets are not exceeded. If I don’t > > split the super packet, surely I have to wait till the end of the > > superpacket’s queue (for want of a better word) and possibly exceed my > > latency target. That looks to me like ‘GSO/TSO’ is potentially bad > > for interflow latencies. What don’t I understand here? > > You are right in principle, of course. I *think* that what Eric means is > that the GSO logic should automatically size the GSO superpackets so the > latency cost is negligible for the actual link rate. I was actually > thinking I would do some measurements at some point to test this at > various rates; since we have a nice piece of code that can adaptively > split GSO packets that should be pretty straight-forward :) > > -Toke > ___ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Testing variants of the MTU latency scaling
On Tue, Apr 24, 2018 at 9:22 PM, Kevin Darbyshire-Bryant < ke...@darbyshire-bryant.me.uk> wrote: > Assuming you’re using luci to configure then enabling both show and use > advanced configuration & show and use dangerous configurations… then enter > ‘ingress’ in the ‘advanced option string to pass to ingress queuing’ will > enable ingress mode. > > Maybe that helps? Just typing ingress in that box seems to disable cake and instead sets default fq_codel as ingress qdisc. /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Testing variants of the MTU latency scaling
On Tue, Apr 24, 2018 at 1:45 PM, Toke Høiland-Jørgensen wrote: > Jonas Mårtensson writes: > > > One thing that is still not clear to me from these results: if I run > > cake on an IFB without ingress mode (i.e. the default?), does the MTU > > scaling have any impact on TCP download throughput? > > Odds are that not using ingress mode will make Cake lose control of the > bottleneck (that is what happened when I tried running a quick test), > and so will mess up both latency and throughput as you hit the bloated > upstream link buffer... So using cake through sqm-scripts in OpenWRT/LEDE for ingress shaping does not currently work very well then? I guess the sqm-scripts should be updated to actually use ingress mode at some point... /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Testing variants of the MTU latency scaling
One thing that is still not clear to me from these results: if I run cake on an IFB without ingress mode (i.e. the default?), does the MTU scaling have any impact on TCP download throughput? /Jonas On Sun, Apr 22, 2018 at 11:09 PM, Jonathan Morton wrote: > > Takeaways (see attached plots): > > > > - The MTU scaling does indeed give a nice benefit in egress mode > > "tcp-download-totals" plot. From just over 6 Mbps to just over 8 Mbps > > of goodput on the 10 Mbit link. There is not a large difference > > between 2MTU and 4MTU, except that 4MTU hurts inter-flow latency > > somewhat. > > > > - The effect for upload flows (where Cake is before the bottleneck; > > 10mbit-upload.png) is negligible. > > > > - The MTU scaling really hurts TCP RTT (intra-flow latency; > > tcp-upload-tcprtt-10mbit.png and rrul-tcprtt.png). > > > > - For bidirectional traffic the combined effect is also negligible. > > > > > > Based on all this, I propose we change the scaling mechanism so that it > > is only active in egress mode, and change it from 4 MTUs to 2. I'll > > merge Kevin's patch to do this unless someone complains loudly :) > > > > If you want me to run other tests, let me know. > > I'm not actually sure what you've measured here - unless you've somehow > managed to swap "ingress" with "egress" mode in a strange manner. I don't > see any systematic measurement of the different MTU scales in ingress mode > in your results, which makes your assertion that it should only be active > in egress mode rather odd. > > - Jonathan Morton > > ___ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] A few puzzling Cake results
On Wed, Apr 18, 2018 at 6:54 PM, Jonathan Morton wrote: > > On 18 Apr, 2018, at 7:11 pm, Toke Høiland-Jørgensen > wrote: > > > > What you're saying here is that you basically don't believe there are > > any applications where a bulk TCP flow would also want low queueing > > latency? :) > > I'm saying that there's a tradeoff between intra-flow induced latency and > packet loss, and I've chosen 4 MTUs as the operating point. Jonathan, in the last commit you went from 1 MTU to 4 MTUs. Stupid question maybe but did you also test 2 and 3 MTUs? /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] A few puzzling Cake results
Dave, in the thread referenced earlier that led to this change you said: "The loss of throughput here compared to non-ingress mode is a blocker for mainlining and for that matter, wedging this into lede." I'm curious, what would the latency be in Toke's experiment with non-ingress mode and with the 4 MTU change reverted? The same as for fq_codel? On Wed, Apr 18, 2018 at 7:02 PM, Dave Taht wrote: > Jonathan: > > I think you are wrong. What we care about is keeping packets in flight > across the network, with a queue length as close to 1 packet as > possible. > > If it breaks ingress mode so be it. > > > On Wed, Apr 18, 2018 at 9:54 AM, Jonathan Morton > wrote: > >> On 18 Apr, 2018, at 7:11 pm, Toke Høiland-Jørgensen > wrote: > >> > >> What you're saying here is that you basically don't believe there are > >> any applications where a bulk TCP flow would also want low queueing > >> latency? :) > > > > I'm saying that there's a tradeoff between intra-flow induced latency > and packet loss, and I've chosen 4 MTUs as the operating point. > > > > Bear in mind that with high packet loss, the retransmissions take an > extra RTT to complete in any case, and there's a higher probability of > incurring an RTO which will *really* hurt your intra-flow latency. > > > > This equation is modified with ECN because a high signalling rate > doesn't result in packet loss or retransmissions, but I'm not presently > making any decisions based on ECN support, except the obvious one of > whether to mark or drop. > > > > - Jonathan Morton > > > > ___ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > > > -- > > Dave Täht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 > ___ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] A few puzzling Cake results
On Wed, Apr 18, 2018 at 1:25 PM, Toke Høiland-Jørgensen wrote: > Toke Høiland-Jørgensen writes: > > > Jonathan Morton writes: > > > >>> On 17 Apr, 2018, at 12:42 pm, Toke Høiland-Jørgensen > wrote: > >>> > >>> - The TCP RTT of the 32 flows is *way* higher for Cake. FQ-CoDel > >>> controls TCP flow latency to around 65 ms, while for Cake it is all > >>> the way up around the 180ms mark. Is the Codel version in Cake too > >>> lenient, or what is going on here? > >> > >> A recent change was to increase the target dynamically so that at > >> least 4 MTUs per flow could fit in each queue without AQM activity. > >> That should improve throughput in high-contention scenarios, but it > >> does come at the expense of intra-flow latency when it's relevant. > > > > Ah, right, that might explain it. In the 128 flow case each flow has > > less than 100 Kbps available to it, so four MTUs are going to take a > > while to dequeue... > > OK, so I went and looked at the code and found this: > > bool over_target = sojourn > p->target && >sojourn > p->mtu_time * bulk_flows * 4; > > > Which means that we scale the allowed sojourn time for each flow by the > time of four packets *times the number of bulk flows*. > > So if there is one active bulk flow, we allow each flow to queue four > packets. But if there are ten active bulk flows, we allow *each* flow to > queue *40* packets. I'm confused. Isn't the sojourn time for a packet a result of the total number of queued packets from all flows? If each flow were allowed to queue 40 packets, the sojourn time would be mtu_time * bulk_flows * 40, no? ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] A few puzzling Cake results
On Tue, Apr 17, 2018 at 2:22 PM, Toke Høiland-Jørgensen wrote: > Y via Cake writes: > > > From: Y > > Subject: Re: [Cake] A few puzzling Cake results > > To: cake@lists.bufferbloat.net > > Date: Tue, 17 Apr 2018 21:05:12 +0900 > > > > Hi. > > > > Any certain fomula of fq_codel flow number? > > Well, given N active bulk flows with packet size L, and assuming the > quantum Q=L (which is the default for FQ-CoDel at full-size 1500-byte > packets), the maximum rate for a sparse flow, R_s, is bounded by > > R_s < R / ((L/L_s)(N+1)) > > Where R is the link rate and L_s is the packet size of the sparse flow. > This assumes that the sparse flow has constant spacing between its > packets, which is often the case for a VoIP flow... For 10-Mbit/s link rate and 32 bulk flows with 1500-byte packets this formula gives roughly 25 pps (packets per second) as maximum for a sparse flow. A VoIP flow is typically 50 pps (20 ms voice payload). Does this mean that cake sets the quantum to less than 750 bytes for a 10-Mbit/s link? Do you see any benefit with cake diffserv if you increase the number of flows? Does the adjusted quantum also explain the "*way* higher" TCP RTT for cake? How? /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?
Well, simplest.qos.help says "Simplest possible configuration: HTB rate limiter with your qdisc attached" so that is probably also a bit misleading. /Jonas On Wed, Apr 11, 2018 at 9:30 PM, Sebastian Moeller wrote: > Sorry, > > just looked at the code and my recollection is wrong. I could have sworn > that I purged cake as a shaper from simple.qos when I created > piece_of_cake, but apparently that was just a fever dream... > > Sorry for the noise. > > > > On Apr 11, 2018, at 21:26, Sebastian Moeller wrote: > > > > > > > > On April 11, 2018 8:55:12 PM GMT+02:00, "Jonas Mårtensson" < > martensson.jo...@gmail.com> wrote: > >> On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen > >> wrote: > >> > >>> Jonathan Morton writes: > >>> > >>>>> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen > >>> wrote: > >>>>> > >>>>> So, um, did we cram so many features into Cake that it no longer > >> uses > >>>>> less CPU? Can anyone confirm these results? > >>>> > >>>> To be sure about this, it seems wise to configure Cake to turn off > >> as > >>>> many of the new features as possible. That means selecting > >> "besteffort > >>>> flows nonat" mode at least. > >>>> > >>>> I forget whether simplest.qos correctly uses the built-in shaper > >> with > >>>> Cake, rather than just layering it with HTB as usual. If not, then > >> of > >>>> course Cake will use more CPU, and we should be grateful that it's > >> by > >>>> a relatively small margin (maybe 15%). > >>> > >>> It is definitely using Cake as the shaper; in besteffort mode, but > >> with > >>> nat and triple-isolation enabled I think. I'll run another test > >> tomorrow > >>> with those disabled. > >> > >> > >> Is there any difference between using simplest.qos and > >> piece_of_cake.qos > >> when Cake is used as qdisc? > > > > Yes, IIRC simplest.qos with cake as qdisc will use HTB as shaper and > cake as leaf qdisc, while piece_of_cake.qos will use cake both as shaper > and leaf qdisc. The former is only useful for comparative testing, for > actual usage I recommend the later. > > > > Best Regards > >Sebastian > > > > > >> > >> /Jonas > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > ___ > > Cake mailing list > > Cake@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/cake > > ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake not more CPU efficient than HTB+FQ-CoDel (anymore)?
On Wed, Apr 11, 2018 at 7:15 PM, Toke Høiland-Jørgensen wrote: > Jonathan Morton writes: > > >> On 11 Apr, 2018, at 6:24 pm, Toke Høiland-Jørgensen > wrote: > >> > >> So, um, did we cram so many features into Cake that it no longer uses > >> less CPU? Can anyone confirm these results? > > > > To be sure about this, it seems wise to configure Cake to turn off as > > many of the new features as possible. That means selecting "besteffort > > flows nonat" mode at least. > > > > I forget whether simplest.qos correctly uses the built-in shaper with > > Cake, rather than just layering it with HTB as usual. If not, then of > > course Cake will use more CPU, and we should be grateful that it's by > > a relatively small margin (maybe 15%). > > It is definitely using Cake as the shaper; in besteffort mode, but with > nat and triple-isolation enabled I think. I'll run another test tomorrow > with those disabled. Is there any difference between using simplest.qos and piece_of_cake.qos when Cake is used as qdisc? /Jonas ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
