Re: [Bloat] SQM tuning question
Hi Aaron, > On Jun 3, 2023, at 20:04, Aaron Wood via Bloat > wrote: > > I’ve found that _usually_ I can set cake’s bandwidth limits to 90-95% of the > advertised bandwidth, and everything “just works”. So long as you’re > routinely able to achieve the bandwidth, it tends to work. > > I’ve found in my testing over the years (I’ve been a user of fq_codel since > 2013) that limiting the upstream bandwidth is the most important one to do. > Downstream bandwidth limits, especially if you have well above 100Mbps > downstream aren’t as critical, but it still can make a big difference if you > routinely saturate your downstream connection. > > It’s especially important to manage the upstream bandwidth when the > connection is asymmetric. For instance my 1Gbps Comcast internet service > only has a 35Mbps upstream limit, which is just a bit more than you need for > the acks that go back upstream in response to 1Gbps of downstream traffic. > > But if your connection quality suffers with time of day (congestion on shared > local ISP infrastructure), or due to weather (radio links can degrade during > the rain), then that can be a bit more annoying. > > But as I said, I’ve found with my Comcast connection that I can just set my > bandwidth limits to 90-95% of advertised and it just works. > > The other thing to look into is DNS. Most of the weird pauses that I see > are, DNS related. I e switches to using a local DNS cache/lookup-forwarding > server, and use DNS over HTTPS to parallel requests to both cloudflare and > google’s HTTPS DNS proxies. That seems to be more stable (long-lived > connections with faster recovery from a dropped packed than UDP has). > > (More continued below) > > On Sat, Jun 3, 2023 at 10:17 AM John D via Bloat > wrote: > Thanks for the detail. It makes sense but it kind of feels like in some > (maybe many) cases the router could know the internet link performance. > Particularly home router-modems often monitor this already. Maybe that's just > not exposed in any standardised way? > > There are two issues I’ve seen here: > > 1) there’s not a standardized way (or even usually any api) for getting the > provisioned rate, or the current link speed, from the modem portion of things. > > 2) local congestion at the ISP can degrade things below the provisioned > rates, and that’s even harder to detect or have available. > > If your upstream connection is DOCSIS 3.1, you should have the PIE AQM > running in the modem, which _should_ help considerably. But at least at my > cable headend, while downstream runs DOCSIS 3.1, upstream is only DOCSIS 3.0, > and definitely isn’t using PIE. > > The PIE AQM, running on the modem itself has the advantage of running on what > is almost always the bottle neck for upstream traffic in my experience: the > provisioned upstream bandwidth in the modem. [SM] Even better, the cable modem requests mini-slots from the scheduler and really has control over the true upstream queue, so even in an overloaded segment (where mini-slot scheduling gets slow) the PIE instance sees the relevant queueing delay to do its thing even if the current rate is well below the "provisioned" rate (which is just a maximum peak rate anyway). Now this will only help for congestion in the DOCSIS segment, as you said that is likely to be the most likely cause for upstream delay anyways... > > I'm guessing if I was into openwrt I could maybe do something, but I prefer > just to find something off the shelf with half decent SQM... If "auto > configuration" isn't a feature then that answers my question and I can get on > choosing the best option. > > I’m not aware of any decent “off the shelf” solutions that can track > bandwidth correctly, aside from DOCSIS 3.1 modems (which still requires the > cable company to provision it correctly to enable it) > > But, as I said, if your connection is stable, you can get the majority of the > benefits just by setting it up once and leaving it be. > > I’ve only tweaked things 1-2 times a year, for the last three years, and that > was only because I was moving. Once I had setup a router that could run cake > at my line rates, I’ve just left it there and it’s been fine. > > Especially since for me, WiFi is my usual bottleneck (my APs only do about > 250-300Mbps) > > -Aaron > > > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat > > wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and > > RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, users, > > or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it keeps > > the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link >
Re: [Bloat] SQM tuning question
Hi John, > On Jun 3, 2023, at 19:17, John D via Bloat > wrote: > > Thanks for the detail. It makes sense but it kind of feels like in some > (maybe many) cases the router could know the internet link performance. Sometimes they do, sometimes they do not, and sometimes the number the router might now is well above the contractual limit, e.g. my ISP for some time configured my VDSL2 link to sync at ~100/40 Mbps and restrict my capacity at an upstream device (BNG) to my contracted ~50/10 Mbps, so knowing the link speed will only give you an upper bound for some link technologies. > Particularly home router-modems often monitor this already. Maybe that's just > not exposed in any standardised way? Yes and no, all the link technologies I looked at (DSL/DOCSIS/PON) have some channel between the ISP side equipment (DSLAM/CMTS/OLT) and the equipment on the user side (CPE: DSL-/cable-/pon-modem) by which the ISP gear can query information from the CPE; but these three seem to be using three different approaches, and more importantly, just because the head-end can query these, does not mean that devices in the home network can do so as well... Plus, as mentioned above, the reported "sync" capacity might not be the capacity required to configure sqm's traffic shaper. > I'm guessing if I was into openwrt I could maybe do something, but I prefer > just to find something off the shelf with half decent SQM... If "auto > configuration" isn't a feature then that answers my question and I can get on > choosing the best option. There are the work-in-progress *-autorate approaches mentioned in my earlier post... Kind Regards Sebastian > > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat > > wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and > > RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, users, > > or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it keeps > > the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link > > rates, and traffic loads. > > • It adapts to dynamically changing link rates with no negative > > impact on utilization. > > > > But everywhere I have read about about hardware which implements SQM > > (including the bufferbloat website) it describes the need to tune based on > > actual internet connection speed. > > These seem to conflict especially that "handles variable bandwidth" bit. > > Have I misunderstood or do the algorithms used in modern hardware just not > > provide this part typically? My connection performance is quite variable > > and I'm worried about crippling SQM to the lowest speed seen. > > SQM in practice requires three components: > > 1: Flow isolation, so that different flows don't affect each others' latency > and are delivered fairly; > > 2: Active Queue Management (AQM) to signal flows to slow down transmissions > when link capacity is exceeded; > > 3: Bandwidth shaping to match the queue to the available capacity. > > CoDel is, in itself, only the AQM component. It does indeed work pretty well > with no additional tuning - but only in combination with the other two > components, or when applied directly to the actual bottleneck. Unfortunately > in most consumer internet links, the actual bottleneck is inaccessible for > this purpose. Thus an artificial bottleneck must be introduced, at which SQM > is applied. > > The most convenient tool for applying all three SQM components at once is > Cake. This includes implementations of advanced flow isolation, CoDel AQM, > and a deficit-mode bandwidth shaper. All you really need to do is to tell it > how much bandwidth you have in each direction, minus a small margin to ensure > it becomes the actual bottleneck and can exert the necessary control. > > When your available bandwidth varies over time, that can be inconvenient. > There are methods, however, of observing how available capacity tends to > change over time (typically on diurnal and weekly patterns, if the variations > are due to congestion in the ISP backhaul or peering) and scheduling > adjustments on that basis. If you have more information on your situation, > we might be able to give more detailed advice. > > - Jonathan Morton > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] SQM tuning question
Have a look at: cake-autorate: https://github.com/lynxthecat/cake-autorate/blob/master/cake-autorate.sh lua sqm-autorate: https://github.com/sqm-autorate/sqm-autorate perl sqm-autorate: https://github.com/tievolu/sqm-autorate all three use active latency probes to detect an increase in delay and interpret that as increase in queueing delay which happens if the offered load exceeds the available capacity, and all three will, by slightly different heuristics, adjust the traffic shaper so that latency under loads stays (mostly) within acceptable (configurable) limits. Note the "organic" traffic over the link is measured and used as "load" so no cyclic saturating speedtest is necessary. I think that evenroute's iqrouter uses a different approach in that they periodically perform speedtests and then set the traffic shaper in accordance to that result. However the three different *-autorates should be runnable on essentially every linux distribution, while for evenroute's implementation one would need to buy their hardware IIRC. Best Regards Sebastian > On Jun 3, 2023, at 18:54, Kenneth Porter via Bloat > wrote: > > --On Saturday, June 03, 2023 7:44 PM +0300 Jonathan Morton via Bloat > wrote: > >> When your available bandwidth varies over time, that can be inconvenient. >> There are methods, however, of observing how available capacity tends to >> change over time (typically on diurnal and weekly patterns, if the >> variations are due to congestion in the ISP backhaul or peering) and >> scheduling adjustments on that basis. If you have more information on >> your situation, we might be able to give more detailed advice. > > Are there any good solutions for regularly recomputing the bandwidth on a > consumer link? I'm running the Linux SQM scripts to launch cake on a recent > Debian system and CoDel on a CentOS 7 gateway. I set the bandwidth manually > based on periodic manual speed tests. I'd love to automate that. > > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] SQM tuning question
I’ve found that _usually_ I can set cake’s bandwidth limits to 90-95% of the advertised bandwidth, and everything “just works”. So long as you’re routinely able to achieve the bandwidth, it tends to work. I’ve found in my testing over the years (I’ve been a user of fq_codel since 2013) that limiting the upstream bandwidth is the most important one to do. Downstream bandwidth limits, especially if you have well above 100Mbps downstream aren’t as critical, but it still can make a big difference if you routinely saturate your downstream connection. It’s especially important to manage the upstream bandwidth when the connection is asymmetric. For instance my 1Gbps Comcast internet service only has a 35Mbps upstream limit, which is just a bit more than you need for the acks that go back upstream in response to 1Gbps of downstream traffic. But if your connection quality suffers with time of day (congestion on shared local ISP infrastructure), or due to weather (radio links can degrade during the rain), then that can be a bit more annoying. But as I said, I’ve found with my Comcast connection that I can just set my bandwidth limits to 90-95% of advertised and it just works. The other thing to look into is DNS. Most of the weird pauses that I see are, DNS related. I e switches to using a local DNS cache/lookup-forwarding server, and use DNS over HTTPS to parallel requests to both cloudflare and google’s HTTPS DNS proxies. That seems to be more stable (long-lived connections with faster recovery from a dropped packed than UDP has). (More continued below) On Sat, Jun 3, 2023 at 10:17 AM John D via Bloat < bloat@lists.bufferbloat.net> wrote: > Thanks for the detail. It makes sense but it kind of feels like in some > (maybe many) cases the router could know the internet link performance. > Particularly home router-modems often monitor this already. Maybe that's > just not exposed in any standardised way? > There are two issues I’ve seen here: 1) there’s not a standardized way (or even usually any api) for getting the provisioned rate, or the current link speed, from the modem portion of things. 2) local congestion at the ISP can degrade things below the provisioned rates, and that’s even harder to detect or have available. If your upstream connection is DOCSIS 3.1, you should have the PIE AQM running in the modem, which _should_ help considerably. But at least at my cable headend, while downstream runs DOCSIS 3.1, upstream is only DOCSIS 3.0, and definitely isn’t using PIE. The PIE AQM, running on the modem itself has the advantage of running on what is almost always the bottle neck for upstream traffic in my experience: the provisioned upstream bandwidth in the modem. I'm guessing if I was into openwrt I could maybe do something, but I prefer > just to find something off the shelf with half decent SQM... If "auto > configuration" isn't a feature then that answers my question and I can get > on choosing the best option. > I’m not aware of any decent “off the shelf” solutions that can track bandwidth correctly, aside from DOCSIS 3.1 modems (which still requires the cable company to provision it correctly to enable it) But, as I said, if your connection is stable, you can get the majority of the benefits just by setting it up once and leaving it be. I’ve only tweaked things 1-2 times a year, for the last three years, and that was only because I was moving. Once I had setup a router that could run cake at my line rates, I’ve just left it there and it’s been fine. Especially since for me, WiFi is my usual bottleneck (my APs only do about 250-300Mbps) -Aaron > On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > >> > On 3 Jun, 2023, at 4:56 pm, John D via Bloat < >> bloat@lists.bufferbloat.net> wrote: >> > >> > On the website it says the following: >> > >> > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth >> and RTT”, and simple AQM algorithm. >> > >> > • It is parameterless — no knobs are required for operators, >> users, or implementers to adjust. >> > • It treats good queue and bad queue differently - that is, it >> keeps the delays low while permitting bursts of traffic. >> > • It controls delay, while insensitive to round-trip delays, link >> rates, and traffic loads. >> > • It adapts to dynamically changing link rates with no negative >> impact on utilization. >> > >> > But everywhere I have read about about hardware which implements SQM >> (including the bufferbloat website) it describes the need to tune based on >> actual internet connection speed. >> > These seem to conflict especially that "handles variable bandwidth" >> bit. Have I misunderstood or do the algorithms used in modern hardware just >> not provide this part typically? My connection performance is quite >> variable and I'm worried about crippling SQM to the lowest speed seen. >> >> SQM in practice requires three components: >> >> 1: Flow isolation, so that different flows
Re: [Bloat] SQM tuning question
Thanks for the detail. It makes sense but it kind of feels like in some (maybe many) cases the router could know the internet link performance. Particularly home router-modems often monitor this already. Maybe that's just not exposed in any standardised way? I'm guessing if I was into openwrt I could maybe do something, but I prefer just to find something off the shelf with half decent SQM... If "auto configuration" isn't a feature then that answers my question and I can get on choosing the best option. On Sat, Jun 3, 2023, 16:44 Jonathan Morton wrote: > > On 3 Jun, 2023, at 4:56 pm, John D via Bloat < > bloat@lists.bufferbloat.net> wrote: > > > > On the website it says the following: > > > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth > and RTT”, and simple AQM algorithm. > > > > • It is parameterless — no knobs are required for operators, > users, or implementers to adjust. > > • It treats good queue and bad queue differently - that is, it > keeps the delays low while permitting bursts of traffic. > > • It controls delay, while insensitive to round-trip delays, link > rates, and traffic loads. > > • It adapts to dynamically changing link rates with no negative > impact on utilization. > > > > But everywhere I have read about about hardware which implements SQM > (including the bufferbloat website) it describes the need to tune based on > actual internet connection speed. > > These seem to conflict especially that "handles variable bandwidth" bit. > Have I misunderstood or do the algorithms used in modern hardware just not > provide this part typically? My connection performance is quite variable > and I'm worried about crippling SQM to the lowest speed seen. > > SQM in practice requires three components: > > 1: Flow isolation, so that different flows don't affect each others' > latency and are delivered fairly; > > 2: Active Queue Management (AQM) to signal flows to slow down > transmissions when link capacity is exceeded; > > 3: Bandwidth shaping to match the queue to the available capacity. > > CoDel is, in itself, only the AQM component. It does indeed work pretty > well with no additional tuning - but only in combination with the other two > components, or when applied directly to the actual bottleneck. > Unfortunately in most consumer internet links, the actual bottleneck is > inaccessible for this purpose. Thus an artificial bottleneck must be > introduced, at which SQM is applied. > > The most convenient tool for applying all three SQM components at once is > Cake. This includes implementations of advanced flow isolation, CoDel AQM, > and a deficit-mode bandwidth shaper. All you really need to do is to tell > it how much bandwidth you have in each direction, minus a small margin to > ensure it becomes the actual bottleneck and can exert the necessary control. > > When your available bandwidth varies over time, that can be inconvenient. > There are methods, however, of observing how available capacity tends to > change over time (typically on diurnal and weekly patterns, if the > variations are due to congestion in the ISP backhaul or peering) and > scheduling adjustments on that basis. If you have more information on your > situation, we might be able to give more detailed advice. > > - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] SQM tuning question
--On Saturday, June 03, 2023 7:44 PM +0300 Jonathan Morton via Bloat wrote: When your available bandwidth varies over time, that can be inconvenient. There are methods, however, of observing how available capacity tends to change over time (typically on diurnal and weekly patterns, if the variations are due to congestion in the ISP backhaul or peering) and scheduling adjustments on that basis. If you have more information on your situation, we might be able to give more detailed advice. Are there any good solutions for regularly recomputing the bandwidth on a consumer link? I'm running the Linux SQM scripts to launch cake on a recent Debian system and CoDel on a CentOS 7 gateway. I set the bandwidth manually based on periodic manual speed tests. I'd love to automate that. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] SQM tuning question
> On 3 Jun, 2023, at 4:56 pm, John D via Bloat > wrote: > > On the website it says the following: > > CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and > RTT”, and simple AQM algorithm. > > • It is parameterless — no knobs are required for operators, users, or > implementers to adjust. > • It treats good queue and bad queue differently - that is, it keeps > the delays low while permitting bursts of traffic. > • It controls delay, while insensitive to round-trip delays, link > rates, and traffic loads. > • It adapts to dynamically changing link rates with no negative impact > on utilization. > > But everywhere I have read about about hardware which implements SQM > (including the bufferbloat website) it describes the need to tune based on > actual internet connection speed. > These seem to conflict especially that "handles variable bandwidth" bit. Have > I misunderstood or do the algorithms used in modern hardware just not provide > this part typically? My connection performance is quite variable and I'm > worried about crippling SQM to the lowest speed seen. SQM in practice requires three components: 1: Flow isolation, so that different flows don't affect each others' latency and are delivered fairly; 2: Active Queue Management (AQM) to signal flows to slow down transmissions when link capacity is exceeded; 3: Bandwidth shaping to match the queue to the available capacity. CoDel is, in itself, only the AQM component. It does indeed work pretty well with no additional tuning - but only in combination with the other two components, or when applied directly to the actual bottleneck. Unfortunately in most consumer internet links, the actual bottleneck is inaccessible for this purpose. Thus an artificial bottleneck must be introduced, at which SQM is applied. The most convenient tool for applying all three SQM components at once is Cake. This includes implementations of advanced flow isolation, CoDel AQM, and a deficit-mode bandwidth shaper. All you really need to do is to tell it how much bandwidth you have in each direction, minus a small margin to ensure it becomes the actual bottleneck and can exert the necessary control. When your available bandwidth varies over time, that can be inconvenient. There are methods, however, of observing how available capacity tends to change over time (typically on diurnal and weekly patterns, if the variations are due to congestion in the ISP backhaul or peering) and scheduling adjustments on that basis. If you have more information on your situation, we might be able to give more detailed advice. - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
[Bloat] SQM tuning question
Hi, On the website it says the following: *CoDel is a novel “no knobs”, “just works”, “handles variable bandwidth and RTT”, and simple AQM algorithm.* - *It is parameterless — no knobs are required for operators, users, or implementers to adjust.* - *It treats good queue and bad queue differently - that is, it keeps the delays low while permitting bursts of traffic.* - *It controls delay, while insensitive to round-trip delays, link rates, and traffic loads.* - *It adapts to dynamically changing link rates with no negative impact on utilization.* But everywhere I have read about about hardware which implements SQM (including the bufferbloat website) it describes the need to tune based on actual internet connection speed. These seem to conflict especially that "handles variable bandwidth" bit. Have I misunderstood or do the algorithms used in modern hardware just not provide this part typically? My connection performance is quite variable and I'm worried about crippling SQM to the lowest speed seen. Thanks for any help. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
