Re: [Bloat] [Cerowrt-devel] tp-link request for SQM

[Luca Muscariello via Bloat]

On Thu, Dec 9, 2021, 19:23 Sebastian Moeller  wrote:

> Hi Luca,
>
>
> > On Dec 9, 2021, at 18:38, Luca Muscariello  wrote:
> >
> > Hi Sebastian
> >
> > On Thu, Dec 9, 2021, 17:09 Sebastian Moeller  wrote:
> > Hi Luca
> >
> > > On Dec 3, 2021, at 15:58, Luca Muscariello 
> wrote:
> > >
> > >
> > >
> > > On Fri, Dec 3, 2021 at 3:35 PM Sebastian Moeller 
> wrote:
> > > Hi Dave,
> > >
> > >
> > > > On Dec 3, 2021, at 15:18, Dave Taht  wrote:
> > > >
> > > > On Fri, Dec 3, 2021 at 4:00 AM Luca Muscariello <
> muscarie...@ieee.org> wrote:
> > > >>
> > > >> Test using a tp-link AP EAP 245
> > > >>
> > > >>
> https://www.waveform.com/tools/bufferbloat?test-id=bbcc5ef5-e677-4f27-aa04-1849db81d0f5
> > > >
> > > > Nice.
> > > >
> > > > A kvetch is that I really wish they also tested up and down at the
> same time.
> > > >
> > > > Another kvetch is I think the test needs to run longer at these
> speeds.
> > > >
> > > > Another another kvetch is they factor in baseline latency to
> determine
> > > > if the link is suitable for gaming or not.
> > >
> > > Which is the sane thing to do... IMHO. For any "twitch"-type
> reation-time gated gaming all players need to be in an acceptable range of
> "RTTs" to and from the server so that all perceive the world similarly and
> nobody has an unfair advantage/disadvantage, so absolute RTT does seem to
> matter. I would agree that jitter is nastier in that is will cause
> "randomish" variations of the RTT, but then the known solution against
> jitter is additional buffering (large enough to simply even out the unequal
> jittered packet arrival times) which in turn just increases the "RTT", no?
> (I guess no game really does this enough, so jitter stays the constant
> problem for internet game-play).
> > > >
> > >
> > > Mobile multiplayer competitive games, like PvP, 5v5 such as CoDM,
> Critical Ops, LoL Wild Rift may have very different ways to face/conceal
> network conditions based on the experience the studio wants to offer to the
> gamer.
> >
> > Care to elaborate, please? As far as I can tell what I describe
> above is pretty universal iff fair "coordination" between different
> player's actions is required. How do you conceal issues about causal
> ordering of events that ideally are identical from all perspectives?
> >
> > Not all games use deterministic lockstep because it requires to
> compensate latency from the different players. Of course if latency is zero
> it is easier. But that assumption or that objective is one of the fallacies
> of distributed computing.
> >
> > Predictive lockstep or even rollback netcode are examples where you fix
> update  accuracy reactively.
>
> Ah, sure, I was talking core principle not implementation. I fully
> agree that predictive/speculative techniques help to make the causality
> problems of different information propagation delays appear less often. In
> retrospect that is what "conceal network conditions" alstready covered ;)
>
>
> > Beyond specific netcode techniques, some mobile games can give you the
> illusion that you're playing PvP while at some point reality is that you
> can continue to play almost offline against bots.
>
> I don't want to sound harsh, but that feels like "cheating".
>

That's what I was referring to when I wrote that the game studio works on
the experience they want to provide. If the game is a competitive game with
esport competitions where accuracy is crucial that has implications on the
way the netcode is written.

If the studio is looking for gamer retention it's important the gamer with
little experience is able to learn, get more skilled and have fun along the
way before they get to the competitive level.

BTW, it is not cheating. It is fishing in some sense.


> >
> > There are games where physics inaccuracies can be concealed. Physics is
> very important for first person shooter games while an ARPG may conceal
> inaccuracies in the middle of the mess that is rendered on screen, think of
> League of Legends Wild Rift.
>
> Ah, okay. Didn't know at all about the last two ;)
>
> Thanks!
>
> Regards
> Sebastan
>
>
> >
> >
> >
> > Regards
> > Sebastian
> >
> >
> > >
> > >
> > >
> > > > We had them participating on this 

Re: [Bloat] [Cerowrt-devel] tp-link request for SQM

[Luca Muscariello via Bloat]

Hi Sebastian

On Thu, Dec 9, 2021, 17:09 Sebastian Moeller  wrote:

> Hi Luca
>
> > On Dec 3, 2021, at 15:58, Luca Muscariello  wrote:
> >
> >
> >
> > On Fri, Dec 3, 2021 at 3:35 PM Sebastian Moeller 
> wrote:
> > Hi Dave,
> >
> >
> > > On Dec 3, 2021, at 15:18, Dave Taht  wrote:
> > >
> > > On Fri, Dec 3, 2021 at 4:00 AM Luca Muscariello 
> wrote:
> > >>
> > >> Test using a tp-link AP EAP 245
> > >>
> > >>
> https://www.waveform.com/tools/bufferbloat?test-id=bbcc5ef5-e677-4f27-aa04-1849db81d0f5
> > >
> > > Nice.
> > >
> > > A kvetch is that I really wish they also tested up and down at the
> same time.
> > >
> > > Another kvetch is I think the test needs to run longer at these speeds.
> > >
> > > Another another kvetch is they factor in baseline latency to determine
> > > if the link is suitable for gaming or not.
> >
> > Which is the sane thing to do... IMHO. For any "twitch"-type
> reation-time gated gaming all players need to be in an acceptable range of
> "RTTs" to and from the server so that all perceive the world similarly and
> nobody has an unfair advantage/disadvantage, so absolute RTT does seem to
> matter. I would agree that jitter is nastier in that is will cause
> "randomish" variations of the RTT, but then the known solution against
> jitter is additional buffering (large enough to simply even out the unequal
> jittered packet arrival times) which in turn just increases the "RTT", no?
> (I guess no game really does this enough, so jitter stays the constant
> problem for internet game-play).
> > >
> >
> > Mobile multiplayer competitive games, like PvP, 5v5 such as CoDM,
> Critical Ops, LoL Wild Rift may have very different ways to face/conceal
> network conditions based on the experience the studio wants to offer to the
> gamer.
>
> Care to elaborate, please? As far as I can tell what I describe
> above is pretty universal iff fair "coordination" between different
> player's actions is required. How do you conceal issues about causal
> ordering of events that ideally are identical from all perspectives?


Not all games use deterministic lockstep because it requires to compensate
latency from the different players. Of course if latency is zero it is
easier. But that assumption or that objective is one of the fallacies of
distributed computing.

Predictive lockstep or even rollback netcode are examples where you fix
update  accuracy reactively.

Beyond specific netcode techniques, some mobile games can give you the
illusion that you're playing PvP while at some point reality is that you
can continue to play almost offline against bots.

There are games where physics inaccuracies can be concealed. Physics is
very important for first person shooter games while an ARPG may conceal
inaccuracies in the middle of the mess that is rendered on screen, think of
League of Legends Wild Rift.


>
> Regards
> Sebastian
>
>
> >
> >
> >
> > > We had them participating on this list at some point
> > >
> > >>
> > >>
> > >> On Thu, Dec 2, 2021 at 7:48 PM Dave Taht  wrote:
> > >>>
> > >>> tp-link, is, so far as I know, the last major home router vendor NOT
> > >>> shipping a SQM system. Perhaps this could be modded up with someones
> > >>> with accounts?
> > >>>
> > >>> https://community.tp-link.com/us/home/forum/topic/511156
> > >>>
> > >>>
> > >>> --
> > >>> I tried to build a better future, a few times:
> > >>> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> > >>>
> > >>> Dave Täht CEO, TekLibre, LLC
> > >>> ___
> > >>> Bloat mailing list
> > >>> Bloat@lists.bufferbloat.net
> > >>> https://lists.bufferbloat.net/listinfo/bloat
> > >
> > >
> > >
> > > --
> > > I tried to build a better future, a few times:
> > > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> > >
> > > Dave Täht CEO, TekLibre, LLC
> > > ___
> > > Cerowrt-devel mailing list
> > > cerowrt-de...@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cerowrt-devel] tp-link request for SQM

[Luca Muscariello via Bloat]

On Fri, Dec 3, 2021 at 3:35 PM Sebastian Moeller  wrote:

> Hi Dave,
>
>
> > On Dec 3, 2021, at 15:18, Dave Taht  wrote:
> >
> > On Fri, Dec 3, 2021 at 4:00 AM Luca Muscariello 
> wrote:
> >>
> >> Test using a tp-link AP EAP 245
> >>
> >>
> https://www.waveform.com/tools/bufferbloat?test-id=bbcc5ef5-e677-4f27-aa04-1849db81d0f5
> >
> > Nice.
> >
> > A kvetch is that I really wish they also tested up and down at the same
> time.
> >
> > Another kvetch is I think the test needs to run longer at these speeds.
> >
> > Another another kvetch is they factor in baseline latency to determine
> > if the link is suitable for gaming or not.
>
> Which is the sane thing to do... IMHO. For any "twitch"-type
> reation-time gated gaming all players need to be in an acceptable range of
> "RTTs" to and from the server so that all perceive the world similarly and
> nobody has an unfair advantage/disadvantage, so absolute RTT does seem to
> matter. I would agree that jitter is nastier in that is will cause
> "randomish" variations of the RTT, but then the known solution against
> jitter is additional buffering (large enough to simply even out the unequal
> jittered packet arrival times) which in turn just increases the "RTT", no?
> (I guess no game really does this enough, so jitter stays the constant
> problem for internet game-play).
> >


Mobile multiplayer competitive games, like PvP, 5v5 such as CoDM, Critical
Ops, LoL Wild Rift may have very different ways to face/conceal network
conditions based on the experience the studio wants to offer to the gamer.



>
> > We had them participating on this list at some point
> >
> >>
> >>
> >> On Thu, Dec 2, 2021 at 7:48 PM Dave Taht  wrote:
> >>>
> >>> tp-link, is, so far as I know, the last major home router vendor NOT
> >>> shipping a SQM system. Perhaps this could be modded up with someones
> >>> with accounts?
> >>>
> >>> https://community.tp-link.com/us/home/forum/topic/511156
> >>>
> >>>
> >>> --
> >>> I tried to build a better future, a few times:
> >>> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >>>
> >>> Dave Täht CEO, TekLibre, LLC
> >>> ___
> >>> Bloat mailing list
> >>> Bloat@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > --
> > I tried to build a better future, a few times:
> > https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
> >
> > Dave Täht CEO, TekLibre, LLC
> > ___
> > Cerowrt-devel mailing list
> > cerowrt-de...@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] tp-link request for SQM

[Luca Muscariello via Bloat]

Test using a tp-link AP EAP 245

https://www.waveform.com/tools/bufferbloat?test-id=bbcc5ef5-e677-4f27-aa04-1849db81d0f5



On Thu, Dec 2, 2021 at 7:48 PM Dave Taht  wrote:

> tp-link, is, so far as I know, the last major home router vendor NOT
> shipping a SQM system. Perhaps this could be modded up with someones
> with accounts?
>
> https://community.tp-link.com/us/home/forum/topic/511156
>
>
> --
> I tried to build a better future, a few times:
> https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org
>
> Dave Täht CEO, TekLibre, LLC
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Luca Muscariello]

For those who might be interested in Little's law
there is a nice paper by John Little on the occasion
of the 50th anniversary  of the result.

https://www.informs.org/Blogs/Operations-Research-Forum/Little-s-Law-as-Viewed-on-its-50th-Anniversary

https://www.informs.org/content/download/255808/2414681/file/little_paper.pdf

Nice read.
Luca

P.S.
Who has not a copy of L. Kleinrock's books? I do have and am not ready to
lend them!

On Fri, Jul 9, 2021 at 11:01 AM Leonard Kleinrock  wrote:

> David,
>
> I totally appreciate  your attention to when and when not analytical
> modeling works. Let me clarify a few things from your note.
>
> First, Little's law (also known as Little’s lemma or, as I use in my book,
> Little’s result) does not assume Poisson arrivals -  it is good for *any*
> arrival process and any service process and is an equality between time
> averages.  It states that the time average of the number in a system (for a
> sample path *w)* is equal to the average arrival rate to the system
> multiplied by the time-averaged time in the system for that sample path.
> This is often written as   NTimeAvg =λ·TTimeAvg .  Moreover, if the
> system is also ergodic, then the time average equals the ensemble average
> and we often write it as N ̄ = λ T ̄ .  In any case, this requires
> neither Poisson arrivals nor exponential service times.
>
> Queueing theorists often do study the case of Poisson arrivals.  True, it
> makes the analysis easier, yet there is a better reason it is often used,
> and that is because the sum of a large number of independent stationary
> renewal processes approaches a Poisson process.  So nature often gives us
> Poisson arrivals.
>
> Best,
> Len
>
>
>
> On Jul 8, 2021, at 12:38 PM, David P. Reed  wrote:
>
> I will tell you flat out that the arrival time distribution assumption
> made by Little's Lemma that allows "estimation of queue depth" is totally
> unreasonable on ANY Internet in practice.
>
>
> The assumption is a Poisson Arrival Process. In reality, traffic arrivals
> in real internet applications are extremely far from Poisson, and, of
> course, using TCP windowing, become highly intercorrelated with crossing
> traffic that shares the same queue.
>
>
> So, as I've tried to tell many, many net-heads (people who ignore
> applications layer behavior, like the people that think latency doesn't
> matter to end users, only throughput), end-to-end packet arrival times on a
> practical network are incredibly far from Poisson - and they are more like
> fractal probability distributions, very irregular at all scales of time.
>
>
> So, the idea that iperf can estimate queue depth by Little's Lemma by just
> measuring saturation of capacity of a path is bogus.The less Poisson, the
> worse the estimate gets, by a huge factor.
>
>
>
>
> Where does the Poisson assumption come from?  Well, like many theorems, it
> is the simplest tractable closed form solution - it creates a simplified
> view, by being a "single-parameter" distribution (the parameter is called
> lambda for a Poisson distribution).  And the analysis of a simple queue
> with poisson arrival distribution and a static, fixed service time is the
> first interesting Queueing Theory example in most textbooks. It is
> suggestive of an interesting phenomenon, but it does NOT characterize any
> real system.
>
>
> It's the queueing theory equivalent of "First, we assume a spherical
> cow...". in doing an example in a freshman physics class.
>
>
> Unfortunately, most networking engineers understand neither queuing theory
> nor application networking usage in interactive applications. Which makes
> them arrogant. They assume all distributions are poisson!
>
>
>
>
> On Tuesday, July 6, 2021 9:46am, "Ben Greear" 
> said:
>
> > Hello,
> >
> > I am interested to hear wish lists for network testing features. We make
> test
> > equipment, supporting lots
> > of wifi stations and a distributed architecture, with built-in udp, tcp,
> ipv6,
> > http, ... protocols,
> > and open to creating/improving some of our automated tests.
> >
> > I know Dave has some test scripts already, so I'm not necessarily
> looking to
> > reimplement that,
> > but more fishing for other/new ideas.
> >
> > Thanks,
> > Ben
> >
> > On 7/2/21 4:28 PM, Bob McMahon wrote:
> > > I think we need the language of math here. It seems like the network
> > power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something
> > useful.
> > > Effective end/end queue depths per Little's law also seems useful.
> Both are
> > available in iperf 2 from a test perspective. Repurposing test
> techniques to
> > actual
> > > traffic could be useful. Hence the question around what exact telemetry
> > is useful to apps making socket write() and read() calls.
> > >
> > > Bob
> > >
> > > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht  > > wrote:
> > >
> > > In terms of trying to find "Quality" I have tried to encourage folk to
> > > both read "zen and 

Re: [Bloat] testing latency under load on webrtc with galene?

[Luca Muscariello]

Dave,

we have done extensive WebRTC (and several other online meeting
apps) testing lately and this paper
https://ieeexplore.ieee.org/document/9153228 reports a
methodology for WebRTC based on Chromium and Selenium Grid and
as test orchestrator Jitsi Torture.

I would avoid feeding clients with BBB as video as it is not
representative of a meeting as encoders are optimized for
different kinds of video. There are several video samples
out there.

We have scaled up clients to hundreds with this methodology.
The paper is short so many details have been omitted but there
are not many other options to do this kind of test at scale.

Luca







describes the methodology



On Tue, Mar 2, 2021 at 2:15 AM Dave Taht  wrote:

> Given that we have a worldwide network of flent servers...
>
> Given how easy galene is to hack on... and a 10 minute install...
>
> given some webrtc scripting... a few more stats... some javascript...
> skull sweat... funding...
>
> It seems plausible to be able to construct a suite of tests that could
> indeed track jitter
> delay and loss across the internet over webrtc. Perpetually uploading
> bigbuckbunny or some
> other suitable movie might be an option, but I have a fondness for
> extracting a sub 30 second segment from  "Max Headroom", which if
> those here have not seen it, predicted much of the fine mess we're all
> in now.
>
> I guess my question is mostly, is a "headless" test feasible? In the
> context of samknow's lack of webrtc test... lowpowered hw
>
> --
> "For a successful technology, reality must take precedence over public
> relations, for Mother Nature cannot be fooled" - Richard Feynman
>
> d...@taht.net  CTO, TekLibre, LLC Tel: 1-831-435-0729
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] BBR implementations, knobs to turn?

[Luca Muscariello]

Hi Erick,

one question about the PGW: is it a policer or a shaper that you have
installed?
Also, have you tried to run a ping session before and in parallel to the
curl sessions?

Luca



On Thu, Nov 19, 2020 at 2:15 PM  wrote:

> Update:
> The 5G router was connected to a new base station.  Now the limiting
> factor of throughput is the policer on the PGW in mobile core, not the
> radio link itself.  The SIM card used is limited to 30Mbit/s.  This
> scenario favours the new server.  I have attached graphs comparing radio
> link limited vs PGW policer results, and a zoomed in graph of the policer
>
>
> We have Huawei RAN and Ericsson RAN, rate limited and not rate limited
> subscriptions, 4G and 5G access, and we are migrating to a new core with
> new PGW (policer).  Starting to be a bit of a matrix to set up tests for.
>
>
> -Erik
>
>
> 
> Fra: Jesper Dangaard Brouer 
> Sendt: 17. november 2020 16:07
> Til: Taraldsen Erik; Priyaranjan Jha
> Kopi: bro...@redhat.com; ncardw...@google.com; bloat@lists.bufferbloat.net
> Emne: Re: [Bloat] BBR implementations, knobs to turn?
>
> On Tue, 17 Nov 2020 10:05:24 +  wrote:
>
> > Thank you for the response Neal
>
> Yes. And it is impressive how many highly qualified people are on the
> bufferbloat list.
>
> > old_hw # uname -r
> > 5.3.0-64-generic
> > (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit
> > GPON access.  Used as proof of concept from the lab at work)
> >
> >
> > new_hw # uname -r
> > 4.18.0-193.19.1.el8_2.x86_64
> > (Centos 8.2 on xenon rack server, discrete 10Gbit network card,
> > 40Gbit server farm link (low utilization on link), intended as fully
> > supported and run service.  Not possible to have newer kernel and
> > still get service agreement in my organization)
>
> Let me help out here.  The CentOS/RHEL8 kernels have a huge amount of
> backports.  I've attached a patch/diff of net/ipv4/tcp_bbr.c changes
> missing in RHEL8.
>
> It looks like these patches are missing in CentOS/RHEL8:
>  [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3
>  [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7
>
> Could missing patch [1] result in the issue Erik is seeing?
> (It explicitly mentions improvements for WiFi...)
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Luca Muscariello]

On Thu, Sep 3, 2020 at 4:32 PM Toke Høiland-Jørgensen  wrote:
>
> Luca Muscariello  writes:
>
> > On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
> >  wrote:
> >>
> >> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
> >>
> >> > Yup, the number of cores is only going to go up, so for CAKE to stay
> >> > relevant it'll need to be able to take advantage of this eventually :)
> >>
> >> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform,
> >> it has a quad core machine with 2 x 2.5GbE NICs.
> >>
> >> When using something like this for routing with HTB+CAKE for bidirectional
> >> shaping below line rate, what would be the main things that would need to
> >> be improved?
> >
> > IMO, hardware offloading for shaping, beyond this specific platform.
> > I ignore if there is any roadmap with that objective.
>
> Yeah, offloading of some sort is another option, but I consider that
> outside of the "CAKE stays relevant" territory, since that will most
> likely involve an entirely programmable packet scheduler. There was some
> discussion of adding such a qdisc to Linux at LPC[0]. The Eiffel[1]
> algorithm seems promising.
>
> -Toke
>
> [0] https://linuxplumbersconf.org/event/7/contributions/679/
> [1] https://www.usenix.org/conference/nsdi19/presentation/saeed

These are all interesting efforts for scheduling but orthogonal to shaping
and not going to help make shaping more scalable.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] CAKE in openwrt high CPU

[Luca Muscariello]

On Thu, Sep 3, 2020 at 3:19 PM Mikael Abrahamsson via Bloat
 wrote:
>
> On Tue, 1 Sep 2020, Toke Høiland-Jørgensen wrote:
>
> > Yup, the number of cores is only going to go up, so for CAKE to stay
> > relevant it'll need to be able to take advantage of this eventually :)
>
> https://www.hardkernel.com/shop/odroid-h2plus/ is an interesting platform,
> it has a quad core machine with 2 x 2.5GbE NICs.
>
> When using something like this for routing with HTB+CAKE for bidirectional
> shaping below line rate, what would be the main things that would need to
> be improved?

IMO, hardware offloading for shaping, beyond this specific platform.
I ignore if there is any roadmap with that objective.

>
> --
> Mikael Abrahamssonemail: 
> swm...@swm.pp.se___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] the future belongs to pacing

[Luca Muscariello]

It is not surprising that BBR comes from Van who's also designed and
implemented pathchar.

I liked reading the paper when it was published and it has the merit to be
simple to read
for a large audience.

I agree very much on the title as bang-bang congestion control (not only
AIMD) could be
deprecated entirely by measurement based approaches like BBR.

In bang-bang cc the sending rate is obtained by a root-finding algorithm
(gradient based) that
is fed by measurements of congestion (queue, loss, latency), whereas in BBR
the sending rate is
an (almost?) explicit function of the measured quantities.

In theory both approaches work, but for the former we have seen a
proliferation of root-finding algorithms
for wireless, large BDP networks, small BDP network, satellite, cellular,
shared-media, non-shared media and more.
Selection of the right one is a question of tuning, which is extremely
complex and static.

If BBR can fix that by having a unique model for all these cases that would
make deprecation, as intended in the paper,
likely to happen.


On Fri, Dec 13, 2019 at 10:25 PM Dave Taht  wrote:

> and everything we know about the tcp macroscopic model, is obsolete,
> according to a  provocative paper by matt mathis and Jamshid Mahdavi
> in sigcomm.
>
> https://ccronline.sigcomm.org/wp-content/uploads/2019/10/acmdl19-323.pdf
>
>
>
>
> On Fri, Dec 13, 2019 at 1:05 PM Carlo Augusto Grazia
>  wrote:
> >
> > Hi Dave,
> > thank you for your email!
> > Toke told me about AQL a couple of weeks ago, I definitely want to test
> it ASAP.
> > BBR struggles a lot on Wi-Fi interfaces (ones with aggregation) with
> kernel 4.14 & 4.19.
> > Anyway, it seems that with BBRv2 on new kernels this problem does not
> exist anymore.
> >
> > Best regards
> > Carlo
> >
> > Il giorno ven 13 dic 2019 alle 20:54 Dave Taht  ha
> scritto:
> >>
> >> https://sci-hub.tw/10.1109/WiMOB.2019.8923418
> >>
> >> It predates the aql work, but the bbr result is puzzling.
> >>
> >>
> >> --
> >> Make Music, Not War
> >>
> >> Dave Täht
> >> CTO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-831-435-0729
> >
> > --
> > 
> > Carlo Augusto Grazia, Ph. D.
> > Assistant Professor
> > 
> > Dept. of Engineering "Enzo Ferrari"
> > University of Modena and Reggio Emilia
> > Via Pietro Vivarelli, 10/1 - 41125 - Modena - Italy
> > Building 26, floor 2, room 28
> > Tel.: +39-059-2056323
> > email: carloaugusto.gra...@unimore.it
> > Link to my personal home page here
> > 
>
>
>
> --
> Make Music, Not War
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
> ___
> Make-wifi-fast mailing list
> make-wifi-f...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [tsvwg] my backlogged comments on the ECT(1) interim call

[Luca Muscariello]

 I had timestamps at the beginning of each main point of discussion,
> with
> > > the intent that after the video is published it would be easier to go
> back
> > > and check precisely what was said. It looks like someone has been
> making
> > > cleanup edits that removed the first half of those so far, but my local
> > > text file still has most of those and I can go back and re-insert them
> if
> > > it seems useful.
> > >
> > >
> > >
> > > @Luca: during your comments in particular I think there might have
> been a
> > > disruption--I had a ?first comment missed, please check video?
> placeholder
> > > and I may have misunderstood the part about video elasticity, but my
> > > interpretation at the time was that Stuart was claiming that video was
> > > elastic in that it would adjust downward to avoid overflowing a loaded
> > > link, and I thought you were claiming that it was not elastic in that
> it
> > > would not exceed a maximum rate, which I summarized as perhaps a
> semantic
> > > disagreement, but if you?d like to help clean that up, it might be
> useful.
> > >
> > >
> > >
> > > From this message, it sounds like the key point you were making was
> that
> > > it also will not go below a certain rate, and perhaps that quality can
> stay
> > > relatively good in spite of high network loss?
> > >
> > >
> > >
> > > Best regards,
> > >
> > > Jake
> > >
> > >
> > >
> > > *From: *Luca Muscariello 
> > > *Date: *Tuesday, April 28, 2020 at 1:54 AM
> > > *To: *Dave Taht 
> > > *Cc: *tsvwg IETF list , bloat <
> bloat@lists.bufferbloat.net
> > > >
> > > *Subject: *Re: [Bloat] my backlogged comments on the ECT(1) interim
> call
> > >
> > >
> > >
> > > Hi Dave and list members,
> > >
> > >
> > >
> > > It was difficult to follow the discussion at the meeting yesterday.
> > >
> > > Who  said what in the first place.
> > >
> > >
> > >
> > > There have been a lot of non-technical comments such as: this solution
> > >
> > > is better than another in my opinion. "better" has often been used
> > >
> > > as when evaluating the taste of an ice cream: White chocolate vs black
> > > chocolate.
> > >
> > > This has taken a significant amount of time at the meeting. I haven't
> > > learned
> > >
> > > much from that kind of discussion and I do not think that helped to
> make
> > >
> > > much progress.
> > >
> > >
> > >
> > > If people can re-make their points in the list it would help the
> debate.
> > >
> > >
> > >
> > > Another point that a few raised is that we have to make a decision as
> fast
> > > as possible.
> > >
> > > I dismissed entirely that argument. Trading off latency with
> resilience of
> > > the Internet
> > >
> > > is entirely against the design principle of the Internet architecture
> > > itself.
> > >
> > > Risk analysis is something that we should keep in mind even when
> > > deploying any experiment
> > >
> > > and should be a substantial part of it.
> > >
> > >
> > >
> > > Someone claimed that on-line meeting traffic is elastic. This is not
> true,
> > > I tried to
> > >
> > > clarify this. These applications (WebEx/Zoom) are low rate, a typical
> > > maximum upstream
> > >
> > > rate is 2Mbps and is not elastic. These applications have often a
> > > stand-alone app
> > >
> > > that is not using the browser WebRTC stack (the standalone app
> typically
> > > works better).
> > >
> > >
> > >
> > > A client sends upstream one or two video qualities unless the video
> camera
> > > is switched off.
> > >
> > > In presence of losses, FEC is used but it is still non elastic.
> > >
> > > Someone claimed (at yesterday's meeting) that fairness is not an issue
> > > (who cares, I heard!)
> > >
> > > Well, fairness can constitute a differentiation advantage between two
> > > companies that are
> > >
> > > commercializing on-line meetings products. Unless at the IETF we accept
> > >
> > > "law-of-the-jungle" behaviou

Re: [Bloat] [tsvwg] my backlogged comments on the ECT(1) interim call

[Luca Muscariello]

The link is that the L queue starves the other queue, and indeed the
envisaged queue protection mechanism
is supposed to react to that behavior by black-listing the misbehaving
sender.
This would be the third coupled component in L4S (the senders, the AQM and
the policer). Which is currently non-mandatory.

The level of starvation is a parameter in dualQ as the service ratio
between the two queues has to be set
but the AQM owner. How to set this ratio is yet another knob that is
unclear how to optimally set
under a general traffic mix,  that includes unresponsive traffic.

I have already raised this issue in the past.



On Tue, Apr 28, 2020 at 9:26 PM Gorry Fairhurst 
wrote:

> This seems all interesting, but isn't this true of any network technology.
> If I use a UDP app with my own style of CC I can just take all the capacity
> if I want.
>
> A solution could be to apply a circuit-breaker/policer in the network to
> perform admission control, but I don't see the link to L4S. Have I missed
> something ?
>
> Gorry
> On 28/04/2020 20:04, Luca Muscariello wrote:
>
> Hi Jake,
>
> Thanks for the notes. Very useful.
> The other issue with the meeting was that the virtual mic queue control
> channel was the WebEx Meeting chat that does not exist in WebEx Teams. So,
> I had to switch to Meetings and lost some pieces of the discussion.
>
> Yes there might be a terminology difference. Elastic traffic is usually
> used in the sense of bandwidth sharing not just to define variable bit
> rates.
>
> The point is that there are incentives to cheat in L4S.
>
> There is a priority queue that my application can enter by providing as
> input ECT(1).
> Applications such as on-line meetings will have a relatively low and
> highly paced rate.
>
> This traffic is conformant to dualQ L queue but is unresponsive to
> congestion notifications.
>
> This is especially true for FEC streams which could be used to ameliorate
> the media quality in presence of losses(e.g. Wi-Fi)
> or increased jitter.
>
>
> That was one more point on why using ECT(1) as input assumes trust or a
> black list after being caught.
>
> In both cases the ECT(1) as input is DoSable.
>
>
>
> On Tue, Apr 28, 2020 at 7:12 PM Holland, Jake  wrote:
>
>> Hi Luca,
>>
>>
>>
>> To your point about the discussion being difficult to follow: I tried to
>> capture the intent of everyone who commented while taking notes:
>>
>> https://etherpad.ietf.org:9009/p/notes-ietf-interim-2020-tsvwg-03
>>
>>
>>
>> I think this was intended to take the place of a need for everyone to
>> re-send the same points to the list, but of course some of the most crucial
>> points could probably use fleshing out with on-list follow up.
>>
>>
>>
>> It got a bit rough in places because I was disconnected a few times and
>> had to cut over to a local text file, and I may have failed to correctly
>> understand or summarize some of the comments, so there’s chances I might
>> have missed something, but I did my best to capture them all.
>>
>>
>>
>> I encourage people to review comments and check whether they came out
>> more or less correct, and to offer formatting and cleanup suggestions if
>> there’s a good way to make it easier to follow.
>>
>>
>>
>> I had timestamps at the beginning of each main point of discussion, with
>> the intent that after the video is published it would be easier to go back
>> and check precisely what was said. It looks like someone has been making
>> cleanup edits that removed the first half of those so far, but my local
>> text file still has most of those and I can go back and re-insert them if
>> it seems useful.
>>
>>
>>
>> @Luca: during your comments in particular I think there might have been a
>> disruption--I had a “first comment missed, please check video” placeholder
>> and I may have misunderstood the part about video elasticity, but my
>> interpretation at the time was that Stuart was claiming that video was
>> elastic in that it would adjust downward to avoid overflowing a loaded
>> link, and I thought you were claiming that it was not elastic in that it
>> would not exceed a maximum rate, which I summarized as perhaps a semantic
>> disagreement, but if you’d like to help clean that up, it might be useful.
>>
>>
>>
>> From this message, it sounds like the key point you were making was that
>> it also will not go below a certain rate, and perhaps that quality can stay
>> relatively good in spite of high network loss?
>>
>>
>>
>> Best regards,
>>
>> Jake
>>
>>

Re: [Bloat] my backlogged comments on the ECT(1) interim call

[Luca Muscariello]

Hi Jake,

Thanks for the notes. Very useful.
The other issue with the meeting was that the virtual mic queue control
channel was the WebEx Meeting chat that does not exist in WebEx Teams. So,
I had to switch to Meetings and lost some pieces of the discussion.

Yes there might be a terminology difference. Elastic traffic is usually
used in the sense of bandwidth sharing not just to define variable bit
rates.

The point is that there are incentives to cheat in L4S.

There is a priority queue that my application can enter by providing as
input ECT(1).
Applications such as on-line meetings will have a relatively low and highly
paced rate.

This traffic is conformant to dualQ L queue but is unresponsive to
congestion notifications.

This is especially true for FEC streams which could be used to ameliorate
the media quality in presence of losses(e.g. Wi-Fi)
or increased jitter.


That was one more point on why using ECT(1) as input assumes trust or a
black list after being caught.

In both cases the ECT(1) as input is DoSable.



On Tue, Apr 28, 2020 at 7:12 PM Holland, Jake  wrote:

> Hi Luca,
>
>
>
> To your point about the discussion being difficult to follow: I tried to
> capture the intent of everyone who commented while taking notes:
>
> https://etherpad.ietf.org:9009/p/notes-ietf-interim-2020-tsvwg-03
>
>
>
> I think this was intended to take the place of a need for everyone to
> re-send the same points to the list, but of course some of the most crucial
> points could probably use fleshing out with on-list follow up.
>
>
>
> It got a bit rough in places because I was disconnected a few times and
> had to cut over to a local text file, and I may have failed to correctly
> understand or summarize some of the comments, so there’s chances I might
> have missed something, but I did my best to capture them all.
>
>
>
> I encourage people to review comments and check whether they came out more
> or less correct, and to offer formatting and cleanup suggestions if there’s
> a good way to make it easier to follow.
>
>
>
> I had timestamps at the beginning of each main point of discussion, with
> the intent that after the video is published it would be easier to go back
> and check precisely what was said. It looks like someone has been making
> cleanup edits that removed the first half of those so far, but my local
> text file still has most of those and I can go back and re-insert them if
> it seems useful.
>
>
>
> @Luca: during your comments in particular I think there might have been a
> disruption--I had a “first comment missed, please check video” placeholder
> and I may have misunderstood the part about video elasticity, but my
> interpretation at the time was that Stuart was claiming that video was
> elastic in that it would adjust downward to avoid overflowing a loaded
> link, and I thought you were claiming that it was not elastic in that it
> would not exceed a maximum rate, which I summarized as perhaps a semantic
> disagreement, but if you’d like to help clean that up, it might be useful.
>
>
>
> From this message, it sounds like the key point you were making was that
> it also will not go below a certain rate, and perhaps that quality can stay
> relatively good in spite of high network loss?
>
>
>
> Best regards,
>
> Jake
>
>
>
> *From: *Luca Muscariello 
> *Date: *Tuesday, April 28, 2020 at 1:54 AM
> *To: *Dave Taht 
> *Cc: *tsvwg IETF list , bloat  >
> *Subject: *Re: [Bloat] my backlogged comments on the ECT(1) interim call
>
>
>
> Hi Dave and list members,
>
>
>
> It was difficult to follow the discussion at the meeting yesterday.
>
> Who  said what in the first place.
>
>
>
> There have been a lot of non-technical comments such as: this solution
>
> is better than another in my opinion. "better" has often been used
>
> as when evaluating the taste of an ice cream: White chocolate vs black
> chocolate.
>
> This has taken a significant amount of time at the meeting. I haven't
> learned
>
> much from that kind of discussion and I do not think that helped to make
>
> much progress.
>
>
>
> If people can re-make their points in the list it would help the debate.
>
>
>
> Another point that a few raised is that we have to make a decision as fast
> as possible.
>
> I dismissed entirely that argument. Trading off latency with resilience of
> the Internet
>
> is entirely against the design principle of the Internet architecture
> itself.
>
> Risk analysis is something that we should keep in mind even when
> deploying any experiment
>
> and should be a substantial part of it.
>
>
>
> Someone claimed that on-line meeting traffic is elastic. This is

Re: [Bloat] my backlogged comments on the ECT(1) interim call

[Luca Muscariello]

Hi Dave and list members,

It was difficult to follow the discussion at the meeting yesterday.
Who  said what in the first place.

There have been a lot of non-technical comments such as: this solution
is better than another in my opinion. "better" has often been used
as when evaluating the taste of an ice cream: White chocolate vs black
chocolate.
This has taken a significant amount of time at the meeting. I haven't
learned
much from that kind of discussion and I do not think that helped to make
much progress.

If people can re-make their points in the list it would help the debate.

Another point that a few raised is that we have to make a decision as fast
as possible.
I dismissed entirely that argument. Trading off latency with resilience of
the Internet
is entirely against the design principle of the Internet architecture
itself.
Risk analysis is something that we should keep in mind even when
deploying any experiment
and should be a substantial part of it.

Someone claimed that on-line meeting traffic is elastic. This is not true,
I tried to
clarify this. These applications (WebEx/Zoom) are low rate, a typical
maximum upstream
rate is 2Mbps and is not elastic. These applications have often a
stand-alone app
that is not using the browser WebRTC stack (the standalone app typically
works better).

A client sends upstream one or two video qualities unless the video camera
is switched off.
In presence of losses, FEC is used but it is still non elastic.
Someone claimed (at yesterday's meeting) that fairness is not an issue (who
cares, I heard!)
Well, fairness can constitute a differentiation advantage between two
companies that are
commercializing on-line meetings products. Unless at the IETF we accept
"law-of-the-jungle" behaviours from Internet applications developers, we
should be careful
about making such claims.
Any opportunity to cheat, that brings a business advantage WILL be used.

/Luca

TL;DR
To Dave: you asked several times what  Cisco does on latency reduction in
network equipment. I tend to be very shy when replying on these questions
as this is not vendor neutral. If chairs think this is not appropriate for
the list, please say it and I'll reply privately only.

What I write below can be found in Cisco products data sheets and is not
trade secret. There are very good blog posts explaining details.
Not surprisingly Cisco implements the state of the art on the topic
and it is totally feasible to do-the-right-thing in software and hardware.

Cisco implements AFD (one queue + a flow table) accompanied by a priority
queue for
flows that have a certain profile in rate and size. The concept is well
known and well
studied in the literature. AFD is safe and can well serve a complex traffic
mix when
accompanied by a priority queue. This prio-queue should not be confused
with a strict
priority queue (e.g. EF in diffserv). There are subtleties related to the
DOCSIS
shared medium which would be too long to describe here.

This is available in Cisco CMTS for the DOCSIS segment. Bottleneck traffic
does not negatively impact non-bottlenecked-traffic such as an on-line
meeting like
the WebEx call we had yesterday. It is safe from a network neutrality
point-of-view
and no applications get hurt.

Cisco implements AFD+prio also for some DC switches such as the Nexus 9k.
There
is a blog post written by Tom Edsal online that explains pretty well how
that works.
This includes mechanisms such as p-fabric to approximate SRPT (shortest
remaining processing time)
and minimize flow completion time for many DC workloads. The mix of the two
brings FCT minimization AND latency minimization. This is silicon and
scales at any speed.
For those who are not familiar with these concepts, please search the
research work of Balaji
Prabhakar and Ron Pang at Stanford.

Wi-Fi: Cisco does airtime fairness in Aironet but I think in the Meraki
series too.
The concept is similar to what described above but there are several
queues, one per STA.
Packets are enqueued in the access (category) queue at dequeue time from
the air-time
packet scheduler.

On Mon, Apr 27, 2020 at 9:24 PM Dave Taht  wrote:

> It looks like the majority of what I say below is not related to the
> fate of the "bit". The push to take the bit was
> strong with this one, and me... can't we deploy more of what we
> already got in places where it matters?
>
> ...
>
> so: A) PLEA: From 10 years now, of me working on bufferbloat, working
> on real end-user and wifi traffic and real networks
>
> I would like folk here to stop benchmarking two flows that run for a long
> time
> and in one direction only... and thus exclusively in tcp congestion
> avoidance mode.
>
> Please. just. stop. Real traffic looks nothing like that. The internet
> looks nothing like that.
> The netops folk I know just roll their eyes up at benchmarks like this
> that prove nothing and tell me to go to ripe meetings instead.
> When y'all talk about "not looking foolish for not mandating ecn now",
> 

Re: [Bloat] [Ecn-sane] 2019-12-31 docsis strict priority dual queue patent granted

[Luca Muscariello]

A Bloom filter based classifier for Bottlenecked vs non bottlenecked flows
was done in here in 2005.

https://team.inria.fr/rap/files/2013/12/KMOR05a.pdf

And associated patent granted since 2011

https://patents.google.com/patent/US7933204B2/en?oq=US7933204B2+United+States++

Luca

On Thu, Jan 23, 2020 at 3:29 AM Dave Taht  wrote:

> I don't normally read patents, but, this one is pretty cable modem
> specific and reads better than the related internet drafts.The
> interaction with maps scheduling is well described.
>
> https://patents.google.com/patent/US10523577B2/en
>
> Of particular irony is the misspelt  "fear/flow cueing"  and I had
> suggested a bloom fillter in all innocence when some of these ideas
> were first discussed.
>
> There are a few other patents cited of interest.
>
> --
> Make Music, Not War
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-435-0729
> ___
> Ecn-sane mailing list
> ecn-s...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat on 4G Connexion

[Luca Muscariello]

On Thu, Oct 24, 2019 at 11:34 AM Toke Høiland-Jørgensen 
wrote:

> Luca Muscariello  writes:
>
> > On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen 
> > wrote:
> >
> >> Rich Brown  writes:
> >>
> >> >> On Oct 23, 2019, at 5:54 AM, >> erik.tarald...@telenor.com>> wrote:
> >> >>
> >> >> If you could influence the 4G vendors to de-bloat their equipment,
> >> >> would you recommend BQL, L4S or codel/cake?
> >> >
> >> > I've been enjoying this discussion and wonder whether the work going
> >> > on in the make-wifi-fast
> >> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is
> relevant.
> >> >
> >> > I only have a 30,000 foot understanding of this work, but it seems the
> >> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
> >> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
> >> > of the actual time it takes to transmit a packet might give additional
> >> > information about the current link speed, with the potential for
> >> > adjusting the codel target, etc.
> >>
> >> Indeed, I suspect something like AQL would work for LTE as well. At the
> >> right level; think this might need to be in the firmware (which in turn
> >> could push back on the host).
> >>
> >> > Separately, I also wonder whether the Air Time Fairness algorithm
> >> > might provide a benefit if the cellphone tower station manufacturers
> >> > chose to get into the game.
> >>
> >> LTE base stations already does TDMA scheduling (which they can do easily
> >> because they are centralised and own the license band); airtime fairness
> >> is about getting the same benefits into WiFi that LTE has been enjoying
> >> from the get-go :)
> >>
> >
> > There is one main difference between ATF and the kind of TDMA
> > realized by an LTE scheduler (but also HSDPA/HSUPA).
> > Toke correct me if I'm wrong.
> >
> > The current ATF scheduler for WiFi does airtime-DRR based on the
> > current PHY rates, is that right? Side question, how do you measure
> > current?
>
> s/current/last/. The ATF scheduler does everything after-the-fact, by
> accounting the actual TX time of a transmission after it has completed.
> So no fancy scheduling or prediction tricks are needed; with the
> tradeoff being coarser granularity of the fairness achieved (i.e., there
> can be unfairness on short timescales).
>
> In the airtime queue limit work that's ongoing, we do ahead-of-time
> airtime estimation to limit queueing in firmware. But this still just
> uses the last TX rate recorded for the given station to calculate the
> estimate.
>
> > In LTE TDMA makes use of what is called multi-user diversity gain
> > by scheduling users when they are at their relative best radio condition.
> > Typically the user with the best current radio condition NORMALIZED
> > over the average radio conditions. The average can be based on a
> > moving average or a sliding window. This is the case of the widely used
> > David Tse's proportional fair scheduler.
> >
> > This means that TDMA is still in place to share air-time fairly but the
> > scheduler will tend to avoid bad radio conditions.
> >
> > From a theoretical point of view if you do that the total capacity
> > of the AP can increase with the number of stations (I think
> logarithmically)
> > as the scheduler surfs across radio quality peaks and not the average
> radio
> > quality. Very smart.
> >
> > In LTE this is doable as the scheduling time slot is 1ms and the
> > feedback channel is as fast. Not all TDMAs are equal.
>
> Yeah, the LTE MAC is pretty cool. Just a shame that the equipment is so
> expensive :(
>

It looks like there is a positive correlation between the size
of the specifications and the cost to build the associated product :)




>
> > Maybe the current scheduler in WiFi can be improved to do that. Maybe.
>
> I think 802.11ax is going in that direction. Nothing nearly as advanced,
> but at least there's the possibility of a dedicated back channel...
>

That's right. ax does much better.



>
> -Toke
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat on 4G Connexion

[Luca Muscariello]

On Wed, Oct 23, 2019 at 2:27 PM Toke Høiland-Jørgensen 
wrote:

> Rich Brown  writes:
>
> >> On Oct 23, 2019, at 5:54 AM, erik.tarald...@telenor.com>> wrote:
> >>
> >> If you could influence the 4G vendors to de-bloat their equipment,
> >> would you recommend BQL, L4S or codel/cake?
> >
> > I've been enjoying this discussion and wonder whether the work going
> > on in the make-wifi-fast
> > (https://lists.bufferbloat.net/pipermail/make-wifi-fast/) is relevant.
> >
> > I only have a 30,000 foot understanding of this work, but it seems the
> > use of AQL (Airtime Queue Limit) maps better onto the vagaries of
> > 4G/5G radio transmissions than BQL. Specifically, having a measurement
> > of the actual time it takes to transmit a packet might give additional
> > information about the current link speed, with the potential for
> > adjusting the codel target, etc.
>
> Indeed, I suspect something like AQL would work for LTE as well. At the
> right level; think this might need to be in the firmware (which in turn
> could push back on the host).
>
> > Separately, I also wonder whether the Air Time Fairness algorithm
> > might provide a benefit if the cellphone tower station manufacturers
> > chose to get into the game.
>
> LTE base stations already does TDMA scheduling (which they can do easily
> because they are centralised and own the license band); airtime fairness
> is about getting the same benefits into WiFi that LTE has been enjoying
> from the get-go :)
>

There is one main difference between ATF and the kind of TDMA
realized by an LTE scheduler (but also HSDPA/HSUPA).
Toke correct me if I'm wrong.

The current ATF scheduler for WiFi does airtime-DRR based on the current
PHY rates,
is that right? Side question, how do you measure current?

In LTE TDMA makes use of what is called multi-user diversity gain
by scheduling users when they are at their relative best radio condition.
Typically the user with the best current radio condition NORMALIZED
over the average radio conditions. The average can be based on a
moving average or a sliding window. This is the case of the widely used
David Tse's proportional fair scheduler.

This means that TDMA is still in place to share air-time fairly but the
scheduler will tend to avoid bad radio conditions.

>From a theoretical point of view if you do that the total capacity
of the AP can increase with the number of stations (I think logarithmically)
as the scheduler surfs across radio quality peaks and not the average radio
quality. Very smart.

In LTE this is doable as the scheduling time slot is 1ms and the feedback
channel
is as fast. Not all TDMAs are equal.
Maybe the current scheduler in WiFi can be improved to do that. Maybe.

Luca




>
> -Toke
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] buffer sizing meeting notes

[Luca Muscariello]

On Sat, Aug 24, 2019 at 9:59 AM Sebastian Moeller  wrote:

> Another nugget from the notes (
> http://yuba.stanford.edu/~bspang/buffer-sizing-meeting/notes/):
>
> This looks like an argument for fq_codel/cake's use of time instead of
> queue length, OR an argument for fq, because in a non-overwhelmed fq_system
> the local bucket's queue length should be somewhat stronger correlated with
> the sojurn times, than the sojurn time of a packet though a shared queue,
> no?
>

I believe that using fq there are good implications in terms of buffer
sizing rules.
The focus is only on backlogged flows. The sparse flow queue can be sized
purely based on queue load.
Which is trivial queuing theory formulas with non-TCP traffic.

For backlogged flows, sizing is just like sizing a buffer for a single TCP
flow. So the computation
can be done based on estimations of the number of backlogged flows and by
setting the cut-off BDP based on the
largest (maximum) min RTT one want to serve optimally.  So AQM is really
needed to get optimality.
The number of backlogged flows could be estimated as a max value or an avg
value, and this of course
changes depending on the network segment (DC, residential, campus, access,
backhaul etc.).

Nick McKeown has done quite a lot of research on the topic (reported in the
slide deck), so I find hilarious the following in his slide deck
"Personal confession: I have no idea what the general answer is" about
sizing buffers!
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] three new internet drafts regarding SCE

[Luca Muscariello]

Remote attendance is free of charge but you have to register to be able to
access.

https://www.ietf.org/registration/ietf105/remotereg.py



On Wed, Jul 17, 2019 at 3:13 PM Dave Taht  wrote:

> IETF 105 runs from July 20-27th in Montreal.
>
> https://datatracker.ietf.org/meeting/105/agenda/
>
> tsvwg meets thursday morning 10-12, and friday 12:20-
>
> Remote attendance via videconferencing tools is straightforward. There
> are of course dozens of other wg meetings of possible interest, in my
> case, I'm still tracking babel's progress through the ietf, in
> particular, and I always try to
> check in on iccrg, also in case anything interesting comes up.
>
> Since not all members of our mailing lists are on the relevant tsvwg
> or tcpmwg  mailing lists, here are some drafts
> from those working on the SCE front (I'm not, but I do read things)
> for aqm and transport enhancements.
>
> https://tools.ietf.org/html/draft-grimes-tcpmwg-tcpsce-00
>
> https://tools.ietf.org/html/draft-morton-tsvwg-sce-00
>
> https://tools.ietf.org/html/draft-heist-tsvwg-sce-one-and-two-flow-tests-00
>
> I would have liked it if the the actual scripts, & flent data files
> were published and referenced in this last draft. (I think the
> pictures were published on some other email thread (?), and I look
> forward to the slides) My own
> (eventual) contribution to this work might be on the wifi front, but
> neither l4s or sce are baked enough yet to bother trying,
> IMHO. My analysis of the battlemesh fq_codel + ecn over wifi data I
> hope to finish this week, but I'll find an other outlet for
> publication. (smallest subset of observations is that we can reduce
> the codel target to 6ms on wifi networks that have powersave disabled,
> and that serious 802.11e queue use still massively sucks. details to
> come later)
>
> There are many, many, many other drafts in progress in tsvwg, of note
> might be:
>
> https://tools.ietf.org/id/draft-white-tsvwg-lld-00.txt
>
> https://tools.ietf.org/id/draft-white-tsvwg-nqb-02.txt
>
> In addition to the perpetually revised l4s related ones.
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> ___
> Ecn-sane mailing list
> ecn-s...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/ecn-sane
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] datapoint from one vendor regarding bloat

[Luca Muscariello]

Defs

https://tools.ietf.org/html/rfc2697


On Thu 11 Apr 2019 at 19:54, Jonathan Morton  wrote:

> > On 11 Apr, 2019, at 1:38 pm, Mikael Abrahamsson 
> wrote:
> >
> > The mbs defines the MBS for the PIR bucket and the cbs defines the CBS
> for the CIR bucket
>
> What do these lumps of jargon refer to?
>
>  - Jonathan Morton
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

[Luca Muscariello]

For enterprise networks interconnection to the cloud (public or private)
can be done with e2e DSCP marking.
This is just part of products available at AWS/Azure w/ or w/o their own
interconnection network (direct connect, express route)
and used regularly to ameliorate QoS of RTC applications such as WebEx,
Skype for business, WebRTC.
As I said, peering is key and cloud providers are doing that very well.
With the current widespread of SaaS and IaaS your app is likely running in
the cloud.

Even in case a branch office is connected using SD-WAN, DSCP is tunnelled
by it, and transparently for the application.

If working from home, the quality of peering of an SP to cloud providers is
key.
I'd say that the Cloud and the decline of transit networks have facilitated
DSCP e2e.




On Sat, Mar 23, 2019 at 7:40 PM Mikael Abrahamsson  wrote:

> On Sat, 23 Mar 2019, Luca Muscariello wrote:
>
> > It the app runs in the cloud and the cloud has direct peering links to
> > your branch office or SP most likely DSCP works.
>
> Do you have numbers to back this statement up? In my experience direct
> peering links has nothing to do with this, instead remaking is done
> equally at the customer edge and peering/transit edge respectively.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

[Luca Muscariello]

WebEx, WebRTC they use it.
QoS works. To answer the question in the title of Michael’s paper.

It the app runs in the cloud and the cloud has direct peering links to your
branch office or SP
most likely DSCP works.

And going back to Roland’s proposal, it seems the most natural approach
instead of hacking the semantics.


On Sat 23 Mar 2019 at 18:48, Michael Welzl  wrote:

> Hi,
>
> I’ll do what academics do and add our own data point, below:
>
> On Mar 23, 2019, at 4:19 PM, Roland Bless  wrote:
>
> Hi Mikael,
>
> On 23.03.19 at 11:02 Mikael Abrahamsson wrote:
>
> On Sat, 23 Mar 2019, Roland Bless wrote:
>
> I suggest to use an additional DSCP to mark L4S packets.
>
>
> DSCP doesn't work end-to-end on the Internet, so what you're suggesting
> isn't a workable solution.
>
>
> It's true that DSCPs may be remarked, but RFC 2474
> already stated
>
>   Packets received with an unrecognized codepoint SHOULD be forwarded
>   as if they were marked for the Default behavior (see Sec. 4), and
>   their codepoints should not be changed.
>
> The rtcweb WG also counts on e2e DSCPs.
> After the LE PHB experience I would suggest to probably use
> DSCP 5 which has got better chances to survive ToS bleaching (maybe
> around 80%), if I understand
> https://www.sciencedirect.com/science/article/pii/S0140366417312835
> correctly. Diffserv behavior is usually configurable and can be changed
> without replacing the network gear.
>
>
> Runa Barik, Michael Welzl, Ahmed Mustafa Elmokashfi, Thomas Dreibholz,
> Stein Gjessing: "Can WebRTC QoS Work? A DSCP Measurement Study", 30th
> International Teletraffic Congress (ITC 30), 3 - 7 September 2018, Vienna,
> Austria.
>
> https://itc-conference.org/_Resources/Persistent/780df4482d0fe80f6180f523ebb9482c6869e98b/Barik18ITC30.pdf
>
> We didn’t measure undefined codepoints though, only EF, AF42 and CS1.
> Table 2 compares bleaching for these codepoints with the results in the
> paper you point to; it’s quite similar.
> Well yes, there’s a significant amount of bleaching, we can’t deny that.
> But sometimes the codepoint survives, and it seems to survive surprisingly
> long into the path (fig.4 in our paper).
>
> In the MAPRG session in Prague, Runa Barik will give a talk about the
> measured delay impact of such opportunistic use of these DSCP values by an
> end system.
>
> Cheers,
> Michael
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

[Luca Muscariello]

+1

On Sat, Mar 23, 2019 at 9:02 AM Roland Bless  wrote:

> Hi,
>
> On 22.03.19 at 19:28 Victor Hou wrote:
>
> > Broadcom has been deeply involved in developing the Low Latency DOCSIS
> > cable industry specification that Bob Briscoe mentioned.  We concur with
> > the L4S use of ECT(1).  L4S can be implemented either in a dual-queue or
> > an fq implementation. SCE cannot be implemented with a dual-queue
> > implementation; the only way to support it is with an fq
> > implementation.  An fq implementation is incompatible with the Low
> > Latency DOCSIS specification developed within the cable industry.
>
> I don't understand your rationale here.
> The basic SCE concept could be used for L4S as well.
> I suggest to use an additional DSCP to mark L4S packets.
> The L4S sender/receiver can simply react to the SCE
> markings the same way that they now react to CE, with
> the difference that it's safer to react to SCE, because
> the signal is unambiguous, whereas CE would be ambiguous
> (could be set by either classic AQM/ECN node or by
> an L4S node).
>
> Regards
>  Roland
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

[Luca Muscariello]

To me there is substantial difference from something like fq_codel or
fq_pie where service differentiation is largely implicit
and approches largely based on explicit marking.

Approaches based on marking face technical and non technical challenges
that have been largely mentioned in these lists.

Fq_codel has a ton of litterature behind both theoretical and experimental
and it is something very close to optimality, in terms of completion time
and latency.

Fq_codel also incentivizes the development of better congestion control as
the reward is immediate. It also makes  Internet performance
predictable.

Once we know that, the logical approach would be to try to approximate that
thing when the full mechanism is not possible because of a variety of
limitations.

This is the case in some DC switches that implement AFD+priority fair
queueing at 40Gbps.

Fq_codel has an outstanding footprint in terms of deployment.
Iliad deployed SFQ in 2005 nation wide and Fq_codel as soon as it was
available in France and is the second largest ISP.
Iliad/Free  controls the development of both the home GW and the DSLAM.
They have recently started to commercialize 10Gbps to the home using
switched Ethernet.
I’m very tempted to test it.

Kudos to them for being able to prove it is possible as long as you control
the development of your equipment.

A logical next step  to me seems to push chipcos to build fq_codel in
silicon.
It is totally feasible.

If on the other hand we say that we can achieve all fq_codel provides with
current chipsets we’ll never create the incentives to make progress.

My2c
Luca

On Sun 17 Mar 2019 at 15:06, Mikael Abrahamsson  wrote:

> On Sat, 16 Mar 2019, Holland, Jake wrote:
>
> > Granted, it still remains to be seen whether SCE in practice can match
> > the results of L4S, and L4S was here first.  But it seems to me L4S comes
> > with some problems that have not yet been examined, and that are nicely
> > dodged by a SCE-based approach.
>
> I'm actually not that interested in an academic competition about what
> solution gives the ultimate "best" outcome in simulation or in a lab.
>
> I am interested in good enough solutions that are actually deployable and
> will get deployed, and doesn't have any pathological behaviour when it
> comes to legacy traffic.
>
> Right now the Internet is full of deep FIFOs and they're not going away,
> and they're not getting FQ_CODEL or CAKE.
>
> CAKE/FQ_CODEL is nice, but it's not being deployed at the typical
> congestion points we have in real life. These devices would have a much
> easier time getting PIE or even RED, if it was just implemented.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] netdevconf "vikings"

[Luca Muscariello]

One of the two vikings, Toke, had the whole PhD thesis filled with
citations from Toy Story.
Not to mean that is less glorious... but...

On Tue, Feb 5, 2019 at 10:07 AM Toke Høiland-Jørgensen  wrote:

> Dave Taht  writes:
>
> > speaking of toke and jesper:
> >
> > "Two networking Vikings, masters Jesper Brouer @JesperBrouer and Toke
> > Høiland-Jørgensen will give a tutorial on XDP at 0x13. Join them,
> > listen, get blessed and learn, write and run ebpf/XDP code. Dont
> > forget your laptop!"
> >
> > https://netdevconf.org/0x13/session.html?tutorial-XDP-hands-on
> >
> > But what sort of blessings do vikings bestow?
>
> Well, it's usually something about a glorious death in battle (which is
> how you get into Valhalla). Not sure how appropriate that will be, so we
> may have to improvise ;)
>
> There's a whole sub-genre of Viking metal. E.g.:
> https://www.youtube.com/watch?v=fu2bgwcv43o
>
> (The youtube comments to that video are glorious)
>
> -Toke
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] #brexit bloat

[Luca Muscariello]

LOL!

On Thu 31 Jan 2019 at 15:07, Dave Taht  wrote:

> https://twitter.com/yorksranter/status/1082601273315217408
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cake] paper: per flow fairness in a data center network

[Luca Muscariello]

I disagree on the claims that DC switches do not implement anything.
They do, from quite some time now.

https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-738488.html



On Thu, Dec 6, 2018 at 4:19 AM Dave Taht  wrote:

> While I strongly agree with their premise:
>
> "Multi-tenant DCNs cannot rely on specialized protocols and mechanisms
> that assume single ownership and end-system compliance. It is
> necessary rather to implement general, well-understood mechanisms
> provided as a network service that require as few assumptions about DC
> workload as possible."
>
> ... And there's a solid set of links to current work, and a very
> interesting comparison to pfabric, their DCTCP emulation is too flawed
> to be convincing, and we really should get around to making the ns2
> fq_codel emulation fully match reality. This is also a scenario where
> I'd like to see cake tried, to demonstrate the effectiveness (or not!)
> of 8 way set associative queuing, cobalt, per host/per flow fq, etc,
> vs some of the workloads they outline.
>
> https://perso.telecom-paristech.fr/drossi/paper/rossi18hpsr.pdf
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> ___
> Cake mailing list
> c...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Mario,

agreed.
Two last comments: one should always used fluid approximation with care,
because they are approximations,
the real model is more complex. Nobody considers that the RTT varies during
the connection lifetime and that ACK can be delayed.
So CWD increases in a non simple way.

This is one paper I wrote some time ago where you can get to an equilibrium
with a desired epsilon in the queue.
This protocol and LoLa seem to have similar objectives.

https://arxiv.org/pdf/1010.5623.pdf

Equations and stability conditions are worth a read. Maybe the equations
can be reused for LoLa. As you seem doing something similar,
even if the X(t) introduces non regularities which cannot  be modelled in a
simple way. Maybe yes.

In another work

G. Carofiglio and L. Muscariello, "On the Impact of TCP and Per-Flow
Scheduling on Internet Performance,"
in *IEEE/ACM Transactions on Networking*, vol. 20, no. 2, pp. 620-633,
April 2012.
https://doi.org/10.1109/TNET.2011.2164553

I, my co-author actually, did the calculations nobody wants to do to obtain
the limit cycle which is pretty messy.
They are all fluid approximations. So in practice the target queuing
latency can be larger or smaller.
These models are useful to get a preliminary idea about how the system
works and a publication...




On Fri, Nov 30, 2018 at 10:55 AM Mario Hock  wrote:

> Luca,
>
> I'm not that happy with the theorem either, since it stresses a
> limitation that can actually be overcome. However, I quoted it to
> demonstrate there is a challenge involved.
>
> In my point of view there is actually a subtle thing that's often lost
> when modeling the queue based on input rates, output rates, and, e.g.,
> queuing theory. The inflight data (e.g., controlled by the CWnds) and
> the resulting standing queue. But this standing queue is (probably) the
> major reasoning behind LoLa and CoDel.
>
> Let's simplify the situation to single sender, to ease the explanation.
> (Also holds for multiple senders.) Let the sender have a CWnd-based
> congestion control, but let's fix this CWnd for now. Also, let's assume
> the CWnd is larger than the BDP (without queuing delay); CWnd = BDP + x.
>
> In this case, the sending rate will be above the bottleneck rate, as
> long as less than x is queued at the bottleneck. This increases the
> queue. As soon as x is queued at the bottleneck, the sending rate will
> (exactly) approach the bottleneck rate. The reason is, that the BDP
> (with buffering) will now be exactly equal to the CWnd (since the RTT is
> increased).
>
> This means, a standing queue will build up. But as soon as it's there,
> sending rate will (on average) be equal to the bottleneck rate.
>
> There is also a stability condition here. If the queuing delay (for some
> reason) falls below x/rate (i.e., amount of queuing falls below x), the
> sending rate increases. If the queuing delay raises above x/rate,
> sending rate is reduced. Note: That this is all with a fixed CWnd. No
> interaction of the congestion control algorithms involved here.
>
> Therefore, there can be a long living standing queue, even if input ==
> output. Of course during the build up process input > output, however
> this build up can e.g. happen in one RTT, while the standing queue can
> survive seconds or minutes (in reality, and indefinitely in theory).
> Though, I found that this correlation is often not modeled in
> queue-theoretical models.
>
> Best, Mario
>
>
> Am 29.11.18 um 23:30 schrieb Luca Muscariello:
> > Mario,
> >
> > putting aside LoLa for a second,
> > I'm not quite sure that the theorem you cite is valid.
> >
> > According to the model R_i is the sending rate.
> > The sum across all flows bottlenecked at the link does not need to be
> > equal to the link capacity.
> > The input rate can be instantaneously below or above the link capacity.
> > Even If the link is never idle and the output rate is always C for all t.
> >
> > If the sum_i R_i = C is false because of what I said, than p_i which is
> > nothing more than
> > the shadow price of the link capacity constraint, can be a function of a
> > constant delay d, i.e. p_i = cost * d for all i.
> >
> > This theorem can be valid only if the input rate of a queue is
> > instantaneously equal to the output queue.
> > We all know that a queue exists just because there is an instantaneous
> > difference of input and output rates
> > at the link. So to conclude,  this theorem if valid iff input == output
> > rate then the queue is always zero, i.e.  d=0.
> >
> > The theorem is either an artefact of the model or just wrong. Or I'm
> > missing something...
> >
> >
> >
> >
> >
> >
> > On Thu, N

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Mario,

putting aside LoLa for a second,
I'm not quite sure that the theorem you cite is valid.

According to the model R_i is the sending rate.
The sum across all flows bottlenecked at the link does not need to be equal
to the link capacity.
The input rate can be instantaneously below or above the link capacity.
Even If the link is never idle and the output rate is always C for all t.

If the sum_i R_i = C is false because of what I said, than p_i which is
nothing more than
the shadow price of the link capacity constraint, can be a function of a
constant delay d, i.e. p_i = cost * d for all i.

This theorem can be valid only if the input rate of a queue is
instantaneously equal to the output queue.
We all know that a queue exists just because there is an instantaneous
difference of input and output rates
at the link. So to conclude,  this theorem if valid iff input == output
rate then the queue is always zero, i.e.  d=0.

The theorem is either an artefact of the model or just wrong. Or I'm
missing something...






On Thu, Nov 29, 2018 at 6:07 PM Mario Hock  wrote:

> Hi Luca,
>
> I'm answering on behalf of Roland, since I am a co-author of the paper.
>
> This is an excellent question, since it goes right at the heart of how
> LoLa works.
>
> Indeed, the paper is a first of a series. A second one, looking deeper
> into the fair flow balancing mechanism, is currently under submission.
>
> Similar as other delay based congestion controls, LoLa tries to achieve
> fairness by allowing each flow to buffer the same amount of data at the
> bottleneck. We have this, e.g., in TCP Vegas, and (in a way) also in
> Copa (a recently proposed congestion control) and many others. If this
> is achieved, we get flow rate fairness independent of a flow's RTT.
>
> Usually (in other congestion controls) this "allowed amount of data" is
> fixed per flow. We presume that this approach does not scale well to
> high speed networks. Since the queuing delay resulting from this amount
> of data is reduced with increasing bottleneck rate. Thus, it becomes
> harder to measure it right. This can easily be seen (and proven) for TCP
> Vegas.
>
> Note: Just using higher fixed values is not an option, since it would
> not work at lower speeds anymore and also not with a large number of flows.
>
> Therefore, LoLa tries to find a suitable value for the "allowed amount
> of data" dynamically. This is X(t).
>
> Our approach is to grow X(t) over time during the Fair Flow Balancing
> phase. This phase ends when the queuing delay reaches 5ms. Thus, (in the
> ideal case) at the end of Fair Flow Balancing, X(t) is just as large
> that all flows at bottleneck create a queuing delay of 5ms, and all
> flows contribute equally to this queue. Hence, flow rate fairness is
> achieved. (Note that LoLa is designed in a way that t is (almost)
> synchronized among the competing flows.)
>
> Generally, other ways of determining a suitable X(t) are conceivable. In
> our approach X(t) is a monotonically increasing function, but it is
> regularly reset as LoLa cycles between its states; i.e., after a queuing
> delay of 5ms is reached, the queue is drained and everything starts
> again. (Thus, the timespan where X(t) is monotonically increased is
> called a "round of fair flow balancing".)
>
> This way we can overcome the constraint given in [1]:
>
> """
> THEOREM 6 (FAIRNESS/DELAY TRADEOFF). For congestion control mechanisms
> that have steady state throughput of the kind R = f(d, p), for some
> function f, delay d and feedback p, if the feedback is based on purely
> end to end delay measurements, you can either have fairness or a fixed
> delay, but not both simultaneously
> """
>
> [1] "ECN or Delay: Lessons Learnt from Analysis of DCQCN and TIMELY"
> Yibo Zhu et al., https://dl.acm.org/citation.cfm?id=2999593
>
> Best, Mario
>
>
> Am 29.11.18 um 17:09 schrieb Luca Muscariello:
> > Hi Roland,
> >
> > It took me quite a lot of time to find this message in the thread...
> > I read the paper you sent and I guess this is the first of a series as
> > many things stay uncovered.
> >
> > Just a quick question: why is X(t) always increasing with  t?
> >
> >
> > On Tue, Nov 27, 2018 at 11:26 AM Bless, Roland (TM)
> > mailto:roland.bl...@kit.edu>> wrote:
> >
> > Hi Luca,
> >
> > Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> >  > A congestion controlled protocol such as TCP or others, including
> > QUIC,
> >  > LEDBAT and so on
> >  > need at least the BDP in the transmission queue to get full link
> >  > efficiency, i.e. the queue never empties out.
>

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Hi Roland,

It took me quite a lot of time to find this message in the thread...
I read the paper you sent and I guess this is the first of a series as many
things stay uncovered.

Just a quick question: why is X(t) always increasing with  t?


On Tue, Nov 27, 2018 at 11:26 AM Bless, Roland (TM) 
wrote:

> Hi Luca,
>
> Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > A congestion controlled protocol such as TCP or others, including QUIC,
> > LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> > efficiency, i.e. the queue never empties out.
>
> This is not true. There are congestion control algorithms
> (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck link
> capacity without filling the buffer to its maximum capacity. The BDP
> rule of thumb basically stems from the older loss-based congestion
> control variants that profit from the standing queue that they built
> over time when they detect a loss:
> while they back-off and stop sending, the queue keeps the bottleneck
> output busy and you'll not see underutilization of the link. Moreover,
> once you get good loss de-synchronization, the buffer size requirement
> for multiple long-lived flows decreases.
>
> > This gives rule of thumbs to size buffers which is also very practical
> > and thanks to flow isolation becomes very accurate.
>
> The positive effect of buffers is merely their role to absorb
> short-term bursts (i.e., mismatch in arrival and departure rates)
> instead of dropping packets. One does not need a big buffer to
> fully utilize a link (with perfect knowledge you can keep the link
> saturated even without a single packet waiting in the buffer).
> Furthermore, large buffers (e.g., using the BDP rule of thumb)
> are not useful/practical anymore at very high speed such as 100 Gbit/s:
> memory is also quite costly at such high speeds...
>
> Regards,
>  Roland
>
> [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> 215-218, Singapore, Singapore, October 2017
> http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
>
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> > the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> > literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> > the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> > RTT you can serve at full utilization.
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> > to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> > use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> > mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> > Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> > that all the others cannot have.
> > However the others, the last one especially can be implemented in
> > silicon with reasonable cost.
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

On Wed, Nov 28, 2018 at 11:40 AM Dave Taht  wrote:

> On Wed, Nov 28, 2018 at 1:56 AM Luca Muscariello
>  wrote:
> >
> > Dave,
> >
> > The single BDP inflight is a rule of thumb that does not account for
> fluctuations of the RTT.
> > And I am not talking about random fluctuations and noise. I am talking
> about fluctuations
> > from a control theoretic point of view to stabilise the system, e.g. the
> trajectory of the system variable that
> > gets to the optimal point no matter the initial conditions (Lyapunov).
>
> I have been trying all day to summon the gumption to make this argument:
>
> IF you have a good idea of the actual RTT...
>
> it is also nearly certain that there will be *at least* one other flow
> you will be competing with...
> therefore the fluctuations from every point of view are dominated by
> the interaction between these flows and
> the goal is, in general, is not to take up a full BDP for your single flow.
>
> And BBR aims for some tiny percentage less than what it thinks it can
> get, when, well, everybody's seen it battle it out with itself and
> with cubic. I hand it FQ at the bottleneck link and it works well.
>
> single flows exist only in the minds of theorists and labs.
>
> There's a relevant passage worth citing in the kleinrock paper, I
> thought (did he write two recently?) that talked about this problem...
> I *swear* when I first read it it had a deeper discussion of the
> second sentence below and had two paragraphs that went into the issues
> with multiple flows:
>
> "ch earlier and led to the Flow Deviation algorithm [28]. 17 The
> reason that the early work of 40 years ago took so long to make its
> current impact is because in [31] it was shown that the mechanism
> presented in [2] and [3] could not be implemented in a decentralized
> algorithm. This delayed the application of Power until the recent work
> by the Google team in [1] demonstrated that the key elements of
> response time and bandwidth could indeed be estimated using a
> distributed control loop sliding window spanning approximately 10
> round-trip times."
>
> but I can't find it today.
>
>
Here it is

https://www.lk.cs.ucla.edu/data/files/Kleinrock/Internet%20Congestion%20Control%20Using%20the%20Power%20Metric-Keep%20the%20Pipe%20Just%20Full%2C%20But%20No%20Fuller%20July%202018.pdf


> > The ACM queue paper talking about Codel makes a fairly intuitive and
> accessible explanation of that.
>
> I haven't re-read the lola paper. I just wanted to make the assertion
> above. And then duck. :)
>
> Also, when I last looked at BBR, it made a false assumption that 200ms
> was "long enough" to probe the actual RTT, when my comcast links and
> others are measured at 680ms+ of buffering.
>

This is essentially the same paper I cited which is Part I.



>
> And I always liked the stanford work, here, which tried to assert that
> a link with n flows requires no more than B = (RTT ×C)/ √ n.
>
> http://yuba.stanford.edu/techreports/TR04-HPNG-060800.pdf


That that paper does not say that that rule ALWAYS apply. It does under
certain conditions.
But my point is about optimality.

I does NOT mean that the system HAS to work ALWAYS in that point because
things change.

And for BBR, I would say that one thing is the design principles another is
the implementations
and we better distinguish between them. The key design principles are all
valid.



>
>
> night!
>
>
night ;)


>
>
> > There is a less accessible literature talking about that, which dates
> back to some time ago
> > that may be useful to re-read again
> >
> > Damon Wischik and Nick McKeown. 2005.
> > Part I: buffer sizes for core routers.
> > SIGCOMM Comput. Commun. Rev. 35, 3 (July 2005), 75-78. DOI=
> http://dx.doi.org/10.1145/1070873.1070884
> > http://klamath.stanford.edu/~nickm/papers/BufferSizing.pdf.pdf
> >
> > and
> >
> > Gaurav Raina, Don Towsley, and Damon Wischik. 2005.
> > Part II: control theory for buffer sizing.
> > SIGCOMM Comput. Commun. Rev. 35, 3 (July 2005), 79-82.
> > DOI=http://dx.doi.org/10.1145/1070873.1070885
> > http://www.statslab.cam.ac.uk/~gr224/PAPERS/Control_Theory_Buffers.pdf
> >
> > One of the thing that Frank Kelly has brought to the literature is about
> optimal control.
> > From a pure optimization point of view we know since Robert Gallagher
> (and Bertsekas 1981) that
> > the optimal sending rate is a function of the shadow price at the
> bottleneck.
> > This shadow price is nothing more than the Lagrange multiplier of the
> capacity constraint
> > at the bottleneck. Some protocols such as XCP or RCP propose to carry
> someth

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Dave,

The single BDP inflight is a rule of thumb that does not account for
fluctuations of the RTT.
And I am not talking about random fluctuations and noise. I am talking
about fluctuations
from a control theoretic point of view to stabilise the system, e.g. the
trajectory of the system variable that
gets to the optimal point no matter the initial conditions (Lyapunov).
The ACM queue paper talking about Codel makes a fairly intuitive and
accessible explanation of that.

There is a less accessible literature talking about that, which dates back
to some time ago
that may be useful to re-read again

Damon Wischik and Nick McKeown. 2005.
Part I: buffer sizes for core routers.
SIGCOMM Comput. Commun. Rev. 35, 3 (July 2005), 75-78. DOI=
http://dx.doi.org/10.1145/1070873.1070884
http://klamath.stanford.edu/~nickm/papers/BufferSizing.pdf.pdf

and

Gaurav Raina, Don Towsley, and Damon Wischik. 2005.
Part II: control theory for buffer sizing.
SIGCOMM Comput. Commun. Rev. 35, 3 (July 2005), 79-82.
DOI=http://dx.doi.org/10.1145/1070873.1070885
http://www.statslab.cam.ac.uk/~gr224/PAPERS/Control_Theory_Buffers.pdf

One of the thing that Frank Kelly has brought to the literature is about
optimal control.
>From a pure optimization point of view we know since Robert Gallagher (and
Bertsekas 1981) that
the optimal sending rate is a function of the shadow price at the
bottleneck.
This shadow price is nothing more than the Lagrange multiplier of the
capacity constraint
at the bottleneck. Some protocols such as XCP or RCP propose to carry
something
very close to a shadow price in the ECN but that's not that simple.
And currently we have a 0/1 "shadow price" which way insufficient.

Optimal control as developed by Frank Kelly since 1998 tells you that you
have
a stability region that is needed to get to the optimum.

Wischik work, IMO, helps quite a lot to understand tradeoffs while
designing AQM
and CC. I feel like the people who wrote the codel ACM Queue paper are very
much aware of this literature,
because Codel design principles seem to take into account that.
And the BBR paper too.


On Tue, Nov 27, 2018 at 9:58 PM Dave Taht  wrote:

> OK, wow, this conversation got long. and I'm still 20 messages behind.
>
> Two points, and I'm going to go back to work, and maybe I'll try to
> summarize a table
> of the competing viewpoints, as there's far more than BDP of
> discussion here, and what
> we need is sqrt(bdp) to deal with all the different conversational flows.
> :)
>
> On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello
>  wrote:
> >
> > I think that this is a very good comment to the discussion at the
> defense about the comparison between
> > SFQ with longest queue drop and FQ_Codel.
> >
> > A congestion controlled protocol such as TCP or others, including QUIC,
> LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> efficiency, i.e. the queue never empties out.
>
> no, I think it needs a BDP in flight.
>
> I think some of the confusion here is that your TCP stack needs to
> keep around a BDP in order to deal with
> retransmits, but that lives in another set of buffers entirely.
>
> > This gives rule of thumbs to size buffers which is also very practical
> and thanks to flow isolation becomes very accurate.
> >
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> RTT you can serve at full utilization.
>
> My own take on the whole BDP argument is that *so long as the flows in
> that BDP are thoroughly mixed* you win.
>
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the E

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I suggest re-reading this

https://queue.acm.org/detail.cfm?id=3022184


On Tue 27 Nov 2018 at 21:58, Dave Taht  wrote:

> OK, wow, this conversation got long. and I'm still 20 messages behind.
>
> Two points, and I'm going to go back to work, and maybe I'll try to
> summarize a table
> of the competing viewpoints, as there's far more than BDP of
> discussion here, and what
> we need is sqrt(bdp) to deal with all the different conversational flows.
> :)
>
> On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello
>  wrote:
> >
> > I think that this is a very good comment to the discussion at the
> defense about the comparison between
> > SFQ with longest queue drop and FQ_Codel.
> >
> > A congestion controlled protocol such as TCP or others, including QUIC,
> LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> efficiency, i.e. the queue never empties out.
>
> no, I think it needs a BDP in flight.
>
> I think some of the confusion here is that your TCP stack needs to
> keep around a BDP in order to deal with
> retransmits, but that lives in another set of buffers entirely.
>
> > This gives rule of thumbs to size buffers which is also very practical
> and thanks to flow isolation becomes very accurate.
> >
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> RTT you can serve at full utilization.
>
> My own take on the whole BDP argument is that *so long as the flows in
> that BDP are thoroughly mixed* you win.
>
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> that all the others cannot have.
> > However the others, the last one especially can be implemented in
> silicon with reasonable cost.
> >
> >
> >
> >
> >
> > On Mon 26 Nov 2018 at 22:30, Jonathan Morton 
> wrote:
> >>
> >> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
> >> >
> >> > So I just thought to continue the discussion- when does the CoDel
> part of fq_codel actually help in the real world?
> >>
> >> Fundamentally, without Codel the only limits on the congestion window
> would be when the sender or receiver hit configured or calculated rwnd and
> cwnd limits (the rwnd is visible on the wire and usually chosen to be large
> enough to be a non-factor), or when the queue overflows.  Large windows
> require buffer memory in both sender and receiver, increasing costs on the
> sender in particular (who typically has many flows to manage per machine).
> >>
> >> Queue overflow tends to result in burst loss and head-of-line blocking
> in the receiver, which is visible to the user as a pause and subsequent
> jump in the progress of their download, accompanied by a major fluctuation
> in the estimated time to completion.  The lost packets also consume
> capacity upstream of the bottleneck which does not contribute to
> application throughput.  These effects are independent of whether overflow
> dropping occurs at the head or tail of the bottleneck queue, though
> recovery occurs more quickly (and fewer packets might be lost) if dropping
> occurs from the head of the queue.
> >>
> >> From a pure throughput-efficiency standpoint, Codel allows using ECN
> for congestion signalling instead of packet loss, potentially eliminating
> packet loss and associated lead-of-line blocking entirely.  Even without
> ECN, the actual cwnd is kept near the minimum necessary to satisfy the BDP
> of the path, reducing memory r

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

On Tue, Nov 27, 2018 at 2:49 PM Mikael Abrahamsson  wrote:

> On Tue, 27 Nov 2018, Luca Muscariello wrote:
>
> > If you, Mikael don't want more than 10ms buffer, how do you achieve that?
>
> class class-default
>random-detect 10 ms 2000 ms
>
> That's the only thing available to me on the platforms I have. If you
> would like this improved, please reach out to the Cisco ASR9k BU and tell
> them to implement ECN and PIE (or something even better). They won't do it
> because I say so, it seems. WRED is all they give me.
>

This is a whole different discussion but if you want to have a per-user
context
at the BNG level + TM + FQ I'm not sure that kind of beast will ever exist.
Unless you have a very small user fan-out the hardware clocks could loop
over
several thousands of contexts.
You should expect those kind of features to be in the CMTS or OLT.


>
> > You change the behaviour of the source and hope flow isolation is
> available.
>
> Sorry, I only transport the packets, I don't create them.
>

I'm sure you create a lot of packets. Don't be humble.


>
> > If you just cut the buffer down to 10ms and do nothing else, the only
> thing
> > you get is a short queue and may throw away half of your link capacity.
>
> If i have lots of queue I might instead get customer complaints about high
> latency for their interactive applications.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Another bit to this.
A router queue is supposed to serve packets no matter what is running at
the controlled end-point, BBR, Cubic or else.
So, delay-based congestion controller still get hurt in today Internet
unless they can get their portion of buffer at the line card.
FQ creates incentives for end-points to send traffic in a smoother way
because the reward to the application is immediate and
measurable. But the end-point does not know in advance if FQ is there or
not.

So going back to sizing the link buffer, the rule I mentioned applies. And
it allows to get best completion times for a wider range of RTTs.
If you, Mikael don't want more than 10ms buffer, how do you achieve that?
You change the behaviour of the source and hope flow isolation is available.
If you just cut the buffer down to 10ms and do nothing else, the only thing
you get is a short queue and may throw away half of your link capacity.



On Tue, Nov 27, 2018 at 1:17 PM Jonathan Morton 
wrote:

> > On 27 Nov, 2018, at 1:21 pm, Mikael Abrahamsson 
> wrote:
> >
> > It's complicated. I've had people throw in my face that I need 2xBDP in
> buffer size to smoothe things out. Personally I don't want more than 10ms
> buffer (max), and I don't see why I should need more than that even if
> transfers are running over hundreds of ms of light-speed-in-medium induced
> delay between the communicating systems.
>
> I think we can agree that the ideal CC algo would pace packets out
> smoothly at exactly the path capacity, neither building a queue at the
> bottleneck nor leaving capacity on the table.
>
> Actually achieving that in practice turns out to be difficult, because
> there's no general way to discover the path capacity in advance.  AQMs like
> Codel, in combination with ECN, get us a step closer by explicitly
> informing each flow when it is exceeding that capacity while the queue is
> still reasonably short.  FQ also helps, by preventing flows from
> inadvertently interfering with each other by imperfectly managing their
> congestion windows.
>
> So with the presently deployed state of the art, we have cwnds oscillating
> around reasonably short queue lengths, backing off sharply in response to
> occasional signals, then probing back upwards when that signal goes away
> for a while.  It's a big improvement over dumb drop-tail FIFOs, but it's
> still some distance from the ideal.  That's because the information
> injected by the bottleneck AQM is a crude binary state.
>
> I do not include DCTCP in the deployed state of the art, because it is not
> deployable in the RFC-compliant Internet; it is effectively incompatible
> with Codel in particular, because it wrongly interprets CE marks and is
> thus noncompliant with the ECN RFC.
>
> However, I agree with DCTCP's goal of achieving finer-grained control of
> the cwnd, through AQMs providing more nuanced information about the state
> of the path capacity and/or bottleneck queue.  An implementation that made
> use of ECT(1) instead of changing the meaning of CE marks would remain
> RFC-compliant, and could get "sufficiently close" to the ideal described
> above.
>
>  - Jonathan Morton
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

A buffer in a router is sized once. RTT varies.
So BDP varies. That’s as simple as that.
So you just cannot be always at optimum because you don’t know what RTT you
have at any time.

Lola si not solving that. No protocol could BTW.
BTW I don’t see  any formal proof about queue occupancy in the paper.



On Tue 27 Nov 2018 at 12:53, Bless, Roland (TM) 
wrote:

> Hi Luca,
>
> Am 27.11.18 um 12:01 schrieb Luca Muscariello:
> > A BDP is not a large buffer. I'm not unveiling a secret.
>
> That depends on speed and RTT (note that typically there are
> several flows with different RTTs sharing the same buffer).
> The essential point is not how much buffer capacity is available,
> but how much is actually used, because that adds queueing delay.
>
> > And it is just a rule of thumb to have an idea at which working point
> > the protocol is working.
>
> No, one can actually prove that this is the best size for
> loss-based CC with backoff factor of 0.5 (assuming a single flow).
>
> > In practice the protocol is usually working below or above that value.
>
> That depends on the protocol.
>
> > This is where AQM and ECN help also. So most of the time the protocol is
> > working at way
> > below 100% efficiency.
>
> > My point was that FQ_codel helps to get very close to the optimum w/o
> > adding useless queueing and latency.
> > With a single queue that's almost impossible. No, sorry. Just impossible.
>
> No, it's possible. Please read the TCP LoLa paper.
>
> Regards,
>  Roland
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

A BDP is not a large buffer. I'm not unveiling a secret.
And it is just a rule of thumb to have an idea at which working point the
protocol is working.
In practice the protocol is usually working below or above that value.
This is where AQM and ECN help also. So most of the time the protocol is
working at way
below 100% efficiency.

My point was that FQ_codel helps to get very close to the optimum w/o
adding useless queueing and latency.
With a single queue that's almost impossible. No, sorry. Just impossible.



On Tue, Nov 27, 2018 at 11:50 AM Mikael Abrahamsson 
wrote:

> On Tue, 27 Nov 2018, Luca Muscariello wrote:
>
> > link fully utilized is defined as Q>0 unless you don't include the
> > packet currently being transmitted. I do, so the TXtteer is never idle.
> > But that's a detail.
>
> As someone who works with moving packets, it's perplexing to me to
> interact with transport peeps who seem enormously focused on "goodput". My
> personal opinion is that most people would be better off with 80% of their
> available bandwidth being in use without any noticable buffer induced
> delay, as opposed to the transport protocol doing its damndest to fill up
> the link to 100% and sometimes failing and inducing delay instead.
>
> Could someone perhaps comment on the thinking in the transport protocol
> design "crowd" when it comes to this?
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

OK. We agree.
That's correct, you need *at least* the BDP in flight so that the
bottleneck queue never empties out.

This can be easily proven using fluid models for any congestion controlled
source no matter if it is
loss-based, delay-based, rate-based, formula-based etc.

A highly paced source gives you the ability to get as close as
theoretically possible to the BDP+epsilon
as possible.

link fully utilized is defined as Q>0 unless you don't include the packet
currently being transmitted. I do,
so the TXtteer is never idle. But that's a detail.



On Tue, Nov 27, 2018 at 11:35 AM Bless, Roland (TM) 
wrote:

> Hi,
>
> Am 27.11.18 um 11:29 schrieb Luca Muscariello:
> > I have never said that you need to fill the buffer to the max size to
> > get full capacity, which is an absurdity.
>
> Yes, it's absurd, but that's what today's loss-based CC algorithms do.
>
> > I said you need at least the BDP so that the queue never empties out.
> > The link is fully utilized IFF the queue is never emptied.
>
> I was also a bit imprecise: you'll need a BDP in flight, but
> you don't need to fill the buffer at all. The latter sentence
> is valid only in the direction: queue not empty -> link fully utilized.
>
> Regards,
>  Roland
>
> >
> >
> >
> > On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM)  > <mailto:roland.bl...@kit.edu>> wrote:
> >
> > Hi Luca,
> >
> > Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > > A congestion controlled protocol such as TCP or others, including
> > QUIC,
> > > LEDBAT and so on
> > > need at least the BDP in the transmission queue to get full link
> > > efficiency, i.e. the queue never empties out.
> >
> > This is not true. There are congestion control algorithms
> > (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck
> link
> > capacity without filling the buffer to its maximum capacity. The BDP
> > rule of thumb basically stems from the older loss-based congestion
> > control variants that profit from the standing queue that they built
> > over time when they detect a loss:
> > while they back-off and stop sending, the queue keeps the bottleneck
> > output busy and you'll not see underutilization of the link.
> Moreover,
> > once you get good loss de-synchronization, the buffer size
> requirement
> > for multiple long-lived flows decreases.
> >
> > > This gives rule of thumbs to size buffers which is also very
> practical
> > > and thanks to flow isolation becomes very accurate.
> >
> > The positive effect of buffers is merely their role to absorb
> > short-term bursts (i.e., mismatch in arrival and departure rates)
> > instead of dropping packets. One does not need a big buffer to
> > fully utilize a link (with perfect knowledge you can keep the link
> > saturated even without a single packet waiting in the buffer).
> > Furthermore, large buffers (e.g., using the BDP rule of thumb)
> > are not useful/practical anymore at very high speed such as 100
> Gbit/s:
> > memory is also quite costly at such high speeds...
> >
> > Regards,
> >  Roland
> >
> > [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> > TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> > Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> > 215-218, Singapore, Singapore, October 2017
> > http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
> >
> > > Which is:
> > >
> > > 1) find a way to keep the number of backlogged flows at a
> > reasonable value.
> > > This largely depends on the minimum fair rate an application may
> > need in
> > > the long term.
> > > We discussed a little bit of available mechanisms to achieve that
> > in the
> > > literature.
> > >
> > > 2) fix the largest RTT you want to serve at full utilization and
> size
> > > the buffer using BDP * N_backlogged.
> > > Or the other way round: check how much memory you can use
> > > in the router/line card/device and for a fixed N, compute the
> largest
> > > RTT you can serve at full utilization.
> > >
> > > 3) there is still some memory to dimension for sparse flows in
> > addition
> > > to that, but this is not based on BDP.
> > > It is just enough to compute the total utilization of sparse flows
> and
> >  

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I have never said that you need to fill the buffer to the max size to get
full capacity, which is an absurdity.

I said you need at least the BDP so that the queue never empties out.
The link is fully utilized IFF the queue is never emptied.



On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM) 
wrote:

> Hi Luca,
>
> Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > A congestion controlled protocol such as TCP or others, including QUIC,
> > LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> > efficiency, i.e. the queue never empties out.
>
> This is not true. There are congestion control algorithms
> (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck link
> capacity without filling the buffer to its maximum capacity. The BDP
> rule of thumb basically stems from the older loss-based congestion
> control variants that profit from the standing queue that they built
> over time when they detect a loss:
> while they back-off and stop sending, the queue keeps the bottleneck
> output busy and you'll not see underutilization of the link. Moreover,
> once you get good loss de-synchronization, the buffer size requirement
> for multiple long-lived flows decreases.
>
> > This gives rule of thumbs to size buffers which is also very practical
> > and thanks to flow isolation becomes very accurate.
>
> The positive effect of buffers is merely their role to absorb
> short-term bursts (i.e., mismatch in arrival and departure rates)
> instead of dropping packets. One does not need a big buffer to
> fully utilize a link (with perfect knowledge you can keep the link
> saturated even without a single packet waiting in the buffer).
> Furthermore, large buffers (e.g., using the BDP rule of thumb)
> are not useful/practical anymore at very high speed such as 100 Gbit/s:
> memory is also quite costly at such high speeds...
>
> Regards,
>  Roland
>
> [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> 215-218, Singapore, Singapore, October 2017
> http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
>
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> > the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> > literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> > the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> > RTT you can serve at full utilization.
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> > to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> > use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> > mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> > Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> > that all the others cannot have.
> > However the others, the last one especially can be implemented in
> > silicon with reasonable cost.
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I think that this is a very good comment to the discussion at the defense
about the comparison between
SFQ with longest queue drop and FQ_Codel.

A congestion controlled protocol such as TCP or others, including QUIC,
LEDBAT and so on
need at least the BDP in the transmission queue to get full link
efficiency, i.e. the queue never empties out.
This gives rule of thumbs to size buffers which is also very practical and
thanks to flow isolation becomes very accurate.

Which is:

1) find a way to keep the number of backlogged flows at a reasonable value.
This largely depends on the minimum fair rate an application may need in
the long term.
We discussed a little bit of available mechanisms to achieve that in the
literature.

2) fix the largest RTT you want to serve at full utilization and size the
buffer using BDP * N_backlogged.
Or the other way round: check how much memory you can use
in the router/line card/device and for a fixed N, compute the largest RTT
you can serve at full utilization.

3) there is still some memory to dimension for sparse flows in addition to
that, but this is not based on BDP.
It is just enough to compute the total utilization of sparse flows and use
the same simple model Toke has used
to compute the (de)prioritization probability.

This procedure would allow to size FQ_codel but also SFQ.
It would be interesting to compare the two under this buffer sizing.
It would also be interesting to compare another mechanism that we have
mentioned during the defense
which is AFD + a sparse flow queue. Which is, BTW, already available in
Cisco nexus switches for data centres.

I think that the the codel part would still provide the ECN feature, that
all the others cannot have.
However the others, the last one especially can be implemented in silicon
with reasonable cost.





On Mon 26 Nov 2018 at 22:30, Jonathan Morton  wrote:

> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
> >
> > So I just thought to continue the discussion- when does the CoDel part
> of fq_codel actually help in the real world?
>
> Fundamentally, without Codel the only limits on the congestion window
> would be when the sender or receiver hit configured or calculated rwnd and
> cwnd limits (the rwnd is visible on the wire and usually chosen to be large
> enough to be a non-factor), or when the queue overflows.  Large windows
> require buffer memory in both sender and receiver, increasing costs on the
> sender in particular (who typically has many flows to manage per machine).
>
> Queue overflow tends to result in burst loss and head-of-line blocking in
> the receiver, which is visible to the user as a pause and subsequent jump
> in the progress of their download, accompanied by a major fluctuation in
> the estimated time to completion.  The lost packets also consume capacity
> upstream of the bottleneck which does not contribute to application
> throughput.  These effects are independent of whether overflow dropping
> occurs at the head or tail of the bottleneck queue, though recovery occurs
> more quickly (and fewer packets might be lost) if dropping occurs from the
> head of the queue.
>
> From a pure throughput-efficiency standpoint, Codel allows using ECN for
> congestion signalling instead of packet loss, potentially eliminating
> packet loss and associated lead-of-line blocking entirely.  Even without
> ECN, the actual cwnd is kept near the minimum necessary to satisfy the BDP
> of the path, reducing memory requirements and significantly shortening the
> recovery time of each loss cycle, to the point where the end-user may not
> notice that delivery is not perfectly smooth, and implementing accurate
> completion time estimators is considerably simplified.
>
> An important use-case is where two sequential bottlenecks exist on the
> path, the upstream one being only slightly higher capacity but lacking any
> queue management at all.  This is presently common in cases where home CPE
> implements inbound shaping on a generic ISP last-mile link.  In that case,
> without Codel running on the second bottleneck, traffic would collect in
> the first bottleneck's queue as well, greatly reducing the beneficial
> effects of FQ implemented on the second bottleneck.  In this topology, the
> overall effect is inter-flow as well as intra-flow.
>
> The combination of Codel with FQ is done in such a way that a separate
> instance of Codel is implemented for each flow.  This means that congestion
> signals are only sent to flows that require them, and non-saturating flows
> are unmolested.  This makes the combination synergistic, where each
> component offers an improvement to the behaviour of the other.
>
>  - Jonathan Morton
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Luca Muscariello]

On Fri 23 Nov 2018 at 18:27, Dave Taht  wrote:

> On Fri, Nov 23, 2018 at 9:17 AM Luca Muscariello
>  wrote:
> >
> > Yes there was some discussion about that.
> > Moving things to hardware should fix that.
> >
> > Evens traffic management in NPU based routers makes use of hardware
> based polling for shaping. These are trade offs one has to face all the
> time.
> >
> > There has been a discussion at the defense about hardware vs software,
> hardware + software, when one, when the other.
>
> I'm still listening/watching.
>
> >
> > BTW Toke is Doctor Toke now :-)
>
> I hope you made him sweat, at least a little. :)
>
>
Difficult to sweet here. It’s freezing, but I’m leaving already. Toke I
think has already started to eat and drink too much...





> >
> >
> >
> > On Fri 23 Nov 2018 at 17:48, Dave Taht  wrote:
> >>
> >> Ahhh good. My morning has improved. I found the coffee, and I have
> >> something way more interesting that CNN on.
> >>
> >> https://www.youtube.com/watch?v=upvx6rpSLSw=youtu.be
> >>
> >> On Fri, Nov 23, 2018 at 8:43 AM Jonathan Morton 
> wrote:
> >> >
> >> > > On 23 Nov, 2018, at 6:26 pm, Dave Taht  wrote:
> >> > >
> >> > >> (I noticed an audience member brought this up in Toke’s thesis
> >> > >> defense)
> >> > >
> >> > > I sadly slept through that. I hope it was recorded.
> >> >
> >> > As it was streamed on YT, YT archived it.
> >> >
> >> >  - Jonathan Morton
> >> >
> >> > ___
> >> > Bloat mailing list
> >> > Bloat@lists.bufferbloat.net
> >> > https://lists.bufferbloat.net/listinfo/bloat
> >>
> >>
> >>
> >> --
> >>
> >> Dave Täht
> >> CTO, TekLibre, LLC
> >> http://www.teklibre.com
> >> Tel: 1-831-205-9740
> >> ___
> >> Bloat mailing list
> >> Bloat@lists.bufferbloat.net
> >> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Luca Muscariello]

Yes there was some discussion about that.
Moving things to hardware should fix that.

Evens traffic management in NPU based routers makes use of hardware based
polling for shaping. These are trade offs one has to face all the time.

There has been a discussion at the defense about hardware vs software,
hardware + software, when one, when the other.


BTW Toke is Doctor Toke now :-)



On Fri 23 Nov 2018 at 17:48, Dave Taht  wrote:

> Ahhh good. My morning has improved. I found the coffee, and I have
> something way more interesting that CNN on.
>
> https://www.youtube.com/watch?v=upvx6rpSLSw=youtu.be
>
> On Fri, Nov 23, 2018 at 8:43 AM Jonathan Morton 
> wrote:
> >
> > > On 23 Nov, 2018, at 6:26 pm, Dave Taht  wrote:
> > >
> > >> (I noticed an audience member brought this up in Toke’s thesis
> > >> defense)
> > >
> > > I sadly slept through that. I hope it was recorded.
> >
> > As it was streamed on YT, YT archived it.
> >
> >  - Jonathan Morton
> >
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] [Cerowrt-devel] closing up my make-wifi-fast lab

[Luca Muscariello]

Hi Bob,

I meant licensed/unlicensed for private/non private.

Luca

On Mon, Aug 27, 2018 at 9:39 AM Bob McMahon 
wrote:

> Hi Luca,
>
> What is non private spectrum defined as per  "I don't yet see how a non
> private spectrum can be shared  w/o LBT."
>
> Thanks,
> Bob
>
>
>
>
> On Mon, Aug 27, 2018 at 12:24 AM Luca Muscariello <
> luca.muscarie...@gmail.com> wrote:
>
>> Jonathan,
>>
>> Not that giant handwaving though.
>> IEEE 802.11ax makes use of "almost TDM" RTS/CTS and scheduling. The
>> almost is necessary as it operates in 2.4/5Ghz bands.
>> Similar to what you describe, and is coming very soon in shipping
>> products.
>>
>> RTS/CTS is still a LBT to create a window where TDM can be done.
>> I don't yet see how a non private spectrum can be shared  w/o LBT.
>>
>> On the other hand, medium sharing is one thing, the other thing is
>> capacity.
>> There is no way to efficiently share a medium if this is used close to
>> its theoretical capacity.
>>
>> Capacity as #of stations per band including #SSID per band. Today scaling
>> can be achieved
>> with careful radio planning for spatial diversity or dynamic bean forming.
>>
>> When you approach capacity with WiFi you only see beacon traffic and
>> almost zero throughput.
>> Cannot forget Mobile World Congress where you can measure several
>> thousands of SSIDs on 2.4
>> and several hundreds of SSID in 5GHz. But even LTE was very close to
>> capacity.
>>
>> Dave,
>> Having air time fairness in open source is a significant achievement. I
>> don't see a failure.
>>
>> Luca
>>
>>
>> On Mon, Aug 27, 2018 at 8:26 AM Jonathan Morton 
>> wrote:
>>
>>> > On 27 Aug, 2018, at 9:00 am, Bob McMahon 
>>> wrote:
>>> >
>>> > Curious to how LBT can be solved at the PHY level and if the potential
>>> solution sets preserve the end to end principle.
>>>
>>> The usual alternatives include TDM, usually coordinated by a master
>>> device (eg. the AP); full-duplex operation via diplexers and/or orthogonal
>>> coding; and simply firing off a packet and retrying with exponential
>>> backoff if an acknowledgement is not heard.
>>>
>>> TDM and diplexing are already used by both DOCSIS and LTE.  They are
>>> proven technology.  However, in DOCSIS the diplexing is greatly simplified
>>> by the use of a copper channel rather than airwaves, and in LTE the
>>> diplexer is fitted only at the tower, not in each client - so the tower can
>>> transmit and receive simultaneously, but an individual client cannot, but
>>> this is still useful because there are many clients per tower.  Effective
>>> diplexers for wireless are expensive.
>>>
>>> Orthogonal coding is already used by GPS and, in a rather esoteric form,
>>> by MIMO-grade wifi.  IMHO it works rather better in GPS than in wifi.  In
>>> GPS, it allows all of the satellites in the constellation to transmit on
>>> the standard frequency simultaneously, while still being individually
>>> distinguishable.  The data rate is very low, however, since each
>>> satellite's signal inherently has a negative SNR (because there's a dozen
>>> others shouting over it) - that's why it takes a full minute for a receiver
>>> to get a fix from cold, because it simply takes that long to download the
>>> ephemeris from the first satellite whose signal is found.
>>>
>>> A future version of wifi could reasonably use TDM, I think, but not
>>> diplexing.  The way this would work is that the AP assigns each station
>>> (including itself) a series of time windows in which to transmit as much as
>>> they like, and broadcasts this schedule along with its beacon.  Also
>>> scheduled would be windows in which the AP listens for new stations,
>>> including possibly other nearby APs with which it may mutually coordinate
>>> time.  A mesh network could thus be constructed entirely out of mutually
>>> coordinating APs if necessary.
>>>
>>> The above paragraph is obviously a giant handwave...
>>>
>>>  - Jonathan Morton
>>>
>>> ___
>>> Bloat mailing list
>>> Bloat@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] [Cerowrt-devel] closing up my make-wifi-fast lab

[Luca Muscariello]

Jonathan,

Not that giant handwaving though.
IEEE 802.11ax makes use of "almost TDM" RTS/CTS and scheduling. The almost
is necessary as it operates in 2.4/5Ghz bands.
Similar to what you describe, and is coming very soon in shipping products.

RTS/CTS is still a LBT to create a window where TDM can be done.
I don't yet see how a non private spectrum can be shared  w/o LBT.

On the other hand, medium sharing is one thing, the other thing is
capacity.
There is no way to efficiently share a medium if this is used close to its
theoretical capacity.

Capacity as #of stations per band including #SSID per band. Today scaling
can be achieved
with careful radio planning for spatial diversity or dynamic bean forming.

When you approach capacity with WiFi you only see beacon traffic and almost
zero throughput.
Cannot forget Mobile World Congress where you can measure several thousands
of SSIDs on 2.4
and several hundreds of SSID in 5GHz. But even LTE was very close to
capacity.

Dave,
Having air time fairness in open source is a significant achievement. I
don't see a failure.

Luca


On Mon, Aug 27, 2018 at 8:26 AM Jonathan Morton 
wrote:

> > On 27 Aug, 2018, at 9:00 am, Bob McMahon 
> wrote:
> >
> > Curious to how LBT can be solved at the PHY level and if the potential
> solution sets preserve the end to end principle.
>
> The usual alternatives include TDM, usually coordinated by a master device
> (eg. the AP); full-duplex operation via diplexers and/or orthogonal coding;
> and simply firing off a packet and retrying with exponential backoff if an
> acknowledgement is not heard.
>
> TDM and diplexing are already used by both DOCSIS and LTE.  They are
> proven technology.  However, in DOCSIS the diplexing is greatly simplified
> by the use of a copper channel rather than airwaves, and in LTE the
> diplexer is fitted only at the tower, not in each client - so the tower can
> transmit and receive simultaneously, but an individual client cannot, but
> this is still useful because there are many clients per tower.  Effective
> diplexers for wireless are expensive.
>
> Orthogonal coding is already used by GPS and, in a rather esoteric form,
> by MIMO-grade wifi.  IMHO it works rather better in GPS than in wifi.  In
> GPS, it allows all of the satellites in the constellation to transmit on
> the standard frequency simultaneously, while still being individually
> distinguishable.  The data rate is very low, however, since each
> satellite's signal inherently has a negative SNR (because there's a dozen
> others shouting over it) - that's why it takes a full minute for a receiver
> to get a fix from cold, because it simply takes that long to download the
> ephemeris from the first satellite whose signal is found.
>
> A future version of wifi could reasonably use TDM, I think, but not
> diplexing.  The way this would work is that the AP assigns each station
> (including itself) a series of time windows in which to transmit as much as
> they like, and broadcasts this schedule along with its beacon.  Also
> scheduled would be windows in which the AP listens for new stations,
> including possibly other nearby APs with which it may mutually coordinate
> time.  A mesh network could thus be constructed entirely out of mutually
> coordinating APs if necessary.
>
> The above paragraph is obviously a giant handwave...
>
>  - Jonathan Morton
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] Van Jacobson's slides on timing wheels at netdevconf

[Luca Muscariello]

and C) you can implement any packet scheduler using a timing wheel using
virtual times.

On Fri, Jul 20, 2018 at 4:09 PM Dave Taht  wrote:

>
> https://www.files.netdevconf.org/d/46def75c2ef345809bbe/files/?p=/Evolving%20from%20AFAP%20%E2%80%93%20Teaching%20NICs%20about%20time.pdf
>
> Talking about replacing queues with timing wheels.
>
> I will refrain from commenting other than noting A) I like it. We
> essentially have limits in the OS on packet scheduling that make it
> harder and harder to have sub 2ms, much less sub 10usec, queues with
> the existing qdiscs and pacing systems B) that NIC support seems
> needed. I can think of a lot of things I'd like to have in a NIC
> (which certainly include default timestamping on rx and multiple kinds
> of X-tuple hash) - but hadn't thought about replacing queues entirely!
>
> I haven't read up on carousel yet either.
>
> Look forward to seeing the video when it comes out.
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> ___
> Make-wifi-fast mailing list
> make-wifi-f...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Is bufferbloat a privacy issue?

[Luca Muscariello]

Good point. Flow isolation gives some kind of “privacy”.

But I guess this is not the worse privacy violation one would be concerned
about in today Internet.

On Thu 19 Jul 2018 at 12:52, Dave Taht  wrote:

> hahaha. Aside from their last slide not recommending fq, that was an
> enjoyable read. fq in this case (but I would deprioritize ping
> responses slightly in the general case) makes the actual observed load
> even more invisible.
>
> I think traceroute would have been a better tool for this study.
>
> and smokeping remains a very useful tool for those that can deploy it.
>
> On Thu, Jul 19, 2018 at 9:44 AM Toke Høiland-Jørgensen 
> wrote:
> >
> > This was presented at today's maprg session at the IETF:
> >
> >
> https://datatracker.ietf.org/meeting/102/materials/slides-102-maprg-is-bufferbloat-a-privacy-issue-brian-trammell-00
> >
> > -Toke
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Seen in passing: mention of Valve's networking scheme and RFC 5348

[Luca Muscariello]

I'm aware of this one. The last time I checked Linux patches seemed to be
abandoned.
Hit ratio could be v v low if you remove UDP encap. Look at IPSEC.


On Wed, Apr 4, 2018 at 12:52 PM, Mikael Abrahamsson <swm...@swm.pp.se>
wrote:

> On Wed, 4 Apr 2018, Luca Muscariello wrote:
>
> And yes, flow queueing, absolutely. Flow isolation, becomes fundamental is
>> such a zoo, or jungle.
>>
>
> There was talk in IETF about a transport protocol that was proposed to do
> a lot of things TCP doesn't do, but still retain some things that has been
> useful with TCP.
>
> I think it was this one:
>
> https://datatracker.ietf.org/doc/draft-ietf-nvo3-gue/
>
> I'd like to see it not over UDP, but rather as a native IP protocol. The
> talk was about having the network being able to look into the state machine
> of the protocol (MSS size, equivalent of SYN, etc) but not into payload
> (which would be end-to-end encrypted). It would also be able to do muxed
> streams/message based to avoid head-of-line-blocking because of single
> packet loss.
>
> So any of this that comes up then the whole FQ machinery might benefit
> frmo being able to identify flows in any new protocol, but I imagine this
> is not a hard thing to do. I still have hopes for the flow label in IPv6 to
> do this job, even though it hasn't seen wide adoption so far.
>
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Seen in passing: mention of Valve's networking scheme and RFC 5348

[Luca Muscariello]

I'm looking at TAPS too as I'm looking for a general transport API other
than TCP/UDP.

The kind of transport services we have developed in here
https://git.fd.io/cicn/ do not fit in the current API.
Full user land implementation, which seems to be accepted nowadays, but not
at all a few years back.
The API however is a hack of the INET API, which is rather obsolete, but
this is what current apps can use.
So, hope TAPS gets traction.

And yes, IPv6, absolutely. Going through IPv4 middle boxes is a nightmare.
No hope there.

And yes, flow queueing, absolutely. Flow isolation, becomes fundamental is
such a zoo, or jungle.











On Wed, Apr 4, 2018 at 10:52 AM, Mikael Abrahamsson 
wrote:

> On Wed, 4 Apr 2018, Dave Taht wrote:
>
> How dead is posix these days? Ietf does not generally do apis well.
>>
>
> POSIX nowadays is
>
> http://pubs.opengroup.org/onlinepubs/9699919799/
>
> My take on it is that the IETF should not be scared to do APIs, even
> though there is a lot of resistance still.
>
> However, the IETF should not do POSIX APIs, but instead something of their
> own.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] occasionally, there's good news to be had for the Internet.

[Luca Muscariello]

Next challenge: FQ into silicon...

One of the reasons of what Jim describes under the "Here lies madness"
section
is related to silicon built-in functionalities and the ability to update
software in edge-routers.

Many chipcos build their board with several  CPU offloading mechanisms,
e.g.
to support 1Gbps in the home.

My home router uses a fourth-generation BCM3384 euro-DOCSIS 3.0
System-on-a-Chip (SoC)
capable of 1Gbps downstream speeds by bonding up to 24x8 DOCSIS channels.
One at 600MHz for the modem and one for the OS at 1GHz.

I get 1Gbps downstream reliably and excessive latency too, ...reliably.

It's a Linux box in the end but no way to enable FQ_codel based on the
current architecture
as TCP pipelines are offloaded to silicon.

the are all sort of acceleration in the SoC, so I don't see it impossible
to get FQ there.



On Mon, Feb 12, 2018 at 9:31 AM, Dave Taht  wrote:

> I wish that more people agreed that fair queuing and AQM were the
> rightest answers to the political debate, of both sides, to
> NetworkNeutrality. RF8290 is fair to flows, no matter whose they are.
> I also would like stuff like what Jim just wrote to somehow escape the
> filter bubble, compressed down into terms found in everyday language.
> Or a checkbox item on an RFP.
>
> https://gettys.wordpress.com/2018/02/11/the-blind-men-and-the-elephant/
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cerowrt-devel] DC behaviors today

[Luca Muscariello]

+1 on all.
Except that Little's Law is very general as it applies to any ergodic
process.
It just derives from the law of large numbers. And BTW, Little's law is a
very powerful law.
We use it unconsciously all the time.



On Tue, Dec 12, 2017 at 11:53 PM, <dpr...@reed.com> wrote:

> Luca's point tends to be correct - variable latency destroys the stability
> of flow control loops, which destroys throughput, even when there is
> sufficient capacity to handle the load.
>
>
>
> This is an indirect result of Little's Lemma (which is strictly true only
> for Poisson arrival, but almost any arrival process will have a similar
> interaction between latency and throughput).
>
>
>
> However, the other reason I say what I say so strongly is this:
>
>
>
> Rant on.
>
>
>
> Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only
> "benchmark setups" (or hot-rod races done for academic reasons or marketing
> reasons to claim some sort of "title") operate at peak supportable load any
> significant part of the time.
>
>
>
> The reason for this is not just "fat pipes are better", but because
> bitrate of the underlying medium is an insignificant fraction of systems
> operational and capital expense.
>
>
>
> SLA's are specified in "uptime" not "bits transported", and a clogged pipe
> is defined as down when latency exceeds a small number.
>
>
>
> Typical operating points of corporate networks where the users are happy
> are single-digit percentage of max load.
>
>
>
> This is also true of computer buses and memory controllers and storage
> interfaces IRL. Again, latency is the primary measure, and the system never
> focuses on operating points anywhere near max throughput.
>
>
>
> Rant off.
>
>
> On Tuesday, December 12, 2017 1:36pm, "Dave Taht" <d...@taht.net> said:
>
> >
> > Luca Muscariello <luca.muscarie...@gmail.com> writes:
> >
> > > I think everything is about response time, even throughput.
> > >
> > > If we compare the time to transmit a single packet from A to B,
> including
> > > propagation delay, transmission delay and queuing delay,
> > > to the time to move a much larger amount of data from A to B we use
> > throughput
> > > in this second case because it is a normalized
> > > quantity w.r.t. response time (bytes over delivery time). For a single
> > > transmission we tend to use latency.
> > > But in the end response time is what matters.
> > >
> > > Also, even instantaneous throughput is well defined only for a time
> scale
> > which
> > > has to be much larger than the min RTT (propagation + transmission
> delays)
> > > Agree also that looking at video, latency and latency budgets are
> better
> > > quantities than throughput. At least more accurate.
> > >
> > > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson <swm...@swm.pp.se>
> > wrote:
> > >
> > > On Mon, 4 Dec 2017, dpr...@reed.com wrote:
> > >
> > > I suggest we stop talking about throughput, which has been the
> > mistaken
> > > idea about networking for 30-40 years.
> > >
> > >
> > > We need to talk both about latency and speed. Yes, speed is talked
> about
> > too
> > > much (relative to RTT), but it's not irrelevant.
> > >
> > > Speed of light in fiber means RTT is approx 1ms per 100km, so from
> > Stockholm
> > > to SFO my RTT is never going to be significantly below 85ms (8625km
> > great
> > > circle). It's current twice that.
> > >
> > > So we just have to accept that some services will never be deliverable
> > > across the wider Internet, but have to be deployed closer to the
> > customer
> > > (as per your examples, some need 1ms RTT to work well), and we need
> > lower
> > > access latency and lower queuing delay. So yes, agreed.
> > >
> > > However, I am not going to concede that speed is "mistaken idea about
> > > networking". No amount of smarter queuing is going to fix the problem
> if
> > I
> > > don't have enough throughput available to me that I need for my
> > application.
> >
> > In terms of the bellcurve here, throughput has increased much more
> > rapidly than than latency has decreased, for most, and in an increasing
> > majority of human-interactive cases (like video streaming), we often
> > have enough throughput.
> >
> > And the age old argument regarding "just have overc

Re: [Bloat] [Cerowrt-devel] DC behaviors today

[Luca Muscariello]

I think everything is about response time, even throughput.

If we compare the time to transmit a single packet from A to B, including
propagation delay, transmission delay and queuing delay,
to the time to move a much larger amount of data from A to B we use
throughput in this second case because it is a normalized
quantity w.r.t. response time (bytes over delivery time). For a single
transmission we tend to use latency.
But in the end response time is what matters.

Also, even instantaneous throughput is well defined only for a time scale
which has to be much larger than the min RTT (propagation +  transmission
delays)
Agree also that looking at video, latency and latency budgets are better
quantities than throughput. At least more accurate.










On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson  wrote:

> On Mon, 4 Dec 2017, dpr...@reed.com wrote:
>
> I suggest we stop talking about throughput, which has been the mistaken
>> idea about networking for 30-40 years.
>>
>
> We need to talk both about latency and speed. Yes, speed is talked about
> too much (relative to RTT), but it's not irrelevant.
>
> Speed of light in fiber means RTT is approx 1ms per 100km, so from
> Stockholm to SFO my RTT is never going to be significantly below 85ms
> (8625km great circle). It's current twice that.
>
> So we just have to accept that some services will never be deliverable
> across the wider Internet, but have to be deployed closer to the customer
> (as per your examples, some need 1ms RTT to work well), and we need lower
> access latency and lower queuing delay. So yes, agreed.
>
> However, I am not going to concede that speed is "mistaken idea about
> networking". No amount of smarter queuing is going to fix the problem if I
> don't have enough throughput available to me that I need for my application.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

https://www.cisco.com/c/en/us/products/collateral/wireless/aironet-3700-series/white-paper-c11-735947.html


On Fri 1 Dec 2017 at 19:43, Dave Taht <d...@taht.net> wrote:

> Luca Muscariello <luca.muscarie...@gmail.com> writes:
>
> > For highly asymmetric links, but also shared media like wifi, QUIC might
> be a
> > better playground for optimisations.
> > Not pervasive as TCP though and maybe off topic in this thread.
>
> I happen to really like QUIC, but a netperf-style tool did not exist for
> it when I last looked, last year.
>
> Also getting to emulating DASH traffic is on my list.
>
> >
> > If the downlink is what one want to optimise, using FEC in the
> downstream, in
> > conjunction with flow control could be very effective.
> > No need to send ACK frequently and having something like FQ_codel in the
> > downstream would avoid fairness problems that might
> > happen though. I don't know if FEC is still in QUIC and used.
> >
> > BTW, for wifi, the ACK stream can be compressed in aggregate of frames
> and sent
> > in bursts. This is similar to DOCSIS upstream.
> > I wonder if this is a phenomenon that is visible in recent WiFi or just
> > negligible.
>
> My guess is meraki deployed something and I think they are in in the top
> 5 in the enterprise market.
>
> I see ubnt added airtime fairness (of some sort), recently.
>
> >
> > On Fri, Dec 1, 2017 at 9:45 AM, Sebastian Moeller <moell...@gmx.de>
> wrote:
> >
> > Hi All,
> >
> > you do realize that the worst case is going to stay at 35KPPS? If we
> assume
> > simply that the 100Mbps download rate is not created by a single
> flow but by
> > many flows (say 70K flows) the discussed ACK frequency reduction
> schemes
> > will not work that well. So ACK thinning is a nice optimization, but
> will
> > not help the fact that some ISPs/link technologies simply are
> asymmetric and
> > the user will suffer under some traffic conditions. Now the 70K flow
> example
> > is too extreme, but the fact is at hight flow number with sparse
> flows (so
> > fewer ACKs per flow in the queue and fewer ACKs per flow reaching
> the end
> > NIC in a GRO-collection interval (I naively assume there is a
> somewhat fixed
> > but small interval in which packets of the same flow are collected
> for GRO))
> > there will be problems. (Again, I am all for allowing the end user to
> > configure ACK filtering thinning, but I would rather see ISPs sell
> less
> > imbalanced links ;) )
> >
> > Best Regards
> > Sebastian
> >
> >
> >
> > > On Dec 1, 2017, at 01:28, David Lang <da...@lang.hm> wrote:
> > >
> > > 35K PPS of acks is insane, one ack every ms is FAR more than
> enough to do
> > 'fast recovery', and outside the datacenter, one ack per 10ms is
> probably
> > more than enough.
> > >
> > > Assuming something that's not too assymetric, thinning out the
> acks may
> > not make any difference in the transfer rate of a single data flow
> in one
> > direction, but if you step back and realize that there may be a need
> to
> > transfer data in the other direction, things change here.
> > >
> > > If you have a fully symmetrical link, and are maxing it out in both
> > direction, going from 35K PPs of aks competing with data packets and
> gonig
> > down to 1k PPS or 100 PPS (or 10 PPS) would result in a noticable
> > improvement in the flow that the acks are competing against.
> > >
> > > Stop thinking in terms of single-flow benchmarks and near idle
> 'upstream'
> > paths.
> > >
> > > David Lang
> > > ___
> > > Bloat mailing list
> > > Bloat@lists.bufferbloat.net
> > > https://lists.bufferbloat.net/listinfo/bloat
> >
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] benefits of ack filtering

[Luca Muscariello]

I think only IPSEC would be a problem for fastACK but not TLS.

On Fri, Dec 1, 2017 at 2:13 PM, Кирилл Луконин  wrote:

> As I noticed from the Meraki document:
>
> "FastACK also relies on packet inspection, and will not work when
> payload is encrypted. However, in our networks, we do not currently
> see an extensive use of encryption techniques like IPSec."
>
> But what about TLS ?
> As for me, this technology will never work in most cases.
>
>
> Best regards,
> Lukonin Kirill.
>
> 2017-12-01 17:53 GMT+05:00 Toke Høiland-Jørgensen :
> > Jan Ceuleers  writes:
> >
> >> On 01/12/17 01:28, David Lang wrote:
> >>> Stop thinking in terms of single-flow benchmarks and near idle
> >>> 'upstream' paths.
> >>
> >> Nobody has said it so I will: on wifi-connected endpoints the upstream
> >> acks also compete for airtime with the downstream flow.
> >
> > There's a related discussion going on over on the make-wifi-fast list
> > related to the FastACK scheme proposed by Meraki at this year's IMC:
> >
> > https://conferences.sigcomm.org/imc/2017/papers/imc17-final203.pdf
> >
> > It basically turns link-layer ACKs into upstream TCP ACKs (and handles
> > some of the corner cases resulting from this) and also seems to contain
> > an ACK compression component.
> >
> > -Toke
> > ___
> > Make-wifi-fast mailing list
> > make-wifi-f...@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
>
>
> --
> Best Regards,
> Lukonin Kirill
> ___
> Make-wifi-fast mailing list
> make-wifi-f...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

If I understand the text right, FastACK runs in the AP and generates an ACK
on behalf (or despite) of the TCP client end.
Then, it decimates dupACKs.

This means that there is a stateful connection tracker in the AP. Not so
simple.
It's almost, not entirely though, a TCP proxy doing Split TCP.


On Fri, Dec 1, 2017 at 1:53 PM, Toke Høiland-Jørgensen  wrote:

> Jan Ceuleers  writes:
>
> > On 01/12/17 01:28, David Lang wrote:
> >> Stop thinking in terms of single-flow benchmarks and near idle
> >> 'upstream' paths.
> >
> > Nobody has said it so I will: on wifi-connected endpoints the upstream
> > acks also compete for airtime with the downstream flow.
>
> There's a related discussion going on over on the make-wifi-fast list
> related to the FastACK scheme proposed by Meraki at this year's IMC:
>
> https://conferences.sigcomm.org/imc/2017/papers/imc17-final203.pdf
>
> It basically turns link-layer ACKs into upstream TCP ACKs (and handles
> some of the corner cases resulting from this) and also seems to contain
> an ACK compression component.
>
> -Toke
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

For highly asymmetric links, but also shared media like wifi, QUIC might be
a better playground for optimisations.
Not pervasive as TCP though and maybe off topic in this thread.

If the downlink is what one want to optimise, using FEC in the downstream,
in conjunction with flow control could be very effective.
No need to send ACK frequently and having something like FQ_codel in the
downstream would avoid fairness problems that might
happen though. I don't know if FEC is still in QUIC and used.


BTW, for wifi, the ACK stream can be compressed in aggregate of frames and
sent in bursts. This is similar to DOCSIS upstream.
I wonder if this is a phenomenon that is visible in recent  WiFi or just
negligible.










On Fri, Dec 1, 2017 at 9:45 AM, Sebastian Moeller  wrote:

> Hi All,
>
> you do realize that the worst case is going to stay at 35KPPS? If we
> assume simply that the 100Mbps download rate is not created by a single
> flow but by many flows (say 70K flows) the discussed ACK frequency
> reduction schemes will not work that well. So ACK thinning is a nice
> optimization, but will not help the fact that some ISPs/link technologies
> simply are asymmetric and the user will suffer under some traffic
> conditions. Now the 70K flow example is too extreme, but the fact is at
> hight flow number with sparse flows (so fewer ACKs per flow in the queue
> and fewer ACKs per flow reaching the end NIC in a GRO-collection interval
> (I naively assume there is a somewhat fixed but small interval in which
> packets of the same flow are collected for GRO)) there will be problems.
> (Again, I am all for allowing the end user to configure ACK filtering
> thinning, but I would rather see ISPs sell less imbalanced links ;) )
>
> Best Regards
> Sebastian
>
> > On Dec 1, 2017, at 01:28, David Lang  wrote:
> >
> > 35K PPS of acks is insane, one ack every ms is FAR more than enough to
> do 'fast recovery', and outside the datacenter, one ack per 10ms is
> probably more than enough.
> >
> > Assuming something that's not too assymetric, thinning out the acks may
> not make any difference in the transfer rate of a single data flow in one
> direction, but if you step back and realize that there may be a need to
> transfer data in the other direction, things change here.
> >
> > If you have a fully symmetrical link, and are maxing it out in both
> direction, going from 35K PPs of aks competing with data packets and gonig
> down to 1k PPS or 100 PPS (or 10 PPS) would result in a noticable
> improvement in the flow that the acks are competing against.
> >
> > Stop thinking in terms of single-flow benchmarks and near idle
> 'upstream' paths.
> >
> > David Lang
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

Agree and think this is a lucid analysis of the problem(s) and solution(s).

But, what can be done to let clients upgrade orders of magnitude faster
than today?
Move transport in user space inside the app? Else?




On Thu, Nov 30, 2017 at 8:48 AM, Jonathan Morton 
wrote:

> I do see your arguments.  Let it be known that I didn't initiate the
> ack-filter in Cake, though it does seem to work quite well.
>
> With respect to BBR, I don't think it depends strongly on the return rate
> of acks in themselves, but rather on the rate of sequence number advance
> that they indicate.  For this purpose, having the receiver emit sparser but
> still regularly spaced acks would be better than having some middlebox
> delete some less-predictable subset of them.  So I think BBR could be a
> good testbed for AckCC implementation, especially as it is inherently paced
> and thus doesn't suffer from burstiness as a conventional ack-clocked TCP
> might.
>
> The real trouble with AckCC is that it requires implementation on the
> client as well as the server.  That's most likely why Google hasn't tried
> it yet; there are no receivers in the wild that would give them valid data
> on its effectiveness.  Adding support in Linux would help here, but aside
> from Android devices, Linux is only a relatively small proportion of
> Google's client traffic - and Android devices are slow to pick up new
> kernel features if they can't immediately turn it into a consumer-friendly
> bullet point.
>
> Meanwhile we have highly asymmetric last-mile links (10:1 is typical, 50:1
> is occasionally seen), where a large fraction of upload bandwidth is
> occupied by acks in order to fully utilise the download bandwidth in TCP.
> Any concurrent upload flows have to compete with that dense ack flow, which
> in various schemes is unfair to either the upload or the download
> throughput.
>
> That is a problem as soon as you have multiple users on the same link, eg.
> a family household at the weekend.  Thinning out those acks in response to
> uplink congestion is a solution.  Maybe not the best possible solution, but
> a deployable one that works.
>
> - Jonathan Morton
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

On Wed, Nov 29, 2017 at 3:31 PM, Mikael Abrahamsson <swm...@swm.pp.se>
wrote:

> On Wed, 29 Nov 2017, Luca Muscariello wrote:
>
> Why does it say to do this? What benefit is there to either end system to
>>> send 35kPPS of ACKs in order to facilitate a 100 megabyte/s of TCP
>>> transfer?
>>>
>>
>> Did you check RFC 3449 ?
>> https://tools.ietf.org/html/rfc3449#section-5.2.1
>>
>
> RFC3449 is all about middleboxes doing things.
>
> I wanted to understand why TCP implementations find it necessary to send
> one ACK per 2xMSS at really high PPS. Especially when NIC offloads and
> middleboxes frequently strip out this information anyway so it never
> reaches the IP stack (right?).
>
>
I would say because it is complex to guess at which PPS to work. You would
need an adaptation mechanism. Need also to change the client and the server
sides. The AckCC Jonathan has mentioned
might be a solution to that.
Probably an ACK pacer in the end host, out of the TCP stack, doing Ack
filtering and decimation can be simpler to implement than the proper
adaptation mechanism in TCP.
Maybe inside sch_fq it would be doable. Maybe not.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Luca Muscariello]

Did you check RFC 3449 ?
https://tools.ietf.org/html/rfc3449#section-5.2.1

It would be interesting to know what is the minimum ACK rate to achieve
full utilisation.
Or the how the downlink rate depends on the uplink ACK rate.
I'm sure I've seen this dependency in some old paper.


On Wed, Nov 29, 2017 at 1:49 PM, Mikael Abrahamsson 
wrote:

> On Wed, 29 Nov 2017, Sebastian Moeller wrote:
>
> Well, ACK filtering/thinning is a simple trade-off: redundancy versus
>> bandwidth. Since the RFCs say a receiver should acknoledge every second
>> full MSS I think the decision whether to filter or not should be kept to
>>
>
> Why does it say to do this? What benefit is there to either end system to
> send 35kPPS of ACKs in order to facilitate a 100 megabyte/s of TCP transfer?
>
> Sounds like a lot of useless interrupts and handling by the stack, apart
> from offloading it to the NIC to do a lot of handling of these mostly
> useless packets so the CPU doesn't have to do it.
>
> Why isn't 1kPPS of ACKs sufficient for most usecases?
>
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] emulating non-duplex media in linux qdiscs

[Luca Muscariello]

Dave,

If the objective is to run experiments with some emulated wifi I could
suggest
a workaround that we are using. It is not based on qdisc only.

It is based on ns3 emulation and linux bridges.

https://git.fd.io/cicn/tree/emu-radio/README.md?h=vicn/master

The radio emulation piece includes 802.11n and LTE.
For wifi you get the contention behaviour you're looking for and listen
before talk etc.
You also get the TX or RX behaviour. There is also the propagation models
from ns3 (fast/slow fading, etc. etc.).

The radio piece is a single Linux process so depending on the scaling
you're looking for this tool
might or might not be useful.

In the repo, there is some documentation, but if you plan to use this stuff
and need help let me know.
There is also a ML, in case it helps.

Luca




On Mon, Oct 9, 2017 at 3:54 AM, Dave Taht  wrote:

> I have been hacking away at netem for a while now in the hope that
> eventually - with a great deal more hacking - it could be used to more
> accurately emulate shared media like wifi and lte.
>
> (Some people try to describe these as simplex (which is not true
> because you can have multiple destinations), and they certainly are
> not duplex, so I tend to say non-duplex and still hope some better
> word emerges)
>
> So... one sticking point for me has been wanting to emulate the fact
> that on shared media, that you cannot transmit and receive at the same
> time; that these are "coupled" events, and what I'd like to be able to
> express might be something like:
>
> tc qdisc add dev eth0 root netem rate 100mbit coupled some_identifier
> ... some tc mirred magic for ifb here ...
> tc qdisc add dev ifb0 root netem rate 10mbit coupled the_same_identifier
>
> "some_identifier" would be a mutex of some sort, and I confess to
> not having much grip on the kernel outside of the net/sched directory.
>
> What facility would be best to try and leverage? It would be created
> (globally) on first use, ref-counted (thus destroyed when it goes to
> zero), atomically updated... posix shared memory seems too heavyweight
> to use
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] the wifi airtime-fair fq_codel stuff on net-next looks mostly good

[Luca Muscariello]

Air time fairness has a strong theoretical foundation.
So I should cite Newton and say that Toke is sitting on giants' schoulders
:)

In multi rate systems  in a shared channel, time is the right resource to
share.

Then one could discuss about which fairness criterion to use, but that's
secondary.

The criterion used by toke is max-min in time.
I guess this is the best you can do in wifi.
This turns out to be proportional fairness in throughput.

In LTE the shared channel is time shared (slotted)
and fairness is slightly different to max-min in time.

In LTE thanks to the feedback channel, multi user diversity can be used to
schedule transmissions towards the UE with best radio conditions at a given
time.
David Tse showed that this is proportional fairness
with multi user diversity gain. The cell throughout increases
logarithmically with the number of users.
And this is the best criterion for many reasons that I skip here.

This is out of target for wifi but what Toke is doing is really solid.

Luca




On Saturday, 15 October 2016, Mikael Abrahamsson  wrote:

> On Wed, 12 Oct 2016, Dave Taht wrote:
>
> http://openwrtsummit.org/#quick-details
>>
>
> I've had the discussion with "radio guys" before regarding "fairness" of
> radio resources. They kept talking about "optimising the cell for
> throughput". I told them "then we should give the speaker with the highest
> bitrate and demand for bits as much radio resources as possible, and starve
> everybody else". This is of course not good for general customer
> satisfaction.
>
> After a lot of discussions back and forth, we came to the same conclusion
> as you seem to have come to (if I understood Tokes talk correctly), in that
> "radio time" is the most fair resource. If someone has bad radio conditions
> then they get lower total throughput than the one with good radio
> conditions, so the fairness is "equal air time". This means everybody get
> equal part of the shared resource, and gives people an incentive to try to
> improve radio reception if they have trouble, and doesn't starve everybody
> else of airtime just because one device is having a bad radio day.
>
> So full support for this approach from me, good job!
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Make-wifi-fast mailing list
> make-wifi-f...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [aqm] TSO sizing fixes and the new paced fq scheduler in Linux 3.12

[Luca MUSCARIELLO]

Le 25/09/2013 17:15, Eric Dumazet a écrit :

On Tue, 2013-09-24 at 14:25 +0200, James Roberts wrote:

No one responded to Luca's Sept 1 comment (on the bloat list) that the
new code seems to do tail drop rather than longest queue drop.


If this is so, bandwidth sharing will not be fair since FQ alone is
not enough. This was shown in the previously cited Bell Labs
paper : http://ect.bell-labs.com/who/stiliadi/papers/jsac99.pdf. The
following table is copied from the paper.

This paper assumes TCP stack can push cwin packets in the queue.
We no longer have this behavior with linux.

If you have drops on FQ, then you have a serious problem with your
settings.

FQ is meant to be used on hosts, not on routers.

For routers, fq_codel is fine.

TCP Small Queues limits the number of packets in Qdisc for any tcp flow
(2 packets). Default FQ setting allows 1 packets.

You can add tail drop on FQ if you really want, but I never had a single
drop in my FQ settings, on 20Gbps links and thousands of flows.

Therefore I did not add complexity to solve a non problem.


Then, I feel like FQ is a bad name to call this newFQ.
It's an implementation of a fair TCP pacer. Which is very useful, but FQ 
is kind of misleading, IMHO.


Luca







___
aqm mailing list
a...@ietf.org
https://www.ietf.org/mailman/listinfo/aqm


--
France Telecom RD - Orange Labs
MUSCARIELLO Luca - OLN/NMP/TRM
38 - 40, rue du General Leclerc
92794 Issy Les Moulineaux Cedex 9 - France
Tel : +33 (0)1 45 29 60 37
http://perso.rd.francetelecom.fr/muscariello

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat