Re: [Cerowrt-devel] [Starlink] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I can't speak for others. I've been successful in early prototyping using
one for simplistic off-diagonal h-matrix testing, i.e. varying the h-matrix
condition numbers. I see this as a small step in the right direction for
conductive, automated, and reproducible testing.

Developing a system that supports multiple RF channels to test against
seems hard. Then add to that causing the energy states per "peer RF
traffic" makes the challenge even more difficult.

Bob



On Wed, Jul 7, 2021 at 6:36 AM Ben Greear  wrote:

> Thanks for the clarification.  There are vendors that make these, but we
> have not tried
> integrating our software control with any of them.  To date, not many of
> our customers
> have been interested in this, so it did not seem worth the effort.
>
> Do you see this as being useful for normal AP vendors/users, or mostly
> just radio manufacturers such
> as BCM?
>
> In case someone has one of these that has a sane API, I'd consider adding
> automation support
> to drive it while running throughput or RvR or whatever other types of
> tests seem interesting.
>
> Thanks,
> Ben
>
> On 7/6/21 3:05 PM, Bob McMahon wrote:
> > Sorry, I should have been more clear. Not a fixed butler matrix but a
> device with solid state, programmable, phase shifters, 0 - 360 degrees.
> It's a way to
> > create multiple phy channels and affect and vary the off diagonal
> elements of a MIMO H-matrix using conducted parts. Then automation software
> can have more
> > robust RF MIMO test scenarios that are reproducible.
> >
> >
> https://web.stanford.edu/~dntse/Chapters_PDF/Fundamentals_Wireless_Communication_chapter7.pdf
> > <
> https://web.stanford.edu/~dntse/Chapters_PDF/Fundamentals_Wireless_Communication_chapter7.pdf
> >
> >
> > Bob
> >
> > On Tue, Jul 6, 2021 at 2:24 PM Ben Greear  > wrote:
> >
> > We tried adding in an external butler matrix in the past, but could
> not notice any useful difference.  Possibly
> > we didn't have the right use case.
> >
> > Typically we are competitive on price for full testing solutions,
> but you can get stand-alone attenuators
> > cheaper from specialized vendors.  Happy to discuss pricing offlist
> if you wish.
> >
> > Thanks,
> > Ben
> >
> > On 7/6/21 1:43 PM, Bob McMahon wrote:
> >  > The four part attenuator part would be more interesting to me if
> it also had a solid state phase shifters.  This allows for testing 2x2 MIMO
> testing per
> >  > affecting the spatial stream eigen vectors/values.
> >  >
> >  > Bob
> >  >
> >  > PS. The price per port isn't competitive. Probably a good idea to
> survey the market competition.
> >  >
> >  > On Tue, Jul 6, 2021 at 6:46 AM Ben Greear <
> gree...@candelatech.com   gree...@candelatech.com
> > >> wrote:
> >  >
> >  > Hello,
> >  >
> >  > I am interested to hear wish lists for network testing
> features.  We make test equipment, supporting lots
> >  > of wifi stations and a distributed architecture, with
> built-in udp, tcp, ipv6, http, ... protocols,
> >  > and open to creating/improving some of our automated tests.
> >  >
> >  > I know Dave has some test scripts already, so I'm not
> necessarily looking to reimplement that,
> >  > but more fishing for other/new ideas.
> >  >
> >  > Thanks,
> >  > Ben
> >  >
> >  > On 7/2/21 4:28 PM, Bob McMahon wrote:
> >  >  > I think we need the language of math here. It seems like
> the network power metric, introduced by Kleinrock and Jaffe in the late
> 70s, is something
> > useful.
> >  >  > Effective end/end queue depths per Little's law also seems
> useful. Both are available in iperf 2 from a test perspective. Repurposing
> test
> > techniques to
> >  > actual
> >  >  > traffic could be useful. Hence the question around what
> exact telemetry is useful to apps making socket write() and read() calls.
> >  >  >
> >  >  > Bob
> >  >  >
> >  >  > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <
> dave.t...@gmail.com   dave.t...@gmail.com
> > >   >  >  >
> >  >  > In terms of trying to find "Quality" I have tried to
> encourage folk to
> >  >  > both read "zen and the art of motorcycle
> maintenance"[0], and Deming's
> >  >  > work on "total quality management".
> >  >  >
> >  >  > My own slice at this network, computer and lifestyle
> "issue" is aiming
> >  >  > for "imperceptible latency" in all things. [1].
> There's a lot of
> >  >  > fallout from that in terms of not just addressing
> queuing delay, 

Re: [Cerowrt-devel] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I think even packets are a network construct. End/end protocols don't write
packets.  They mostly make writes() and reads and have no clue about
packets. Except for, of course, UDP which you know everything about being
the original designer.

Agreed the telemetry is most interesting and a huge void. Curious to more
of your thoughts on it, metrics, etc.

Note: iperf 2 has write to read latencies. It requires clock sync. My
systems sync to the GPS atomic as the commonNote/ reference. I think
end/end queue depths can be calculated per Little's law (shown below per
inP.) https://sourceforge.net/projects/iperf2/

[rjmcmahon@rjm-nas ~]$ iperf -s -i 1

Server listening on TCP port 5001
TCP window size:  128 KByte (default)

[  1] local 192.168.1.94%enp2s0 port 5001 connected with 192.168.1.100 port
59142 (MSS=1448) (trip-times) (sock=4) (peer 2.1.3-rc) on 2021-07-01
20:57:37 (PDT)
[ ID] IntervalTransferBandwidthBurst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  1] 0.00-1.00 sec   596 MBytes  5.00 Gbits/sec  0.170/0.153/1.492/0.078
ms (4769/131082)  104 KByte 3674521  22841=787:18657:2467:623:84:41:66:116
[  1] 1.00-2.00 sec   596 MBytes  5.00 Gbits/sec  0.167/0.156/0.434/0.015
ms (4768/131086)  102 KByte 3742630  23346=1307:18975:2171:578:105:53:56:101
[  1] 2.00-3.00 sec   596 MBytes  5.00 Gbits/sec  0.168/0.157/1.337/0.033
ms (4769/131046)  103 KByte 3710006  23263=1470:18602:2148:725:107:53:60:98
[  1] 3.00-4.00 sec   596 MBytes  5.00 Gbits/sec  0.166/0.158/0.241/0.008
ms (4768/131082)  102 KByte 3756478  23960=1452:19714:2123:449:79:32:38:73
[  1] 4.00-5.00 sec   596 MBytes  5.00 Gbits/sec  0.166/0.157/0.247/0.008
ms (4769/131061)  102 KByte 3756193  23653=1234:19529:2206:439:89:36:44:76
[  1] 5.00-6.00 sec   596 MBytes  5.00 Gbits/sec  0.166/0.158/0.245/0.007
ms (4768/131072)  101 KByte 3758826  23478=1081:19356:2284:535:73:35:39:75
[  1] 6.00-7.00 sec   596 MBytes  5.00 Gbits/sec  0.168/0.158/0.283/0.009
ms (4768/131096)  102 KByte 3728988  23477=1338:19301:1995:535:104:46:59:99
[  1] 7.00-8.00 sec   596 MBytes  5.00 Gbits/sec  0.163/0.150/0.400/0.010
ms (4769/131047) 99.7 KByte 3826119  23496=1213:19404:2101:498:83:57:43:97
[  1] 8.00-9.00 sec   596 MBytes  5.00 Gbits/sec  0.158/0.149/0.236/0.008
ms (4768/131082) 96.6 KByte 3951089  23652=1328:19498:2074:493:77:41:53:88
[  1] 9.00-10.00 sec   596 MBytes  5.00 Gbits/sec  0.158/0.149/0.235/0.008
ms (4769/131061) 96.4 KByte 3958720  23725=1509:19410:2051:463:91:46:47:108
[  1] 0.00-10.00 sec  5.82 GBytes  5.00 Gbits/sec  0.165/0.149/1.492/0.028
ms (47685/131072)  101 KByte 3784172
 234891=12719:192446:21620:5338:892:440:505:931

[rjmcmahon@ryzen3950 iperf2-code]$ iperf -c 192.168.1.94 -i 1 --trip-times
-b 5g -e

Client connecting to 192.168.1.94, TCP port 5001 with pid 68866 (1 flows)
Write buffer size: 131072 Byte
TCP window size: 85.0 KByte (default)

[  1] local 192.168.1.100%enp4s0 port 59142 connected with 192.168.1.94
port 5001 (MSS=1448) (trip-times) (sock=3) (ct=0.33 ms) on 2021-07-01
20:57:37 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
Cwnd/RTTNetPwr
[  1] 0.00-1.00 sec   596 MBytes  5.00 Gbits/sec  4770/0  5
 295K/111 us  5631373
[  1] 1.00-2.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 295K/120 us  5207927
[  1] 2.00-3.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 306K/110 us  5681375
[  1] 3.00-4.00 sec   596 MBytes  5.00 Gbits/sec  4769/0  0
 306K/107 us  5841891
[  1] 4.00-5.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 306K/110 us  5681375
[  1] 5.00-6.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 306K/109 us  5733498
[  1] 6.00-7.00 sec   596 MBytes  5.00 Gbits/sec  4769/0  0
 306K/115 us  5435499
[  1] 7.00-8.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 306K/111 us  5630192
[  1] 8.00-9.00 sec   596 MBytes  5.00 Gbits/sec  4769/0  0
 306K/110 us  5682567
[  1] 9.00-10.00 sec   596 MBytes  5.00 Gbits/sec  4768/0  0
 306K/109 us  5733498

[rjmcmahon@rjm-nas ~]$ iperf -s -i 1 --histograms=10u

Server listening on TCP port 5001 with pid 5166
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
Enabled rx-histograms bin-width=0.010 ms, bins=1000 (clients must use
--trip-times)
TCP window size:  128 KByte (default)

[  1] local 192.168.1.94%enp2s0 port 5001 connected with 192.168.1.100 port
59146 (MSS=1448) (trip-times) (sock=4) (peer 2.1.3-rc) on 2021-07-01
21:01:42 (PDT)
[ ID] IntervalTransferBandwidthBurst Latency
avg/min/max/stdev (cnt/size) inP NetPwr  Reads=Dist
[  1] 0.00-1.00 sec   596 

Re: [Cerowrt-devel] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I think we need the language of math here. It seems like the network power
metric, introduced by Kleinrock and Jaffe in the late 70s, is something
useful. Effective end/end queue depths per Little's law also seems useful.
Both are available in iperf 2 from a test perspective. Repurposing test
techniques to actual traffic could be useful. Hence the question around
what exact telemetry is useful to apps making socket write() and read()
calls.

Bob

On Fri, Jul 2, 2021 at 10:07 AM Dave Taht  wrote:

> In terms of trying to find "Quality" I have tried to encourage folk to
> both read "zen and the art of motorcycle maintenance"[0], and Deming's
> work on "total quality management".
>
> My own slice at this network, computer and lifestyle "issue" is aiming
> for "imperceptible latency" in all things. [1]. There's a lot of
> fallout from that in terms of not just addressing queuing delay, but
> caching, prefetching, and learning more about what a user really needs
> (as opposed to wants) to know via intelligent agents.
>
> [0] If you want to get depressed, read Pirsig's successor to "zen...",
> lila, which is in part about what happens when an engineer hits an
> insoluble problem.
> [1] https://www.internetsociety.org/events/latency2013/
>
>
>
> On Thu, Jul 1, 2021 at 6:16 PM David P. Reed  wrote:
> >
> > Well, nice that the folks doing the conference  are willing to consider
> that quality of user experience has little to do with signalling rate at
> the physical layer or throughput of FTP transfers.
> >
> >
> >
> > But honestly, the fact that they call the problem "network quality"
> suggests that they REALLY, REALLY don't understand the Internet isn't the
> hardware or the routers or even the routing algorithms *to its users*.
> >
> >
> >
> > By ignoring the diversity of applications now and in the future, and the
> fact that we DON'T KNOW what will be coming up, this conference will likely
> fall into the usual trap that net-heads fall into - optimizing for some
> imaginary reality that doesn't exist, and in fact will probably never be
> what users actually will do given the chance.
> >
> >
> >
> > I saw this issue in 1976 in the group developing the original Internet
> protocols - a desire to put *into the network* special tricks to optimize
> ASR33 logins to remote computers from terminal concentrators (aka remote
> login), bulk file transfers between file systems on different time-sharing
> systems, and "sessions" (virtual circuits) that required logins. And then
> trying to exploit underlying "multicast" by building it into the IP layer,
> because someone thought that TV broadcast would be the dominant application.
> >
> >
> >
> > Frankly, to think of "quality" as something that can be "provided" by
> "the network" misses the entire point of "end-to-end argument in system
> design". Quality is not a property defined or created by The Network. If
> you want to talk about Quality, you need to talk about users - all the
> users at all times, now and into the future, and that's something you can't
> do if you don't bother to include current and future users talking about
> what they might expect to experience that they don't experience.
> >
> >
> >
> > There was much fighting back in 1976 that basically involved "network
> experts" saying that the network was the place to "solve" such issues as
> quality, so applications could avoid having to solve such issues.
> >
> >
> >
> > What some of us managed to do was to argue that you can't "solve" such
> issues. All you can do is provide a framework that enables different uses
> to *cooperate* in some way.
> >
> >
> >
> > Which is why the Internet drops packets rather than queueing them, and
> why diffserv cannot work.
> >
> > (I know the latter is conftroversial, but at the moment, ALL of diffserv
> attempts to talk about end-to-end applicaiton specific metrics, but never,
> ever explains what the diffserv control points actually do w.r.t. what the
> IP layer can actually control. So it is meaningless - another violation of
> the so-called end-to-end principle).
> >
> >
> >
> > Networks are about getting packets from here to there, multiplexing the
> underlying resources. That's it. Quality is a whole different thing.
> Quality can be improved by end-to-end approaches, if the underlying network
> provides some kind of thing that actually creates a way for end-to-end
> applications to affect queueing and routing decisions, and more importantly
> getting "telemetry" from the network regarding what is actually going on
> with the other end-to-end users sharing the infrastructure.
> >
> >
> >
> > This conference won't talk about it this way. So don't waste your time.
> >
> >
> >
> >
> >
> >
> >
> > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" 
> said:
> >
> > > The program committee members are *amazing*. Perhaps, finally, we can
> > > move the bar for the internet's quality metrics past endless, blind
> > > repetitions of speedtest.
> > >
> 

Re: [Cerowrt-devel] [Starlink] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks very much for this response. I need to dig in a bit more for sure.

iperf 2 will give every UDP packet's OWD (if the clocks are sync'd) and
will also provide TCP write to read latencies, both supported in histogram
forms. So that's raw samples so to speak. I'm hooking up some units across
geography including across the Pacific (sync'd to GPS atomic time) to see
how "fractal" these distributions look, at least anecdotally.

On top of all the "spherical cow queueing theory" (which made me laugh,)
we've got bluetooth sometimes sharing the radio. So the transport latency
of TCP writes can be all over the map so-to-speak. And bluetooth traffic is
also highly correlated.

Bob




On Thu, Jul 8, 2021 at 12:38 PM David P. Reed  wrote:

> I will tell you flat out that the arrival time distribution assumption
> made by Little's Lemma that allows "estimation of queue depth" is totally
> unreasonable on ANY Internet in practice.
>
>
>
> The assumption is a Poisson Arrival Process. In reality, traffic arrivals
> in real internet applications are extremely far from Poisson, and, of
> course, using TCP windowing, become highly intercorrelated with crossing
> traffic that shares the same queue.
>
>
>
> So, as I've tried to tell many, many net-heads (people who ignore
> applications layer behavior, like the people that think latency doesn't
> matter to end users, only throughput), end-to-end packet arrival times on a
> practical network are incredibly far from Poisson - and they are more like
> fractal probability distributions, very irregular at all scales of time.
>
>
>
> So, the idea that iperf can estimate queue depth by Little's Lemma by just
> measuring saturation of capacity of a path is bogus.The less Poisson, the
> worse the estimate gets, by a huge factor.
>
>
>
>
>
> Where does the Poisson assumption come from?  Well, like many theorems, it
> is the simplest tractable closed form solution - it creates a simplified
> view, by being a "single-parameter" distribution (the parameter is called
> lambda for a Poisson distribution).  And the analysis of a simple queue
> with poisson arrival distribution and a static, fixed service time is the
> first interesting Queueing Theory example in most textbooks. It is
> suggestive of an interesting phenomenon, but it does NOT characterize any
> real system.
>
>
>
> It's the queueing theory equivalent of "First, we assume a spherical
> cow...". in doing an example in a freshman physics class.
>
>
>
> Unfortunately, most networking engineers understand neither queuing theory
> nor application networking usage in interactive applications. Which makes
> them arrogant. They assume all distributions are poisson!
>
>
>
>
>
> On Tuesday, July 6, 2021 9:46am, "Ben Greear" 
> said:
>
> > Hello,
> >
> > I am interested to hear wish lists for network testing features. We make
> test
> > equipment, supporting lots
> > of wifi stations and a distributed architecture, with built-in udp, tcp,
> ipv6,
> > http, ... protocols,
> > and open to creating/improving some of our automated tests.
> >
> > I know Dave has some test scripts already, so I'm not necessarily
> looking to
> > reimplement that,
> > but more fishing for other/new ideas.
> >
> > Thanks,
> > Ben
> >
> > On 7/2/21 4:28 PM, Bob McMahon wrote:
> > > I think we need the language of math here. It seems like the network
> > power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something
> > useful.
> > > Effective end/end queue depths per Little's law also seems useful.
> Both are
> > available in iperf 2 from a test perspective. Repurposing test
> techniques to
> > actual
> > > traffic could be useful. Hence the question around what exact telemetry
> > is useful to apps making socket write() and read() calls.
> > >
> > > Bob
> > >
> > > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht  > > wrote:
> > >
> > > In terms of trying to find "Quality" I have tried to encourage folk to
> > > both read "zen and the art of motorcycle maintenance"[0], and Deming's
> > > work on "total quality management".
> > >
> > > My own slice at this network, computer and lifestyle "issue" is aiming
> > > for "imperceptible latency" in all things. [1]. There's a lot of
> > > fallout from that in terms of not just addressing queuing delay, but
> > > caching, prefetching, and learning more about what a user really needs
> > > (as opposed to wants) to know via intelligent agents.
> > >
> > > [0] If you want to get depressed, read Pirsig's successor to "zen...",
> > > lila, which is in part about what happens when an engineer hits an
> > > insoluble problem.
> > > [1] https://www.internetsociety.org/events/latency2013/
> > 
> > >
> > >
> > >
> > > On Thu, Jul 1, 2021 at 6:16 PM David P. Reed  > > wrote:
> > > >
> > > > Well, nice that the folks doing the conference  are willing to
> > consider that quality of user 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
connect times, latency of packets (with UDP), latency of "frames" with
simulated video traffic (TCP and UDP), xfer times of bursts with low duty
cycle traffic, and TCP RTT (sampling based.) It also has support for
sampling (per interval reports) down to 100 usecs if configured with
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
released all this as open source.

OWD only works if the end realtime clocks are synchronized using a "machine
level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
provide sufficient level of clock accuracy and the GPS pulse per second *
to colo and vm customers.

https://iperf2.sourceforge.io/iperf-manpage.html

Bob

On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  wrote:

>
> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com> said:
>
> > I think latency/delay is becoming seen to be as important certainly, if
> not a more direct proxy for end user QoE. This is all still evolving and I
> have to say is a super interesting & fun thing to work on. :-)
>
> If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
>
> "It's the end-to-end latency, stupid!"
>
> And I mean, by end-to-end, latency to complete a task at a relevant layer
> of abstraction.
>
> At the link level, it's packet send to packet receive completion.
>
> But at the transport level including retransmission buffers, it's datagram
> (or message) origination until the acknowledgement arrives for that message
> being delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
>
> At the WWW level, it's mouse click to display update corresponding to
> completion of the request.
>
> What should be noted is that lower level latencies don't directly predict
> the magnitude of higher-level latencies. But longer lower level latencies
> almost always amplfify higher level latencies. Often non-linearly.
>
> Throughput is very, very weakly related to these latencies, in contrast.
>
> The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
> exactly the right place (the so-called input queue of the bottleneck
> process, which is often a link, but not always).
>
> Can we get that slogan into Harvard Business Review? Can we get it taught
> in Managerial Accounting at HBS? (which does address logistics/supply chain
> queueing).
>
>
>
>
>
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
To be clear, it's a OS write() using a socket opened with TCP and the final
OS read() of that write. The write size is set using -l or --length. OWD
requires --trip-times option.

Bob

On Mon, Jul 12, 2021 at 11:21 AM Bob McMahon 
wrote:

> iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
> connect times, latency of packets (with UDP), latency of "frames" with
> simulated video traffic (TCP and UDP), xfer times of bursts with low duty
> cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
>
> OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide sufficient level of clock accuracy and the GPS pulse
> per second * to colo and vm customers.
>
> https://iperf2.sourceforge.io/iperf-manpage.html
>
> Bob
>
> On Mon, Jul 12, 2021 at 10:40 AM David P. Reed 
> wrote:
>
>>
>> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
>> jason_living...@comcast.com> said:
>>
>> > I think latency/delay is becoming seen to be as important certainly, if
>> not a more direct proxy for end user QoE. This is all still evolving and I
>> have to say is a super interesting & fun thing to work on. :-)
>>
>> If I could manage to sell one idea to the management hierarchy of
>> communications industry CEOs (operators, vendors, ...) it is this one:
>>
>> "It's the end-to-end latency, stupid!"
>>
>> And I mean, by end-to-end, latency to complete a task at a relevant layer
>> of abstraction.
>>
>> At the link level, it's packet send to packet receive completion.
>>
>> But at the transport level including retransmission buffers, it's
>> datagram (or message) origination until the acknowledgement arrives for
>> that message being delivered after whatever number of retransmissions,
>> freeing the retransmission buffer.
>>
>> At the WWW level, it's mouse click to display update corresponding to
>> completion of the request.
>>
>> What should be noted is that lower level latencies don't directly predict
>> the magnitude of higher-level latencies. But longer lower level latencies
>> almost always amplfify higher level latencies. Often non-linearly.
>>
>> Throughput is very, very weakly related to these latencies, in contrast.
>>
>> The amplification process has to do with the presence of queueing.
>> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
>> exactly the right place (the so-called input queue of the bottleneck
>> process, which is often a link, but not always).
>>
>> Can we get that slogan into Harvard Business Review? Can we get it taught
>> in Managerial Accounting at HBS? (which does address logistics/supply chain
>> queueing).
>>
>>
>>
>>
>>
>>
>>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Agreed that UDP is important but it's also much easier to test and debug
for WiFi coders. We find it's the connect() and TCP control loop that
challenges WiFi logic systems on end hosts. APs are a whole different story
per things like OFDMA.

Nothing is simple anymore it seems. Reminds me of the standard model
developed over time at CERN. It ain't E=MC**2

Bob

On Mon, Jul 12, 2021 at 1:36 PM David Lang  wrote:

> I have seen some performance tests that do explicit DNS timing tests
> separate
> from other throughput/latency tests.
>
> Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP
> performance (and especially probability of loss at congested links) is
> very
> important.
>
> David Lang
>
> On Mon, 12 Jul 2021, Ben Greear wrote:
>
> > UDP is better for getting actual packet latency, for sure.  TCP is
> > typical-user-experience-latency though,
> > so it is also useful.
> >
> > I'm interested in the test and visualization side of this.  If there
> were a
> > way to give engineers
> > a good real-time look at a complex real-world network, then they have
> > something to go on while trying
> > to tune various knobs in their network to improve it.
> >
> > I'll let others try to figure out how build and tune the knobs, but the
> data
> > acquisition and
> > visualization is something we might try to accomplish.  I have a feeling
> I'm
> > not the
> > first person to think of this, howeverprobably someone already has
> done
> > such
> > a thing.
> >
> > Thanks,
> > Ben
> >
> > On 7/12/21 1:04 PM, Bob McMahon wrote:
> >> I believe end host's TCP stats are insufficient as seen per the
> "failed"
> > congested control mechanisms over the last decades. I think Jaffe
> pointed
> > this out in
> >> 1979 though he was using what's been deemed on this thread as
> "spherical
> > cow queueing theory."
> >>
> >> "Flow control in store-and-forward computer networks is appropriate for
> > decentralized execution. A formal description of a class of
> "decentralized
> > flow control
> >> algorithms" is given. The feasibility of maximizing power with such
> > algorithms is investigated. On the assumption that communication links
> behave
> > like M/M/1
> >> servers it is shown that no "decentralized flow control algorithm" can
> > maximize network power. Power has been suggested in the literature as a
> > network
> >> performance objective. It is also shown that no objective based only on
> the
> > users' throughputs and average delay is decentralizable. Finally, a
> > restricted class
> >> of algorithms cannot even approximate power."
> >>
> >> https://ieeexplore.ieee.org/document/1095152
> >>
> >> Did Jaffe make a mistake?
> >>
> >> Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem for
> OWD
> > feedback loops
> >> aren't effective. How does one design a control loop around things that
> are
> > non-parametric? It also begs the question, what are the feed forward
> knobs
> > that can
> >> actually help?
> >>
> >> Bob
> >>
> >> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > > wrote:
> >>
> >> Measuring one or a few links provides a bit of data, but seems like
> if
> > someone is trying to understand
> >> a large and real network, then the OWD between point A and B needs
> to
> > just be input into something much
> >> more grand.  Assuming real-time OWD data exists between 100 to 1000
> > endpoint pairs, has anyone found a way
> >> to visualize this in a useful manner?
> >>
> >> Also, considering something better than ntp may not really scale to
> > 1000+ endpoints, maybe round-trip
> >> time is only viable way to get this type of data.  In that case,
> maybe
> > clever logic could use things
> >> like trace-route to get some idea of how long it takes to get
> 'onto'
> > the internet proper, and so estimate
> >> the last-mile latency.  My assumption is that the last-mile latency
> is
> > where most of the pervasive
> >> assymetric network latencies would exist (or just ping 8.8.8.8
> which is
> > 20ms from everywhere due to
> >> $magic).
> >>
> >> Endpoints could also triangulate a bit if needed, using some anchor
> > points in the network
> >> under test.
> >>
> >> Thanks,
> >> Ben
> >>
> >> On 7/12/21 11:21 AM, Bob McMahon wrote:
> >>  > iperf 2 supports OWD and gives full histograms for TCP write to
> > read, TCP connect times, latency of packets (with UDP), latency of
> "frames"
> > with
> >>  > simulated video traffic (TCP and UDP), xfer times of bursts with
> low
> > duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for
> > sampling (per
> >>  > interval reports) down to 100 usecs if configured with
> > --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released
> > all this as open source.
> >>  >
> >>  > OWD only works if the end 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
We in WiFi find UDP, while useful, also has severe limitations. The impact
to the TCP control loop matters a lot for things like aggregation.

Visualizations can be useful but also a bit limiting. We use stats
techniques such as PCA which is more mathematical and less visual.

We find syscall connect() times as a bit more relevant to user experience
than ICMP pings which are typically originated and terminated in kernel
space.

Bob

On Mon, Jul 12, 2021 at 1:32 PM Ben Greear  wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers
> a good real-time look at a complex real-world network, then they have
> something to go on while trying
> to tune various knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and
> visualization is something we might try to accomplish.  I have a feeling
> I'm not the
> first person to think of this, howeverprobably someone already has
> done such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the "failed"
> congested control mechanisms over the last decades. I think Jaffe pointed
> this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate for
> decentralized execution. A formal description of a class of "decentralized
> flow control
> > algorithms" is given. The feasibility of maximizing power with such
> algorithms is investigated. On the assumption that communication links
> behave like M/M/1
> > servers it is shown that no "decentralized flow control algorithm" can
> maximize network power. Power has been suggested in the literature as a
> network
> > performance objective. It is also shown that no objective based only on
> the users' throughputs and average delay is decentralizable. Finally, a
> restricted class
> > of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> distributions and computing gaussians per the central limit theorem for OWD
> feedback loops
> > aren't effective. How does one design a control loop around things that
> are non-parametric? It also begs the question, what are the feed forward
> knobs that can
> > actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful manner?
> >
> > Also, considering something better than ntp may not really scale to
> 1000+ endpoints, maybe round-trip
> > time is only viable way to get this type of data.  In that case,
> maybe clever logic could use things
> > like trace-route to get some idea of how long it takes to get 'onto'
> the internet proper, and so estimate
> > the last-mile latency.  My assumption is that the last-mile latency
> is where most of the pervasive
> > assymetric network latencies would exist (or just ping 8.8.8.8 which
> is 20ms from everywhere due to
> > $magic).
> >
> > Endpoints could also triangulate a bit if needed, using some anchor
> points in the network
> > under test.
> >
> > Thanks,
> > Ben
> >
> > On 7/12/21 11:21 AM, Bob McMahon wrote:
> >  > iperf 2 supports OWD and gives full histograms for TCP write to
> read, TCP connect times, latency of packets (with UDP), latency of "frames"
> with
> >  > simulated video traffic (TCP and UDP), xfer times of bursts with
> low duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for sampling (per
> >  > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >  >
> >  > OWD only works if the end realtime clocks are synchronized using
> a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't
> > provide
> >  > sufficient level of clock accuracy and the GPS pulse per second *
> to colo and vm customers.
> >  >
> >  > https://iperf2.sourceforge.io/iperf-manpage.html
> >  >
> >  > Bob
> >  >
> >  > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <
> dpr...@deepplum.com   

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I believe end host's TCP stats are insufficient as seen per the "failed"
congested control mechanisms over the last decades. I think Jaffe pointed
this out in 1979 though he was using what's been deemed on this thread as
"spherical cow queueing theory."

"Flow control in store-and-forward computer networks is appropriate for
decentralized execution. A formal description of a class of "decentralized
flow control algorithms" is given. The feasibility of maximizing power with
such algorithms is investigated. On the assumption that communication links
behave like M/M/1 servers it is shown that no "decentralized flow control
algorithm" can maximize network power. Power has been suggested in the
literature as a network performance objective. It is also shown that no
objective based only on the users' throughputs and average delay is
decentralizable. Finally, a restricted class of algorithms cannot even
approximate power."

https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's
distributions and computing gaussians per the central limit theorem for OWD
feedback loops aren't effective. How does one design a control loop around
things that are non-parametric? It also begs the question, what are the
feed forward knobs that can actually help?

Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  wrote:

> Measuring one or a few links provides a bit of data, but seems like if
> someone is trying to understand
> a large and real network, then the OWD between point A and B needs to just
> be input into something much
> more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> to visualize this in a useful manner?
>
> Also, considering something better than ntp may not really scale to 1000+
> endpoints, maybe round-trip
> time is only viable way to get this type of data.  In that case, maybe
> clever logic could use things
> like trace-route to get some idea of how long it takes to get 'onto' the
> internet proper, and so estimate
> the last-mile latency.  My assumption is that the last-mile latency is
> where most of the pervasive
> assymetric network latencies would exist (or just ping 8.8.8.8 which is
> 20ms from everywhere due to
> $magic).
>
> Endpoints could also triangulate a bit if needed, using some anchor points
> in the network
> under test.
>
> Thanks,
> Ben
>
> On 7/12/21 11:21 AM, Bob McMahon wrote:
> > iperf 2 supports OWD and gives full histograms for TCP write to read,
> TCP connect times, latency of packets (with UDP), latency of "frames" with
> > simulated video traffic (TCP and UDP), xfer times of bursts with low
> duty cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per
> > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >
> > OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide
> > sufficient level of clock accuracy and the GPS pulse per second * to
> colo and vm customers.
> >
> > https://iperf2.sourceforge.io/iperf-manpage.html
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  > wrote:
> >
> >
> > On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com > said:
> >
> >  > I think latency/delay is becoming seen to be as important
> certainly, if not a more direct proxy for end user QoE. This is all still
> evolving and I have
> > to say is a super interesting & fun thing to work on. :-)
> >
> > If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
> >
> > "It's the end-to-end latency, stupid!"
> >
> > And I mean, by end-to-end, latency to complete a task at a relevant
> layer of abstraction.
> >
> > At the link level, it's packet send to packet receive completion.
> >
> > But at the transport level including retransmission buffers, it's
> datagram (or message) origination until the acknowledgement arrives for
> that message being
> > delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
> >
> > At the WWW level, it's mouse click to display update corresponding
> to completion of the request.
> >
> > What should be noted is that lower level latencies don't directly
> predict the magnitude of higher-level latencies. But longer lower level
> latencies almost
> > always amplfify higher level latencies. Often non-linearly.
> >
> > Throughput is very, very weakly related to these latencies, in
> contrast.
> >
> > The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and 

Re: [Cerowrt-devel] [Starlink] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
The four part attenuator part would be more interesting to me if it also
had a solid state phase shifters.  This allows for testing 2x2 MIMO testing
per affecting the spatial stream eigen vectors/values.

Bob

PS. The price per port isn't competitive. Probably a good idea to survey
the market competition.

On Tue, Jul 6, 2021 at 6:46 AM Ben Greear  wrote:

> Hello,
>
> I am interested to hear wish lists for network testing features.  We make
> test equipment, supporting lots
> of wifi stations and a distributed architecture, with built-in udp, tcp,
> ipv6, http, ... protocols,
> and open to creating/improving some of our automated tests.
>
> I know Dave has some test scripts already, so I'm not necessarily looking
> to reimplement that,
> but more fishing for other/new ideas.
>
> Thanks,
> Ben
>
> On 7/2/21 4:28 PM, Bob McMahon wrote:
> > I think we need the language of math here. It seems like the network
> power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something useful.
> > Effective end/end queue depths per Little's law also seems useful. Both
> are available in iperf 2 from a test perspective. Repurposing test
> techniques to actual
> > traffic could be useful. Hence the question around what exact telemetry
> is useful to apps making socket write() and read() calls.
> >
> > Bob
> >
> > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht  dave.t...@gmail.com>> wrote:
> >
> > In terms of trying to find "Quality" I have tried to encourage folk
> to
> > both read "zen and the art of motorcycle maintenance"[0], and
> Deming's
> > work on "total quality management".
> >
> > My own slice at this network, computer and lifestyle "issue" is
> aiming
> > for "imperceptible latency" in all things. [1]. There's a lot of
> > fallout from that in terms of not just addressing queuing delay, but
> > caching, prefetching, and learning more about what a user really
> needs
> > (as opposed to wants) to know via intelligent agents.
> >
> > [0] If you want to get depressed, read Pirsig's successor to
> "zen...",
> > lila, which is in part about what happens when an engineer hits an
> > insoluble problem.
> > [1] https://www.internetsociety.org/events/latency2013/ <
> https://www.internetsociety.org/events/latency2013/>
> >
> >
> >
> > On Thu, Jul 1, 2021 at 6:16 PM David P. Reed  > wrote:
> >  >
> >  > Well, nice that the folks doing the conference  are willing to
> consider that quality of user experience has little to do with signalling
> rate at the
> > physical layer or throughput of FTP transfers.
> >  >
> >  >
> >  >
> >  > But honestly, the fact that they call the problem "network
> quality" suggests that they REALLY, REALLY don't understand the Internet
> isn't the hardware or
> > the routers or even the routing algorithms *to its users*.
> >  >
> >  >
> >  >
> >  > By ignoring the diversity of applications now and in the future,
> and the fact that we DON'T KNOW what will be coming up, this conference
> will likely fall
> > into the usual trap that net-heads fall into - optimizing for some
> imaginary reality that doesn't exist, and in fact will probably never be
> what users
> > actually will do given the chance.
> >  >
> >  >
> >  >
> >  > I saw this issue in 1976 in the group developing the original
> Internet protocols - a desire to put *into the network* special tricks to
> optimize ASR33
> > logins to remote computers from terminal concentrators (aka remote
> login), bulk file transfers between file systems on different time-sharing
> systems, and
> > "sessions" (virtual circuits) that required logins. And then trying
> to exploit underlying "multicast" by building it into the IP layer, because
> someone
> > thought that TV broadcast would be the dominant application.
> >  >
> >  >
> >  >
> >  > Frankly, to think of "quality" as something that can be
> "provided" by "the network" misses the entire point of "end-to-end argument
> in system design".
> > Quality is not a property defined or created by The Network. If you
> want to talk about Quality, you need to talk about users - all the users at
> all times,
> > now and into the future, and that's something you can't do if you
> don't bother to include current and future users talking about what they
> might expect to
> > experience that they don't experience.
> >  >
> >  >
> >  >
> >  > There was much fighting back in 1976 that basically involved
> "network experts" saying that the network was the place to "solve" such
> issues as quality,
> > so applications could avoid having to solve such issues.
> >  >
> >  >
> >  >
> >  > What some of us managed to do was to argue that you can't "solve"
> such issues. All you can do is provide a framework that enables different
> uses to
> > *cooperate* in 

Re: [Cerowrt-devel] [Starlink] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Sorry, I should have been more clear. Not a fixed butler matrix but a
device with solid state, programmable, phase shifters, 0 - 360 degrees.
It's a way to create multiple phy channels and affect and vary the off
diagonal elements of a MIMO H-matrix using conducted parts. Then
automation software can have more robust RF MIMO test scenarios that are
reproducible.

https://web.stanford.edu/~dntse/Chapters_PDF/Fundamentals_Wireless_Communication_chapter7.pdf

Bob

On Tue, Jul 6, 2021 at 2:24 PM Ben Greear  wrote:

> We tried adding in an external butler matrix in the past, but could not
> notice any useful difference.  Possibly
> we didn't have the right use case.
>
> Typically we are competitive on price for full testing solutions, but you
> can get stand-alone attenuators
> cheaper from specialized vendors.  Happy to discuss pricing offlist if you
> wish.
>
> Thanks,
> Ben
>
> On 7/6/21 1:43 PM, Bob McMahon wrote:
> > The four part attenuator part would be more interesting to me if it also
> had a solid state phase shifters.  This allows for testing 2x2 MIMO testing
> per
> > affecting the spatial stream eigen vectors/values.
> >
> > Bob
> >
> > PS. The price per port isn't competitive. Probably a good idea to survey
> the market competition.
> >
> > On Tue, Jul 6, 2021 at 6:46 AM Ben Greear  > wrote:
> >
> > Hello,
> >
> > I am interested to hear wish lists for network testing features.  We
> make test equipment, supporting lots
> > of wifi stations and a distributed architecture, with built-in udp,
> tcp, ipv6, http, ... protocols,
> > and open to creating/improving some of our automated tests.
> >
> > I know Dave has some test scripts already, so I'm not necessarily
> looking to reimplement that,
> > but more fishing for other/new ideas.
> >
> > Thanks,
> > Ben
> >
> > On 7/2/21 4:28 PM, Bob McMahon wrote:
> >  > I think we need the language of math here. It seems like the
> network power metric, introduced by Kleinrock and Jaffe in the late 70s, is
> something useful.
> >  > Effective end/end queue depths per Little's law also seems
> useful. Both are available in iperf 2 from a test perspective. Repurposing
> test techniques to
> > actual
> >  > traffic could be useful. Hence the question around what exact
> telemetry is useful to apps making socket write() and read() calls.
> >  >
> >  > Bob
> >  >
> >  > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht   >> wrote:
> >  >
> >  > In terms of trying to find "Quality" I have tried to
> encourage folk to
> >  > both read "zen and the art of motorcycle maintenance"[0], and
> Deming's
> >  > work on "total quality management".
> >  >
> >  > My own slice at this network, computer and lifestyle "issue"
> is aiming
> >  > for "imperceptible latency" in all things. [1]. There's a lot
> of
> >  > fallout from that in terms of not just addressing queuing
> delay, but
> >  > caching, prefetching, and learning more about what a user
> really needs
> >  > (as opposed to wants) to know via intelligent agents.
> >  >
> >  > [0] If you want to get depressed, read Pirsig's successor to
> "zen...",
> >  > lila, which is in part about what happens when an engineer
> hits an
> >  > insoluble problem.
> >  > [1] https://www.internetsociety.org/events/latency2013/ <
> https://www.internetsociety.org/events/latency2013/>
> >  >
> >  >
> >  >
> >  > On Thu, Jul 1, 2021 at 6:16 PM David P. Reed <
> dpr...@deepplum.com   dpr...@deepplum.com
> > >> wrote:
> >  >  >
> >  >  > Well, nice that the folks doing the conference  are
> willing to consider that quality of user experience has little to do with
> signalling rate at the
> >  > physical layer or throughput of FTP transfers.
> >  >  >
> >  >  >
> >  >  >
> >  >  > But honestly, the fact that they call the problem "network
> quality" suggests that they REALLY, REALLY don't understand the Internet
> isn't the
> > hardware or
> >  > the routers or even the routing algorithms *to its users*.
> >  >  >
> >  >  >
> >  >  >
> >  >  > By ignoring the diversity of applications now and in the
> future, and the fact that we DON'T KNOW what will be coming up, this
> conference will
> > likely fall
> >  > into the usual trap that net-heads fall into - optimizing for
> some imaginary reality that doesn't exist, and in fact will probably never
> be what users
> >  > actually will do given the chance.
> >  >  >
> >  >  >
> >  >  >
> >  >  > I saw this issue in 1976 in the group developing the
> original Internet 

Re: [Cerowrt-devel] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
A bit off topic per the control and queueing theory discussion; a four
second latency is going to fail our regression automation rigs. Way too
many WiFi users, particularly for games, require sub few hundreds of
milliseconds and sometimes even much lower. A TCP connect() getting behind
a 4 second buffer bloat queue is also a big fail for these test rigs.
Completely agree that average latency isn't what "users" complain about -
it typically requires a tail analysis.

Bob


On Fri, Jul 9, 2021 at 12:31 PM David P. Reed  wrote:

> Len - I admit I made a mistake in challenging Little's Law as being based
> on Poisson processes. It is more general. But it tells you an "average" in
> its base form, and latency averages are not useful for end user
> applications.
>
>
>
> However, Little's Law does assume something that is not actually valid
> about the kind of distributions seen in the network, and in fact, it is NOT
> true that networks converge on Poisson arrival times.
>
>
>
> The key issue is well-described in the sandard analysis of the M/M/1 queue
> (e.g. https://en.wikipedia.org/wiki/M/M/1_queue) , which is done only for
> Poisson processes, and is also limited to "stable" systems. But networks
> are never stable when fully loaded. They get unstable and those
> instabilities persist for a long time in the network. Instability is at
> core the underlying *requirement* of the Internet's usage.
>
>
>
> So specifically: real networks, even large ones, and certainly the
> Internet today, are not asymptotic limits of sums of stationary stochastic
> arrival processes. Each esternal terminal of any real network has a real
> user there, running a real application, and the network is a complex graph.
> This makes it completely unlike a single queue. Even the links within a
> network carry a relatively small number of application flows. There's no
> ability to apply the Law of Large Numbers to the distributions, because any
> particular path contains only a small number of serialized flows with
> hightly variable rates.
>
>
>
> Here's an example of what really happens in a real network (I've observed
> this in 5 different cities on ATT's cellular network, back when it was
> running Alcatel Lucent HSPA+ gear in those cities).
>
> But you can see this on any network where transient overload occurs,
> creating instability.
>
>
>
>
>
> At 7 AM, the data transmission of the network is roughty stable. That's
> because no links are overloaded within the network. Little's Law can tell
> you by observing the delay and throughput on any path that the average
> delay in the network is X.
>
>
>
> Continue sampling delay in the network as the day wears on. At about 10
> AM, ping delay starts to soar into the multiple second range. No packers
> are lost. The peak ping time is about 4000 milliseconds - 4 seconds in most
> of the networks. This is in downtown, no radio errors are reported, no link
> errors.
>
> So it is all queueing delay.
>
>
>
> Now what Little's law doesn't tell you much about average delay, because
> clearly *some* subpiece of the network is fully saturated. But what is
> interesting here is what is happening and where. You can't tell what is
> saturated, and in fact the entire network is quite unstable, because the
> peak is constantly varying and you don't know where the throughput is. All
> the packets are now arriving 4 seconds or so later.
>
>
>
> Why is the situaton not worse than 4 seconds? Well, there are multiple
> things going on:
>
>
>
> 1) TCP may be doing a lot of retransmissions (non-Poisson at all, not
> random either. The arrival process is entirely deterministic in each
> source, based on the retransmission timeout) or it may not be.
>
>
>
> 2) Users are pissed off, because they clicked on a web page, and got
> nothing back. They retry on their screen, or they try another site.
> Meanwhile, the underlying TCP connection remains there, pumping the network
> full of more packets on that old path, which is still backed up with
> packets that haven't been delivered that are sitting in queues. The real
> arrival process is not Poisson at all, its a deterministic, repeated
> retrsnsmission plus a new attempt to connect to a new site.
>
>
>
> 3) When the users get a web page back eventually, it is filled with names
> of other pieces needed to display that web page, which causes some number
> (often as many as 100) new pages to be fetched, ALL at the same time.
> Certainly not a stochastic process that will just obey the law of large
> numbers.
>
>
>
> All of these things are the result of initial instability, causing queues
> to build up.
>
>
>
> So what is the state of the system? is it stable? is it stochastic? Is it
> the sum of enough stochastic stable flows to average out to Poisson?
>
>
>
> The answer is clearly NO. Control theory (not queuing theory) suggests
> that this system is completely uncontrolled and unstable.
>
>
>
> So if the system is in this state, what does 

Re: [Cerowrt-devel] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
One example question is, if it seems useful to control and queuing theory
experts to feedback the non-parametric OWD distributions to the sending
device's transport layer control loop? We find kolmogorov-smirnov distance
matrices as useful for clustering non-parametric distributions and chose to
use it because experimentally OWD distributions have been non-parametric
where the application of the central limit theorem lost the affecting
information. I'm wondering if the KS distances have any use in real word
traffic beyond our post analysis techniques?

Bob

On Sat, Jul 10, 2021 at 12:51 PM Bob McMahon 
wrote:

> "Analyzing that is really difficult, and if we don’t measure and sense, we
> have no hope of understanding, controlling, or ameliorating such
> situations."
>
> It is truly a high honor to observe the queueing theory and control theory
> discussions to the world class experts here. We simple test guys must
> measure things and we'd like those things to be generally useful to all who
> can help towards improvements. Hence back to my original question, what
> network, or other, telemetry do experts here see as useful towards
> measuring active traffic to help with this?
>
> Just some background, and my apologies for the indulgence, but we'd like
> our automation rigs to be able to better emulate "real world scenarios" and
> use stochastic based regression type signals when something goes wrong
> which, for us, is typically a side effect to a driver or firmware code
> change and commit. (Humans need machine level support for this.) It's also
> very frustrating that modern data centers aren't generally providing GPS
> atomic time to servers. (I think part of the idea behind IP packets, etc.
> was to mitigate fault domains and the PSTN stratum clocks were a huge weak
> point.) I find, today, not having a common clock reference "accurate and
> precise enough" is hindering progress towards understanding the complexity
> and towards the ameliorating, at least from our attempts to map "bothersome
> to machine and/or humans and relevant real world phenomenon" into our
> automation environments allowing us to catch things early in the eng life
> cycle.
>
> A few of us have pushed over the last five or more years to add one way
> delay (OWD) of the test traffic (which is not the same as 1/2 RTT nor an
> ICMP ping delay) into iperf 2. That code is available to anyone. The lack
> of adoption applied to OWD has been disheartening. One common response has
> been, "We don't need that because users can't get their devices sync'd
> to the atomic clock anyway." (Also 3 is a larger number than 2 so iperf3
> must be better than iperf2 so let us keep using that as our measurement
> tool - though I digress  ;) ;)
>
> Bob
>
> PS. One can get a stratum 1 clock with a raspberry pi working in a home
> for about $200. I've got one in my home (along with a $2500 OCXO from
> spectracom) and the Pi is reasonable.
> https://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html
>
> On Fri, Jul 9, 2021 at 4:01 PM Leonard Kleinrock  wrote:
>
>> David,
>>
>> No question that non-stationarity and instability are what we often see
>> in networks.  And, non-stationarity and instability are both topics that
>> lead to very complex analytical problems in queueing theory.  You can find
>> some results on the transient analysis in the queueing theory literature
>> (including the second volume of my Queueing Systems book), but they are
>> limited and hard. Nevertheless, the literature does contain some works on
>> transient analysis of queueing systems as applied to network congestion
>> control - again limited. On the other hand, as you said, control theory
>> addresses stability head on and does offer some tools as well, but again,
>> it is hairy.
>>
>> Averages are only averages, but they can provide valuable information.
>> For sure, latency can and does confound behavior.  But, as you point out,
>> it is the proliferation of control protocols that are, in some cases,
>> deployed willy-nilly in networks without proper evaluation of their
>> behavior that can lead to the nasty cycle of large transient latency,
>> frantic repeating of web requests, protocols sending multiple copies, lack
>> of awareness of true capacity or queue size or throughput, etc, all of
>> which you articulate so well, create the chaos and frustration in the
>> network.  Analyzing that is really difficult, and if we don’t measure and
>> sense, we have no hope of understanding, controlling, or ameliorating such
>> situations.
>>
>> Len
>>
>> On Jul 9, 2021, at 12:31 PM, David P. Reed  wrote:
>>
>> Len - I admit I made a mistake in challenging Little's Law as being based
>> on Poisson processes. It is more general. But it tells you an "average" in
>> its base form, and latency averages are not useful for end user
>> applications.
>>
>>
>> However, Little's Law does assume something that is not actually valid
>> about the kind of distributions seen in the 

Re: [Cerowrt-devel] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
"Analyzing that is really difficult, and if we don’t measure and sense, we
have no hope of understanding, controlling, or ameliorating such
situations."

It is truly a high honor to observe the queueing theory and control theory
discussions to the world class experts here. We simple test guys must
measure things and we'd like those things to be generally useful to all who
can help towards improvements. Hence back to my original question, what
network, or other, telemetry do experts here see as useful towards
measuring active traffic to help with this?

Just some background, and my apologies for the indulgence, but we'd like
our automation rigs to be able to better emulate "real world scenarios" and
use stochastic based regression type signals when something goes wrong
which, for us, is typically a side effect to a driver or firmware code
change and commit. (Humans need machine level support for this.) It's also
very frustrating that modern data centers aren't generally providing GPS
atomic time to servers. (I think part of the idea behind IP packets, etc.
was to mitigate fault domains and the PSTN stratum clocks were a huge weak
point.) I find, today, not having a common clock reference "accurate and
precise enough" is hindering progress towards understanding the complexity
and towards the ameliorating, at least from our attempts to map "bothersome
to machine and/or humans and relevant real world phenomenon" into our
automation environments allowing us to catch things early in the eng life
cycle.

A few of us have pushed over the last five or more years to add one way
delay (OWD) of the test traffic (which is not the same as 1/2 RTT nor an
ICMP ping delay) into iperf 2. That code is available to anyone. The lack
of adoption applied to OWD has been disheartening. One common response has
been, "We don't need that because users can't get their devices sync'd
to the atomic clock anyway." (Also 3 is a larger number than 2 so iperf3
must be better than iperf2 so let us keep using that as our measurement
tool - though I digress  ;) ;)

Bob

PS. One can get a stratum 1 clock with a raspberry pi working in a home for
about $200. I've got one in my home (along with a $2500 OCXO from
spectracom) and the Pi is reasonable.
https://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

On Fri, Jul 9, 2021 at 4:01 PM Leonard Kleinrock  wrote:

> David,
>
> No question that non-stationarity and instability are what we often see in
> networks.  And, non-stationarity and instability are both topics that lead
> to very complex analytical problems in queueing theory.  You can find some
> results on the transient analysis in the queueing theory literature
> (including the second volume of my Queueing Systems book), but they are
> limited and hard. Nevertheless, the literature does contain some works on
> transient analysis of queueing systems as applied to network congestion
> control - again limited. On the other hand, as you said, control theory
> addresses stability head on and does offer some tools as well, but again,
> it is hairy.
>
> Averages are only averages, but they can provide valuable information. For
> sure, latency can and does confound behavior.  But, as you point out, it is
> the proliferation of control protocols that are, in some cases, deployed
> willy-nilly in networks without proper evaluation of their behavior that
> can lead to the nasty cycle of large transient latency, frantic repeating
> of web requests, protocols sending multiple copies, lack of awareness of
> true capacity or queue size or throughput, etc, all of which you articulate
> so well, create the chaos and frustration in the network.  Analyzing that
> is really difficult, and if we don’t measure and sense, we have no hope of
> understanding, controlling, or ameliorating such situations.
>
> Len
>
> On Jul 9, 2021, at 12:31 PM, David P. Reed  wrote:
>
> Len - I admit I made a mistake in challenging Little's Law as being based
> on Poisson processes. It is more general. But it tells you an "average" in
> its base form, and latency averages are not useful for end user
> applications.
>
>
> However, Little's Law does assume something that is not actually valid
> about the kind of distributions seen in the network, and in fact, it is NOT
> true that networks converge on Poisson arrival times.
>
>
> The key issue is well-described in the sandard analysis of the M/M/1 queue
> (e.g. https://en.wikipedia.org/wiki/M/M/1_queue) , which is done only for
> Poisson processes, and is also limited to "stable" systems. But networks
> are never stable when fully loaded. They get unstable and those
> instabilities persist for a long time in the network. Instability is at
> core the underlying *requirement* of the Internet's usage.
>
>
> So specifically: real networks, even large ones, and certainly the
> Internet today, are not asymptotic limits of sums of stationary stochastic
> arrival processes. Each esternal terminal of any real network has a 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks for this. I find it both interesting and useful. Learning from those
who came before me reminds me of "standing on the shoulders of giants." I
try to teach my kids that it's not so much us as the giants we choose - so
choose judiciously and, more importantly, be grateful when they provide
their shoulders from which to see.

One challenge I faced with iperf 2 was around flow control's effects on
latency. I find if iperf 2 rate limits on writes then the end/end
latencies, RTT look good because the pipe is basically empty, while rate
limiting reads to the same value fills the window and drives the RTT up.
One might conclude, from a network perspective, the write side is better.
But in reality, the write rate limiting is just pushing the delay into the
application's logic, i.e. the relevant bytes may not be in the pipe but
they aren't at the receiver either, they're stuck somewhere in the "tx
application space."

It wasn't obvious to me how to address this. We added burst measurements
(burst xfer time, and bursts/sec) which, I think, helps.

Bob

On Tue, Jul 13, 2021 at 10:49 AM David P. Reed  wrote:

> Bob -
>
> On Tuesday, July 13, 2021 1:07pm, "Bob McMahon" 
> said:
>
> > "Control at endpoints benefits greatly from even small amounts of
> > information supplied by the network about the degree of congestion
> present
> > on the path."
> >
> > Agreed. The ECN mechanism seems like a shared thermostat in a building.
> > It's basically an on/off where everyone is trying to set the temperature.
> > It does affect, in a non-linear manner, but still an effect. Better than
> a
> > thermostat set at infinity or 0 Kelvin for sure.
> >
> > I find the assumption that congestion occurs "in network" as not always
> > true. Taking OWD measurements with read side rate limiting suggests that
> > equally important to mitigating bufferbloat driven latency using
> congestion
> > signals is to make sure apps read "fast enough" whatever that means. I
> > rarely hear about how important it is for apps to prioritize reads over
> > open sockets. Not sure why that's overlooked and bufferbloat gets all the
> > attention. I'm probably missing something.
>
> In the early days of the Internet protocol and also even ARPANET Host-Host
> protocol there were those who conflated host-level "flow control" (matching
> production rate of data into the network to the destination *process*
> consumption rate of data on a virtual circuit with a source capable of
> variable and unbounded bit rate) with "congestion control" in the network.
> The term "congestion control" wasn't even used in the Internetworking
> project when it was discussing design in the late 1970's. I tried to use it
> in our working group meetings, and every time I said "congestion" the
> response would be phrased as "flow".
>
> The classic example was printing a file's contents from disk to an ASR33
> terminal on an TIP (Terminal IMP). There was flow control in the end-to-end
> protocol to avoid overflowing the TTY's limited buffer. But those who grew
> up with ARPANET knew that thare was no way to accumulate queueing in the
> IMP network, because of RFNM's that required permission for each new packet
> to be sent. RFNM's implicitly prevented congestion from being caused by a
> virtual circuit. But a flow control problem remained, because at the higher
> level protocol, buffering would overflow at the TIP.
>
> TCP adopted a different end-to-end *flow* control, so it solved the flow
> control problem by creating a Windowing mechanism. But it did not by itself
> solve the *congestion* control problem, even congestion built up inside the
> network by a wide-open window and a lazy operating system at the receiving
> end that just said, I've got a lot of virtual memory so I'll open the
> window to maximum size.
>
> There was a lot of confusion, because the guys who came from the ARPANET
> environment, with all links being the same speed and RFNM limits on rate,
> couldn't see why the Internet stack was so collapse-prone. I think Multics,
> for example, as a giant virtual memory system caused congestion by opening
> up its window too much.
>
> This is where Van Jacobson discovered that dropped packets were a "good
> enough" congestion signal because of "fate sharing" among the packets that
> flowed on a bottleneck path, and that windowing (invented for flow control
> by the receiver to protect itself from overflow if the receiver couldn't
> receive fast enough) could be used to slow down the sender to match the
> rate of senders to the capacity of the internal bottleneck link. An elegant
> "hack" that actually worked really well in practice.
>
> Now we view it as a bug if the receiver opens its window too much, or
> otherwise doesn't translate dropped packets (or other incipient-congestion
> signals) to shut down the source transmission rate as quickly as possible.
> Fortunately, the proper state of the internet - the one it should seek as
> its ideal 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
"the infinite TCP flow that converges to a steady behavior is purely
academic"

We find this to be mostly true. Sadly, the tools such as iperf drove to
this condition. While still useful, not realistic.

We added, in iperf 2, the ability to test TCP bursts (--burst-size and
--burst-period) over low duty cycles and get completely different sets of
phenomena with TCP.

It seems (past) time that peak average throughput driving the control loop
needs reconsideration, particularly if TCP_NODELAY is set on a socket. This
is particularly challenging for WiFi because "requests for low latency"
usually triggers no aggregation which doesn't always reduce tail latencies.

Bob

On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk  wrote:

> Ben,
>
> it depends on what one tries to measure. Doing a rate scan using UDP (to
> measure latency distributions under load) is the best thing that we have
> but without actually knowing how resources are shared (fair share as in
> WiFi, FIFO as nearly everywhere else) it becomes very difficult to
> interpret the results or provide a proper argument on latency. You are
> right - TCP stats are a proxy for user experience but I believe they are
> difficult to reproduce (we are always talking about very short TCP flows -
> the infinite TCP flow that converges to a steady behavior is purely
> academic).
>
> By the way, Little's law is a strong tool when it comes to averages. To be
> able to say more (e.g. 1% of the delays is larger than x) one requires more
> information (e.g. the traffic - On-OFF pattern) see [1].  I am not sure
> when does such information readily exist.
>
> Best
> Amr
>
> [1] https://dl.acm.org/doi/10.1145/3341617.3326146 or if behind a paywall
> https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf
>
> 
> Amr Rizk (amr.r...@uni-due.de)
> University of Duisburg-Essen
>
> -Ursprüngliche Nachricht-
> Von: Bloat  Im Auftrag von Ben Greear
> Gesendet: Montag, 12. Juli 2021 22:32
> An: Bob McMahon 
> Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast <
> make-wifi-f...@lists.bufferbloat.net>; Leonard Kleinrock ;
> David P. Reed ; Cake List ;
> co...@lists.bufferbloat.net; cerowrt-devel <
> cerowrt-devel@lists.bufferbloat.net>; bloat 
> Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main
> point
>
> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though, so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers a good real-time look at a complex real-world
> network, then they have something to go on while trying to tune various
> knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and visualization is something we might try to
> accomplish.  I have a feeling I'm not the first person to think of this,
> howeverprobably someone already has done such a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the
> > "failed" congested control mechanisms over the last decades. I think
> > Jaffe pointed this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate
> > for decentralized execution. A formal description of a class of
> > "decentralized flow control algorithms" is given. The feasibility of
> > maximizing power with such algorithms is investigated. On the
> > assumption that communication links behave like M/M/1 servers it is
> shown that no "decentralized flow control algorithm" can maximize network
> power. Power has been suggested in the literature as a network performance
> objective. It is also shown that no objective based only on the users'
> throughputs and average delay is decentralizable. Finally, a restricted
> class of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem
> > for OWD feedback loops aren't effective. How does one design a control
> loop around things that are non-parametric? It also begs the question, what
> are the feed forward knobs that can actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
"Control at endpoints benefits greatly from even small amounts of
information supplied by the network about the degree of congestion present
on the path."

Agreed. The ECN mechanism seems like a shared thermostat in a building.
It's basically an on/off where everyone is trying to set the temperature.
It does affect, in a non-linear manner, but still an effect. Better than a
thermostat set at infinity or 0 Kelvin for sure.

I find the assumption that congestion occurs "in network" as not always
true. Taking OWD measurements with read side rate limiting suggests that
equally important to mitigating bufferbloat driven latency using congestion
signals is to make sure apps read "fast enough" whatever that means. I
rarely hear about how important it is for apps to prioritize reads over
open sockets. Not sure why that's overlooked and bufferbloat gets all the
attention. I'm probably missing something.

Bob

On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk  wrote:

> Ben,
>
> it depends on what one tries to measure. Doing a rate scan using UDP (to
> measure latency distributions under load) is the best thing that we have
> but without actually knowing how resources are shared (fair share as in
> WiFi, FIFO as nearly everywhere else) it becomes very difficult to
> interpret the results or provide a proper argument on latency. You are
> right - TCP stats are a proxy for user experience but I believe they are
> difficult to reproduce (we are always talking about very short TCP flows -
> the infinite TCP flow that converges to a steady behavior is purely
> academic).
>
> By the way, Little's law is a strong tool when it comes to averages. To be
> able to say more (e.g. 1% of the delays is larger than x) one requires more
> information (e.g. the traffic - On-OFF pattern) see [1].  I am not sure
> when does such information readily exist.
>
> Best
> Amr
>
> [1] https://dl.acm.org/doi/10.1145/3341617.3326146 or if behind a paywall
> https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf
>
> 
> Amr Rizk (amr.r...@uni-due.de)
> University of Duisburg-Essen
>
> -Ursprüngliche Nachricht-
> Von: Bloat  Im Auftrag von Ben Greear
> Gesendet: Montag, 12. Juli 2021 22:32
> An: Bob McMahon 
> Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast <
> make-wifi-f...@lists.bufferbloat.net>; Leonard Kleinrock ;
> David P. Reed ; Cake List ;
> co...@lists.bufferbloat.net; cerowrt-devel <
> cerowrt-devel@lists.bufferbloat.net>; bloat 
> Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main
> point
>
> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though, so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers a good real-time look at a complex real-world
> network, then they have something to go on while trying to tune various
> knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and visualization is something we might try to
> accomplish.  I have a feeling I'm not the first person to think of this,
> howeverprobably someone already has done such a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the
> > "failed" congested control mechanisms over the last decades. I think
> > Jaffe pointed this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate
> > for decentralized execution. A formal description of a class of
> > "decentralized flow control algorithms" is given. The feasibility of
> > maximizing power with such algorithms is investigated. On the
> > assumption that communication links behave like M/M/1 servers it is
> shown that no "decentralized flow control algorithm" can maximize network
> power. Power has been suggested in the literature as a network performance
> objective. It is also shown that no objective based only on the users'
> throughputs and average delay is decentralizable. Finally, a restricted
> class of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem
> > for OWD feedback loops aren't effective. How does one design a control
> loop around things that are non-parametric? It also begs the question, what
> are the feed forward knobs that can actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, 

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
fair enough, but for this "RF emulator device" being able to support
distance matrices, even hollow symmetric ones, is much better than what's
typically done. The variable solid state phase shifters are 0-360 so don't
provide real time delays either.

This is another "something is better than nothing" type proposal. I think
it can be deployed at a relatively low cost which allows for more
standardized, automated test rigs and much less human interactions and
human errors.

Bob

On Mon, Aug 2, 2021 at 9:30 PM David Lang  wrote:

> symmetry is not always (or usually) true. stations are commonly heard at
> much
> larger distances than they can talk, mobile devices have much less
> transmit
> power (becuase they are operating on batteries) than fixed stations, and
> when
> you adjust the transmit power on a station, you don't adjust it's receive
> sensitivity.
>
> David Lang
>
>   On Mon, 2 Aug 2021, Bob McMahon wrote:
>
> > Date: Mon, 2 Aug 2021 20:23:06 -0700
> > From: Bob McMahon 
> > To: David Lang 
> > Cc: Ben Greear ,
> > Luca Muscariello ,
> > Cake List ,
> > Make-Wifi-fast ,
> > Leonard Kleinrock , starl...@lists.bufferbloat.net,
> > co...@lists.bufferbloat.net,
> > cerowrt-devel ,
> > bloat 
> > Subject: Re: [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug
> 2:
> > Internet Quality workshop CFP for the internet architecture board
> >
> > The distance matrix defines signal attenuations/loss between pairs.  It's
> > straightforward to create a distance matrix that has hidden nodes because
> > all "signal  loss" between pairs is defined.  Let's say a 120dB
> attenuation
> > path will cause a node to be hidden as an example.
> >
> > AB CD
> > A   -   35   120   65
> > B -  65   65
> > C   -   65
> > D -
> >
> > So in the above, AC are hidden from each other but nobody else is. It
> does
> > assume symmetry between pairs but that's typically true.
> >
> > The RF device takes these distance matrices as settings and calculates
> the
> > five branch tree values (as demonstrated in the video). There are
> > limitations to solutions though but I've found those not to be an issue
> to
> > date. I've been able to produce hidden nodes quite readily. Add the phase
> > shifters and spatial stream powers can also be affected, but this isn't
> > shown in this simple example.
> >
> > Bob
> >
> > On Mon, Aug 2, 2021 at 8:12 PM David Lang  wrote:
> >
> >> I guess it depends on what you are intending to test. If you are not
> going
> >> to
> >> tinker with any of the over-the-air settings (including the number of
> >> packets
> >> transmitted in one aggregate), the details of what happen over the air
> >> don't
> >> matter much.
> >>
> >> But if you are going to be doing any tinkering with what is getting
> sent,
> >> and
> >> you ignore the hidden transmitter type problems, you will create a
> >> solution that
> >> seems to work really well in the lab and falls on it's face out in the
> >> wild
> >> where spectrum overload and hidden transmitters are the norm (at least
> in
> >> urban
> >> areas), not rare corner cases.
> >>
> >> you don't need to include them in every test, but you need to have a way
> >> to
> >> configure your lab to include them before you consider any
> >> settings/algorithm
> >> ready to try in the wild.
> >>
> >> David Lang
> >>
> >> On Mon, 2 Aug 2021, Bob McMahon wrote:
> >>
> >>> We find four nodes, a primary BSS and an adjunct one quite good for
> lots
> >> of
> >>> testing.  The six nodes allows for a primary BSS and two adjacent ones.
> >> We
> >>> want to minimize complexity to necessary and sufficient.
> >>>
> >>> The challenge we find is having variability (e.g. montecarlos) that's
> >>> reproducible and has relevant information. Basically, the distance
> >> matrices
> >>> have h-matrices as their elements. Our chips can provide these
> >> h-matrices.
> >>>
> >>> The parts for solid state programmable attenuators and phase shifters
> >>> aren't very expensive. A device that supports a five branch tree and
> 2x2
> >>> MIMO seems a very good starting point.
> >>>
> >>> Bob
> >>>
> >>> On Mon, Aug 2, 2021 at 4:55 PM Ben Greear 
> >> wrote:
> >>>
>  On 8/2/21 4:16 PM, David Lang wrote:
> > If you are going to setup a test environment for wifi, you need to
>  include the ability to make a fe cases that only happen with RF, not
> >> with
>  wired networks and
> > are commonly overlooked
> >
> > 1. station A can hear station B and C but they cannot hear each other
> > 2. station A can hear station B but station B cannot hear station A
> 3.
>  station A can hear that station B is transmitting, but not with a
> strong
>  enough signal to
> > decode the signal (yes in theory you can work around interference,
> but
>  in practice interference is still a real thing)
> >
> > David Lang
> >
> 
>  To add to 

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
That distance matrices manage energy between nodes.  The slides show a 5
branch tree to realize 4 nodes (and that distance matrix) and a diagram for
11 degrees of freedom for 6 nodes (3 BSS)  The python code will compute the
branch attenuations based on a supplied distance matrix. There will of
course be some distance matrices that cannot be achieved  per the reduction
of the degrees of freedom required to realize this using
physical devices in a cost effective manner.

Bob

On Mon, Aug 2, 2021 at 4:16 PM David Lang  wrote:

> If you are going to setup a test environment for wifi, you need to include
> the
> ability to make a fe cases that only happen with RF, not with wired
> networks and
> are commonly overlooked
>
> 1. station A can hear station B and C but they cannot hear each other
> 2. station A can hear station B but station B cannot hear station A
> 3. station A can hear that station B is transmitting, but not with a
> strong
> enough signal to decode the signal (yes in theory you can work around
> interference, but in practice interference is still a real thing)
>
> David Lang
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
The distance matrix defines signal attenuations/loss between pairs.  It's
straightforward to create a distance matrix that has hidden nodes because
all "signal  loss" between pairs is defined.  Let's say a 120dB attenuation
path will cause a node to be hidden as an example.

 AB CD
A   -   35   120   65
B -  65   65
C   -   65
D -

So in the above, AC are hidden from each other but nobody else is. It does
assume symmetry between pairs but that's typically true.

The RF device takes these distance matrices as settings and calculates the
five branch tree values (as demonstrated in the video). There are
limitations to solutions though but I've found those not to be an issue to
date. I've been able to produce hidden nodes quite readily. Add the phase
shifters and spatial stream powers can also be affected, but this isn't
shown in this simple example.

Bob

On Mon, Aug 2, 2021 at 8:12 PM David Lang  wrote:

> I guess it depends on what you are intending to test. If you are not going
> to
> tinker with any of the over-the-air settings (including the number of
> packets
> transmitted in one aggregate), the details of what happen over the air
> don't
> matter much.
>
> But if you are going to be doing any tinkering with what is getting sent,
> and
> you ignore the hidden transmitter type problems, you will create a
> solution that
> seems to work really well in the lab and falls on it's face out in the
> wild
> where spectrum overload and hidden transmitters are the norm (at least in
> urban
> areas), not rare corner cases.
>
> you don't need to include them in every test, but you need to have a way
> to
> configure your lab to include them before you consider any
> settings/algorithm
> ready to try in the wild.
>
> David Lang
>
> On Mon, 2 Aug 2021, Bob McMahon wrote:
>
> > We find four nodes, a primary BSS and an adjunct one quite good for lots
> of
> > testing.  The six nodes allows for a primary BSS and two adjacent ones.
> We
> > want to minimize complexity to necessary and sufficient.
> >
> > The challenge we find is having variability (e.g. montecarlos) that's
> > reproducible and has relevant information. Basically, the distance
> matrices
> > have h-matrices as their elements. Our chips can provide these
> h-matrices.
> >
> > The parts for solid state programmable attenuators and phase shifters
> > aren't very expensive. A device that supports a five branch tree and 2x2
> > MIMO seems a very good starting point.
> >
> > Bob
> >
> > On Mon, Aug 2, 2021 at 4:55 PM Ben Greear 
> wrote:
> >
> >> On 8/2/21 4:16 PM, David Lang wrote:
> >>> If you are going to setup a test environment for wifi, you need to
> >> include the ability to make a fe cases that only happen with RF, not
> with
> >> wired networks and
> >>> are commonly overlooked
> >>>
> >>> 1. station A can hear station B and C but they cannot hear each other
> >>> 2. station A can hear station B but station B cannot hear station A 3.
> >> station A can hear that station B is transmitting, but not with a strong
> >> enough signal to
> >>> decode the signal (yes in theory you can work around interference, but
> >> in practice interference is still a real thing)
> >>>
> >>> David Lang
> >>>
> >>
> >> To add to this, I think you need lots of different station devices,
> >> different capabilities (/n, /ac, /ax, etc)
> >> different numbers of spatial streams, and different distances from the
> >> AP.  From download queueing perspective, changing
> >> the capabilities may be sufficient while keeping all stations at same
> >> distance.  This assumes you are not
> >> actually testing the wifi rate-ctrl alg. itself, so different throughput
> >> levels for different stations would be enough.
> >>
> >> So, a good station emulator setup (and/or pile of real stations) and a
> few
> >> RF chambers and
> >> programmable attenuators and you can test that setup...
> >>
> >>  From upload perspective, I guess same setup would do the job.
> >> Queuing/fairness might depend a bit more on the
> >> station devices, emulated or otherwise, but I guess a clever AP could
> >> enforce fairness in upstream direction
> >> too by implementing per-sta queues.
> >>
> >> Thanks,
> >> Ben
> >>
> >> --
> >> Ben Greear 
> >> Candela Technologies Inc  http://www.candelatech.com
> >>
> >
> >
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is 

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
We find four nodes, a primary BSS and an adjunct one quite good for lots of
testing.  The six nodes allows for a primary BSS and two adjacent ones. We
want to minimize complexity to necessary and sufficient.

The challenge we find is having variability (e.g. montecarlos) that's
reproducible and has relevant information. Basically, the distance matrices
have h-matrices as their elements. Our chips can provide these h-matrices.

The parts for solid state programmable attenuators and phase shifters
aren't very expensive. A device that supports a five branch tree and 2x2
MIMO seems a very good starting point.

Bob

On Mon, Aug 2, 2021 at 4:55 PM Ben Greear  wrote:

> On 8/2/21 4:16 PM, David Lang wrote:
> > If you are going to setup a test environment for wifi, you need to
> include the ability to make a fe cases that only happen with RF, not with
> wired networks and
> > are commonly overlooked
> >
> > 1. station A can hear station B and C but they cannot hear each other
> > 2. station A can hear station B but station B cannot hear station A 3.
> station A can hear that station B is transmitting, but not with a strong
> enough signal to
> > decode the signal (yes in theory you can work around interference, but
> in practice interference is still a real thing)
> >
> > David Lang
> >
>
> To add to this, I think you need lots of different station devices,
> different capabilities (/n, /ac, /ax, etc)
> different numbers of spatial streams, and different distances from the
> AP.  From download queueing perspective, changing
> the capabilities may be sufficient while keeping all stations at same
> distance.  This assumes you are not
> actually testing the wifi rate-ctrl alg. itself, so different throughput
> levels for different stations would be enough.
>
> So, a good station emulator setup (and/or pile of real stations) and a few
> RF chambers and
> programmable attenuators and you can test that setup...
>
>  From upload perspective, I guess same setup would do the job.
> Queuing/fairness might depend a bit more on the
> station devices, emulated or otherwise, but I guess a clever AP could
> enforce fairness in upstream direction
> too by implementing per-sta queues.
>
> Thanks,
> Ben
>
> --
> Ben Greear 
> Candela Technologies Inc  http://www.candelatech.com
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I found the following talk relevant to distances between all the nodes.
https://www.youtube.com/watch?v=PNoUcQTCxiM

Distance is an abstract idea but applies to energy into a node as well as
phylogenetic trees. It's the same problem, i.e. fitting a distance matrix
using some sort of tree. I've found the five branch tree works well for
four nodes.

Bob

On Mon, Aug 2, 2021 at 5:37 PM Leonard Kleinrock  wrote:

> These cases are what my student, Fouad Tobagi and I called the Hidden
> Terminal Problem (with the Busy Tone solution) back in 1975.
>
> Len
>
>
> > On Aug 2, 2021, at 4:16 PM, David Lang  wrote:
> >
> > If you are going to setup a test environment for wifi, you need to
> include the ability to make a fe cases that only happen with RF, not with
> wired networks and are commonly overlooked
> >
> > 1. station A can hear station B and C but they cannot hear each other
> > 2. station A can hear station B but station B cannot hear station A 3.
> station A can hear that station B is transmitting, but not with a strong
> enough signal to decode the signal (yes in theory you can work around
> interference, but in practice interference is still a real thing)
> >
> > David Lang
> >
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Cake] [Make-wifi-fast] [Starlink] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Another thing to keep in mind is we're using a poor man's version of
emulating "passive channels" so the device transmit powers can provide
power asymmetry. The distance matrix is about the h-matrices (as shown
early in the slides.)  Even with that though, the h-matrix elements aren't
likely symmetric but it's a reasonable starting point to assume they are.
Also being able to switch them in near realtime allows for some forms of
transmit and receive asymmetry in the emulated channels as well.

Bob

On Mon, Aug 2, 2021 at 9:44 PM David Lang  wrote:

> I agree that we don't want to make perfect the enemy of better.
>
> A lot of the issues I'm calling out can be simulated/enhanced with
> different
> power levels.
>
> over wifi distances, I don't think time delays are going to be noticable
> (we're
> talking 10s to low 100s of feet, not miles)
>
> David Lang
>
> On Mon, 2 Aug 2021, Bob McMahon wrote:
>
> > fair enough, but for this "RF emulator device" being able to support
> > distance matrices, even hollow symmetric ones, is much better than what's
> > typically done. The variable solid state phase shifters are 0-360 so
> don't
> > provide real time delays either.
> >
> > This is another "something is better than nothing" type proposal. I think
> > it can be deployed at a relatively low cost which allows for more
> > standardized, automated test rigs and much less human interactions and
> > human errors.
> >
> > Bob
> >
> > On Mon, Aug 2, 2021 at 9:30 PM David Lang  wrote:
> >
> >> symmetry is not always (or usually) true. stations are commonly heard at
> >> much
> >> larger distances than they can talk, mobile devices have much less
> >> transmit
> >> power (becuase they are operating on batteries) than fixed stations, and
> >> when
> >> you adjust the transmit power on a station, you don't adjust it's
> receive
> >> sensitivity.
> >>
> >> David Lang
> >>
> >>   On Mon, 2 Aug 2021, Bob McMahon wrote:
> >>
> >>> Date: Mon, 2 Aug 2021 20:23:06 -0700
> >>> From: Bob McMahon 
> >>> To: David Lang 
> >>> Cc: Ben Greear ,
> >>> Luca Muscariello ,
> >>> Cake List ,
> >>> Make-Wifi-fast ,
> >>> Leonard Kleinrock , starl...@lists.bufferbloat.net
> ,
> >>> co...@lists.bufferbloat.net,
> >>> cerowrt-devel ,
> >>> bloat 
> >>> Subject: Re: [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug
> >> 2:
> >>> Internet Quality workshop CFP for the internet architecture board
> >>>
> >>> The distance matrix defines signal attenuations/loss between pairs.
> It's
> >>> straightforward to create a distance matrix that has hidden nodes
> because
> >>> all "signal  loss" between pairs is defined.  Let's say a 120dB
> >> attenuation
> >>> path will cause a node to be hidden as an example.
> >>>
> >>> AB CD
> >>> A   -   35   120   65
> >>> B -  65   65
> >>> C   -   65
> >>> D -
> >>>
> >>> So in the above, AC are hidden from each other but nobody else is. It
> >> does
> >>> assume symmetry between pairs but that's typically true.
> >>>
> >>> The RF device takes these distance matrices as settings and calculates
> >> the
> >>> five branch tree values (as demonstrated in the video). There are
> >>> limitations to solutions though but I've found those not to be an issue
> >> to
> >>> date. I've been able to produce hidden nodes quite readily. Add the
> phase
> >>> shifters and spatial stream powers can also be affected, but this isn't
> >>> shown in this simple example.
> >>>
> >>> Bob
> >>>
> >>> On Mon, Aug 2, 2021 at 8:12 PM David Lang  wrote:
> >>>
>  I guess it depends on what you are intending to test. If you are not
> >> going
>  to
>  tinker with any of the over-the-air settings (including the number of
>  packets
>  transmitted in one aggregate), the details of what happen over the air
>  don't
>  matter much.
> 
>  But if you are going to be doing any tinkering with what is getting
> >> sent,
>  and
>  you ignore the hidden transmitter type problems, you will create a
>  solution that
>  seems to work really well in the lab and falls on it's face out in the
>  wild
>  where spectrum overload and hidden transmitters are the norm (at least
> >> in
>  urban
>  areas), not rare corner cases.
> 
>  you don't need to include them in every test, but you need to have a
> way
>  to
>  configure your lab to include them before you consider any
>  settings/algorithm
>  ready to try in the wild.
> 
>  David Lang
> 
>  On Mon, 2 Aug 2021, Bob McMahon wrote:
> 
> > We find four nodes, a primary BSS and an adjunct one quite good for
> >> lots
>  of
> > testing.  The six nodes allows for a primary BSS and two adjacent
> ones.
>  We
> > want to minimize complexity to necessary and sufficient.
> >
> > The challenge we find is having variability (e.g. montecarlos) that's
> 

Re: [Cerowrt-devel] [Starlink] [Cake] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks - your wording is more accurate. The path loss matrix is hollow
symmetric while the RF channel is reciprocal.

The challenge comes when adding phase shifters. Then it's not just a path
loss matrix anymore.

Bob

On Sat, Aug 7, 2021 at 10:04 PM Dick Roy  wrote:

>
>
>
> --
>
> *From:* Starlink [mailto:starlink-boun...@lists.bufferbloat.net] *On
> Behalf Of *Bob McMahon
> *Sent:* Monday, August 2, 2021 8:23 PM
> *To:* David Lang
> *Cc:* starl...@lists.bufferbloat.net; Make-Wifi-fast; Cake List;
> co...@lists.bufferbloat.net; cerowrt-devel; bloat
> *Subject:* Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due Aug
> 2: Internet Quality workshop CFP for the internet architecture board
>
>
>
> The distance matrix defines signal attenuations/loss between pairs.
>
> *[RR] Which makes it a path loss matrix rather than a distance matrix
> actually.*
>
> It's straightforward to create a distance matrix that has hidden nodes
> because all "signal  loss" between pairs is defined.  Let's say a 120dB
> attenuation path will cause a node to be hidden as an example.
>
>  AB CD
>
> A   -   35   120   65
>
> B -  65   65
>
> C   -   65
>
> D -
>
> So in the above, AC are hidden from each other but nobody else is. It does
> assume symmetry between pairs but that's typically true.
>
> *[RR] I’m guessing you really mean reciprocal rather than symmetric. An RF
> channel is reciprocal if the loss when A is transmitting to B is the same
> as that when B is transmitting to A. When the tx powers and rx
> sensitivities are such that when combined with the path loss(es) the “link
> budget” is  the same in both directions, the links are balanced and
> therefore have the same capacity. *
>
>
>
> The RF device takes these distance matrices as settings and calculates the
> five branch tree values (as demonstrated in the video).
>
> There are limitations to solutions though but I've found those not to be
> an issue to date. I've been able to produce hidden nodes quite readily. Add
> the phase shifters and spatial stream powers can also be affected, but this
> isn't shown in this simple example.
>
> Bob
>
>
>
> On Mon, Aug 2, 2021 at 8:12 PM David Lang  wrote:
>
> I guess it depends on what you are intending to test. If you are not going
> to
> tinker with any of the over-the-air settings (including the number of
> packets
> transmitted in one aggregate), the details of what happen over the air
> don't
> matter much.
>
> But if you are going to be doing any tinkering with what is getting sent,
> and
> you ignore the hidden transmitter type problems, you will create a
> solution that
> seems to work really well in the lab and falls on it's face out in the
> wild
> where spectrum overload and hidden transmitters are the norm (at least in
> urban
> areas), not rare corner cases.
>
> you don't need to include them in every test, but you need to have a way
> to
> configure your lab to include them before you consider any
> settings/algorithm
> ready to try in the wild.
>
> David Lang
>
> On Mon, 2 Aug 2021, Bob McMahon wrote:
>
> > We find four nodes, a primary BSS and an adjunct one quite good for lots
> of
> > testing.  The six nodes allows for a primary BSS and two adjacent ones.
> We
> > want to minimize complexity to necessary and sufficient.
> >
> > The challenge we find is having variability (e.g. montecarlos) that's
> > reproducible and has relevant information. Basically, the distance
> matrices
> > have h-matrices as their elements. Our chips can provide these
> h-matrices.
> >
> > The parts for solid state programmable attenuators and phase shifters
> > aren't very expensive. A device that supports a five branch tree and 2x2
> > MIMO seems a very good starting point.
> >
> > Bob
> >
> > On Mon, Aug 2, 2021 at 4:55 PM Ben Greear 
> wrote:
> >
> >> On 8/2/21 4:16 PM, David Lang wrote:
> >>> If you are going to setup a test environment for wifi, you need to
> >> include the ability to make a fe cases that only happen with RF, not
> with
> >> wired networks and
> >>> are commonly overlooked
> >>>
> >>> 1. station A can hear station B and C but they cannot hear each other
> >>> 2. station A can hear station B but station B cannot hear station A 3.
> >> station A can hear that station B is transmitting, but not with a strong
> >> enough signal to
> >>> decode the signal (yes in theory you can work around interference, but
> >> in practice interference is still a real thing)
> >>>
> >>> David Lang
> >>>
> >>
> >> To add to this, I think you need lots of different station devices,
> >> different capabilities (/n, /ac, /ax, etc)
> >> different numbers of spatial streams, and different distances from the
> >> AP.  From download queueing perspective, changing
> >> the capabilities may be sufficient while keeping all stations at same
> >> distance.  This assumes you are not
> >> actually testing the wifi rate-ctrl 

Re: [Cerowrt-devel] [Starlink] [Cake] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
We have hundreds of test rigs in multiple labs all over geography. Each rig
is shielded from the others using things like RF enclosures. We want
reproducibility in the RF paths/channels as well as variability. Most have
built fixed rigs using conducted equipment. This is far from anything real.
A butler matrix produces great condition numbers but that makes it too easy
for MIMO rate selection algorithms.

Our real world test is using a real house that has been rented. Not cheap
nor scalable.

There is quite a gap between the two. A RF path device that supports both
variable range and variable mixing is a step towards closing the gap.

Bob

On Sat, Aug 7, 2021 at 10:07 PM Dick Roy  wrote:

>
>
>
> --
>
> *From:* Starlink [mailto:starlink-boun...@lists.bufferbloat.net] *On
> Behalf Of *Bob McMahon
> *Sent:* Monday, August 2, 2021 6:24 PM
> *To:* Leonard Kleinrock
> *Cc:* starl...@lists.bufferbloat.net; Make-Wifi-fast; Cake List;
> co...@lists.bufferbloat.net; cerowrt-devel; bloat
> *Subject:* Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due Aug
> 2: Internet Quality workshop CFP for the internet architecture board
>
>
>
> I found the following talk relevant to distances between all the nodes.
> https://www.youtube.com/watch?v=PNoUcQTCxiM
>
> Distance is an abstract idea but applies to energy into a node as well as
> phylogenetic trees. It's the same problem, i.e. fitting a distance matrix
> using some sort of tree. I've found the five branch tree works well for
> four nodes.
>
> *[RR] These trees are means for approximating a higher dimensional
> real-world problem with a lower dimensional structure.  You may be doing
> this to save hardware when trying to cable up some complex test scenarios,
> however I’m wondering why?  Why not just put the STAs in the lab and turn
> them on rather than cabling them?*
>
>
>
> Bob
>
>
>
> On Mon, Aug 2, 2021 at 5:37 PM Leonard Kleinrock  wrote:
>
> These cases are what my student, Fouad Tobagi and I called the Hidden
> Terminal Problem (with the Busy Tone solution) back in 1975.
>
> Len
>
>
> > On Aug 2, 2021, at 4:16 PM, David Lang  wrote:
> >
> > If you are going to setup a test environment for wifi, you need to
> include the ability to make a fe cases that only happen with RF, not with
> wired networks and are commonly overlooked
> >
> > 1. station A can hear station B and C but they cannot hear each other
> > 2. station A can hear station B but station B cannot hear station A 3.
> station A can hear that station B is transmitting, but not with a strong
> enough signal to decode the signal (yes in theory you can work around
> interference, but in practice interference is still a real thing)
> >
> > David Lang
> >
>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Starlink] [Cake] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
The four nodes being connected per a five branch tree using variable
attenuators on the branches can produce hidden nodes quite readily. The
solutions for the attenuations are straight forward per being given the
desired distance matrices.

The challenging part is supporting distance matrices that have h-matrices
as the elements. The solid state variable phase shifters combined with
chips that can dump the h-matrix works fairly well per running monte carlos
and grabbing such a distance matrix. Mapping these distance matrices,
matrices with h-matrices as the elements, to "real world" conditions is not
so easy. But having the ability to do so at a reasonable price and in a
reproducible way is very worthwhile to automated testing systems.

Bob

On Sat, Aug 7, 2021 at 9:35 PM Dick Roy  wrote:

>
>
> -Original Message-
> From: Starlink [mailto:starlink-boun...@lists.bufferbloat.net] On Behalf
> Of
> David Lang
> Sent: Monday, August 2, 2021 9:31 PM
> To: Bob McMahon
> Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast; Cake List;
> co...@lists.bufferbloat.net; cerowrt-devel; bloat
> Subject: Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2:
> Internet Quality workshop CFP for the internet architecture board
>
> symmetry is not always (or usually) true.
> [RR] There is a big difference between "symmetric RF channels" and
> "balanced
> RF links".  Be careful not to confuse the two.
>
> stations are commonly heard at much
> larger distances than they can talk, mobile devices have much less
> transmit
> power (becuase they are operating on batteries) than fixed stations, and
> when
> you adjust the transmit power on a station, you don't adjust it's receive
> sensitivity.
>
> [RR] Not quite true. Rx sensitivity is a function of MCS (the modulation
> and
> coding scheme) and those levels can be adjusted, both up and down, by
> changing the MCS.  This is in fact one of the major tools that needs to be
> integrated into wireless systems today.  It's generally overlooked, though
> not always! Starlink should be doing this if they are not already BTW!
>
> David Lang
>
>   On Mon, 2 Aug 2021, Bob McMahon wrote:
>
> > Date: Mon, 2 Aug 2021 20:23:06 -0700
> > From: Bob McMahon 
> > To: David Lang 
> > Cc: Ben Greear ,
> > Luca Muscariello ,
> > Cake List ,
> > Make-Wifi-fast ,
> > Leonard Kleinrock , starl...@lists.bufferbloat.net,
> > co...@lists.bufferbloat.net,
> > cerowrt-devel ,
> > bloat 
> > Subject: Re: [Cake] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug
> 2:
> > Internet Quality workshop CFP for the internet architecture board
> >
> > The distance matrix defines signal attenuations/loss between pairs.  It's
> > straightforward to create a distance matrix that has hidden nodes because
> > all "signal  loss" between pairs is defined.  Let's say a 120dB
> attenuation
> > path will cause a node to be hidden as an example.
> >
> > AB CD
> > A   -   35   120   65
> > B -  65   65
> > C   -   65
> > D -
> >
> > So in the above, AC are hidden from each other but nobody else is. It
> does
> > assume symmetry between pairs but that's typically true.
> >
> > The RF device takes these distance matrices as settings and calculates
> the
> > five branch tree values (as demonstrated in the video). There are
> > limitations to solutions though but I've found those not to be an issue
> to
> > date. I've been able to produce hidden nodes quite readily. Add the phase
> > shifters and spatial stream powers can also be affected, but this isn't
> > shown in this simple example.
> >
> > Bob
> >
> > On Mon, Aug 2, 2021 at 8:12 PM David Lang  wrote:
> >
> >> I guess it depends on what you are intending to test. If you are not
> going
> >> to
> >> tinker with any of the over-the-air settings (including the number of
> >> packets
> >> transmitted in one aggregate), the details of what happen over the air
> >> don't
> >> matter much.
> >>
> >> But if you are going to be doing any tinkering with what is getting
> sent,
> >> and
> >> you ignore the hidden transmitter type problems, you will create a
> >> solution that
> >> seems to work really well in the lab and falls on it's face out in the
> >> wild
> >> where spectrum overload and hidden transmitters are the norm (at least
> in
> >> urban
> >> areas), not rare corner cases.
> >>
> >> you don't need to include them in every test, but you need to have a way
> >> to
> >> configure your lab to include them before you consider any
> >> settings/algorithm
> >> ready to try in the wild.
> >>
> >> David Lang
> >>
> >> On Mon, 2 Aug 2021, Bob McMahon wrote:
> >>
> >>> We find four nodes, a primary BSS and an adjunct one quite good for
> lots
> >> of
> >>> testing.  The six nodes allows for a primary BSS and two adjacent ones.
> >> We
> >>> want to minimize complexity to necessary and sufficient.
> >>>
> >>> The challenge we find is having 

Re: [Cerowrt-devel] [Bloat] [Make-wifi-fast] [Starlink] [Cake] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Some people put them on roombas. Doesn't work well inside these
http://ramseytest.com/index.php

On Sun, Aug 8, 2021 at 11:48 AM Jonathan Morton 
wrote:

> > On 8 Aug, 2021, at 9:36 pm, Aaron Wood  wrote:
> >
> > Less common, but something I still see, is that a moving station has
> continual issues staying in proper MIMO phase(s) with the AP.  Or I think
> that's what's happening.  Slow, continual movement of the two, relative to
> each other, and the packet rate drops through the floor until they stop
> having relative motion.  And I assume that also applies to time-varying
> path-loss and path-distance (multipath reflections).
>
> So is it time to mount test stations on model railway wagons?
>
>  - Jonathan Morton

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Starlink] [Cake] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
This amplitude only channel estimate shown was taken from radios connected
using conducted equipment or cables. It illustrates how non-ideal conducted
equipment based testing is, i.e. our signal processing and MCS rate
selection engineers aren't being sufficiently challenged!

The cost of $2.5K for a butler matrix is just one component. Each antenna
is connected to a programmable attenuator. Then the shielded cabling. Then
one of these per engineer and tens to low hundreds per each automated test
engineer. This doesn't include the cost of programmers to write the code.
The expenses grow quickly. Hence the idea to amortize a better design
across the industry (if viable.)

Modeling the distance matrix (suggestions for a better name?) and realizing
D1 path loss using a five branch tree and programmable attenuators has
proven to work for testing things like hidden nodes and for TX op
arbitrations. The next missing piece is to realize the mixing, the h(n,n)
below with programmability and at a reasonable price. That's where the
programmable phase shifters come in. Our chips will dump their chan
estimates relatively quickly so we can run monte carlos and calibrate the
equipment, producing the spatial stream eigen values or condition numbers
as well. Early prototyping showed that phase shifters will affect spatial
stream powers per the algorithms and this should work. Being able to affect
both the path loss and mixing within 10 ms of a command seems a reasonable
ask if using solid state parts. No need for roombas.


[image: CodeCogsEqn (2).png]

Of course, all of these RF effects affect network availability and, hence,
queueing too. We've done a lot of work with iperf 2 around latencies to
help qualify that. That's released as open source.

Complex indeed,

Bob

On Tue, Aug 10, 2021 at 11:11 AM Dick Roy  wrote:

> To add a bit more, as is easily seen below, the amplitudes of each of the
> transfer functions between the three transmit and three receive antennas
> are extremely similar.  This is to be expected, of course, since the
> “aperture” of each array is very small compared to the distance between
> them.  What is much more interesting and revealing is the relative phases.
> Obviously this requires coherent receivers, and ultimately if you want to
> control the spatial distribution of power (aka SDMA (or MIMO in some
> circles) coherent transmitters. It turns out that just knowing the
> amplitude of the transfer functions is not really all that useful for
> anything other than detecting a broken solder joint:^)))
>
>
>
> Also, do not forget that depending how these experiments were conducted,
> the estimates are either of the RF channel itself (aka path loss),or of the
> RF channel in combination with the transfer functions of the transmitters
> and//or receivers.  What this means is the CALIBRATION is CRUCIAL!  Those
> who do not calibrate, are doomed to fail   I suspect that it is in
> calibration where the major difference in performance between vendors’’
> products can be found :^
>
>
>
> It’s complicated …
>
>
> --
>
> *From:* Bob McMahon [mailto:bob.mcma...@broadcom.com]
> *Sent:* Tuesday, August 10, 2021 10:07 AM
> *To:* dick...@alum.mit.edu
> *Cc:* Rodney W. Grimes; Cake List; Make-Wifi-fast;
> starl...@lists.bufferbloat.net; codel; cerowrt-devel; bloat
> *Subject:* Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due Aug
> 2: Internet Quality workshop CFP for the internet architecture board
>
>
>
> The slides show that for WiFi every transmission produces a complex
> frequency response, aka the h-matrix. This is valid for that one
> transmission only.  The slides show an amplitude plot for a 3 radio device
> hence the 9 elements per the h-matrix. It's assumed that the WiFi STA/AP is
> stationary such that doppler effects aren't a consideration. WiFi isn't a
> car trying to connect to a cell tower.  The plot doesn't show the phase
> effects but they are included as the output of the channel estimate is a
> complex frequency response. Each RX produces the h-matrix ahead of the MAC.
> These may not be symmetric in the real world but that's ok as
> transmission and reception is one way only, i.e. the treating them as
> repcripocol and the matrix as hollows symmetric isn't going to be a "test
> blocker" as the goal is to be able to use software and programmable devices
> to change them in near real time. The current approach used by many using
> butler matrices to produce off-diagonal effects  is woefully inadequate.
> And we're paying about $2.5K per each butler.
>
> Bob
>
>
>
> On Tue, Aug 10, 2021 at 9:13 AM Dick Roy  wrote:
>
> Well, I hesitate to drag this out, however Maxwell's equations and the
> invariance of the laws of physics ensure that all path loss matrices are
> reciprocal.  What that means is that at any for any given set of fixed
> boundary conditions (nothing moving/changing!), the propagation loss
> between
> any two 

Re: [Cerowrt-devel] [Starlink] [Cake] [Make-wifi-fast] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
The slides show that for WiFi every transmission produces a complex
frequency response, aka the h-matrix. This is valid for that one
transmission only.  The slides show an amplitude plot for a 3 radio device
hence the 9 elements per the h-matrix. It's assumed that the WiFi STA/AP is
stationary such that doppler effects aren't a consideration. WiFi isn't a
car trying to connect to a cell tower.  The plot doesn't show the phase
effects but they are included as the output of the channel estimate is a
complex frequency response. Each RX produces the h-matrix ahead of the MAC.
These may not be symmetric in the real world but that's ok as
transmission and reception is one way only, i.e. the treating them as
repcripocol and the matrix as hollows symmetric isn't going to be a "test
blocker" as the goal is to be able to use software and programmable devices
to change them in near real time. The current approach used by many using
butler matrices to produce off-diagonal effects  is woefully inadequate.
And we're paying about $2.5K per each butler.

Bob


On Tue, Aug 10, 2021 at 9:13 AM Dick Roy  wrote:

> Well, I hesitate to drag this out, however Maxwell's equations and the
> invariance of the laws of physics ensure that all path loss matrices are
> reciprocal.  What that means is that at any for any given set of fixed
> boundary conditions (nothing moving/changing!), the propagation loss
> between
> any two points in the domain is the same in both directions. The
> "multipathing" in one direction is the same in the other because the
> two-parameter (angle1,angle2) scattering cross sections of all objects
> (remember they are fixed here) are independent of the ordering of the
> angles.
>
> Very importantly, path loss is NOT the same as the link loss (aka link
> budget) which involves tx power and rx noise figure (and in the case of
> smart antennas, there is a link per spatial stream and how those links are
> managed/controlled really matters, but let's just keep it simple for this
> discussion) and these generally are different on both ends of a link for a
> variety of reasons. The other very important issue is that of the
> ""measurement plane", or "where tx power and rx noise figure are being
> measured/referenced to and how well the interface at that plane is
> "matched".  We generally assume that the matching is perfect, however it
> never is. All of these effects contribute to the link loss which determines
> the strength of the signal coming out of the receiver (not the receive
> antenna, the receiver) for a given signal strength coming out of the
> transmitter (not the transmit antenna, the tx output port).
>
> In the real world, things change.  Sources and sinks move as do many of the
> objects around them.  This creates a time-varying RF environment, and now
> the path loss matrix is a function of time and a few others things, so it
> matters WHEN something is transmitted, and WHEN it is received, and the two
> WHEN's are generally separated by "the speed of light" which is a ft/ns
> roughly. As important is the fact that it's no longer really a path loss
> matrix containing a single scalar because among other things, the time
> varying environment induces change in the transmitted waveform on its way
> to
> the receiver most commonly referred to as the Doppler effect which means
> there is a frequency translation/shift for each (multi-)path of which there
> are in general an uncountably infinite number because this is a continuous
> world in which we live (the space quantization experiment being conducted
> in
> the central US aside:^)). As a consequence of these physical laws, the
> entries in the path loss matrix become complex functions of a number of
> variables including time. These functions are quite often characterized in
> terms of Doppler and delay-spread, terms used to describe in just a few
> parameters the amount of "distortion" a complex function causes.
>
> Hope this helps ... probably a bit more than you really wanted to know as
> queuing theorists, but ...
>
> -Original Message-
> From: Starlink [mailto:starlink-boun...@lists.bufferbloat.net] On Behalf
> Of
> Rodney W. Grimes
> Sent: Tuesday, August 10, 2021 7:10 AM
> To: Bob McMahon
> Cc: Cake List; Make-Wifi-fast; starl...@lists.bufferbloat.net;
> co...@lists.bufferbloat.net; cerowrt-devel; bloat
> Subject: Re: [Starlink] [Cake] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2:
> Internet Quality workshop CFP for the internet architecture board
>
> > The distance matrix defines signal attenuations/loss between pairs.  It's
> > straightforward to create a distance matrix that has hidden nodes because
> > all "signal  loss" between pairs is defined.  Let's say a 120dB
> attenuation
> > path will cause a node to be hidden as an example.
> >
> >  AB CD
> > A   -   35   120   65
> > B -  65   65
> > C   -   65
> > D -
> >
> > So in the above, AC are 

Re: [Cerowrt-devel] [Starlink] Anhyone have a spare couple a hundred million ... Elon may need to start a go-fund-me page!

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
 sorry about that

The below was written two decades ago and we're still fiddling around with
fraudband. Hey, today in 2021, comcast will sell a select few 2 Gb/s
symmetric over a fiber strand using a juniper switch, leased of course,
designed in 2011. Talk about not keeping up with modern mfg of ASICs, and
associated energy efficiencies. In the meantime we continue on destroying
the planet and Musk wants our offspring to live on Mars, while Bezos thinks
he's creating a new industry in space tourism. To the writeup about why we
need to rethink broadband. Eli Noam is also quite prescient written in 1994
(http://www.columbia.edu/dlc/wp/citi/citinoam11.html )

Rather than realistic, I think you are instead being 'reasonable.' There is
a big difference. I am reminded of a quote:

"A reasonable man adapts himself to suit his environment. An unreasonable
man persists in attempting to adapt his environment to suit himself.
Therefore, all progress depends on the unreasonable man."
--George Bernard Shaw

Most CEO's, excluding start-ups, fall into the reasonable man camp (though
they were unreasonable once). Make the most of your current surroundings
while expending the minimal effort. It's good business sense for short term
results, but ignores the long term. It dictates a forward strategy of
incrementalism. It ignores the cumulative costs of the incremental
improvements and the diminishing returns of each successive increment.
Therefore each new increment has to be spaced farther out in time, which is
not desirable from a long-term business point of view. That business case
deteriorates longer term, but is easier to see shorter term. Short-term
thinking, driven by Wall Street, seems to be the only mode corporate
America can operate in.

This incrementalism mentality with 18 month upgrade cycles is fine for
consumer gadgets where the consumer knows they will have an accounting loss
on the gadget from the day they buy it. The purchaser of the gadget never
expects to make money on it. That's why they are called "consumers." That's
one of the only environments where 18 month upgrade cycles can persist.

Network infrastructure deployment is an entirely different food chain.
Under the current models, the purchaser of equipment (e.g. a service
provider) is not a consumer. It is a business that has to make a net profit
off selling services enabled by the equipment. This defies 18 month upgrade
cycles of "consumer" goods. A couple thousand bucks per subscriber takes a
long time for a network operator to recover, when you rely on a couple
percent of that, in NET income not revenue, per month. It is not conducive
to Wall Street driven companies. Thus, the next step has to be a 10-year
step.

Yet, consumers spend thousands every couple years on consumables they will
lose money on (essentially a 100% loss). Many even finance these purchases
at the ridiculous rates of credit cards, adding further to their accounting
loss. The value of these goods and services to the consumer is
justified/rationalized in non-accounting-based ways. In that light,
customer-owned networks are not such a stretch. In fact they would be an
asset that can be written off for tax purposes. The main difference is it
isn't in your physical possession, in your home, so you can't show people
your new gadget. Not directly anyway.

The "realistic" view of network infrastructure deployment (as opposed to
the reasonable view) is that today's access network infrastructure is the
wrong platform to grow from, and the wrong business model to operate under.
It can't grow like a consumer product (CD players, DVD players, PC's, etc)
because it is not a consumer product and the consumer does not have the
freedom of choice in content and applications providers (which was an
important part of the growth of those consumer markets).

Piling new money into the old infrastructure and its operating model
failure is not a realistic approach, because of diminishing returns. It was
never intended to provide real broadband connectivity, to each user, and
the operating costs are way too high. Besides, real broadband undermines
the legacy businesses of the monopoly owners.

A 100x increase in the base platform is needed in order to have a platform
that accommodates future growth in services and applications. That way it
doesn't require yet another infrastructure incremental upgrade each step of
the way. This connectivity platform also must be decoupled from the content
and services.

Access network growth cannot progress in small increments or on 18 month
upgrade cycles. It can't be small increments because these increments
enable nothing new and add little if any tangible value. They simply make
the present-day excuses for applications less annoying to use. This
approach will never make it through the next increment, and is arguably the
chasm where we sit today. It can't be 18 month cycles because the
equipment's accounting life is much longer than that. It 

Re: [Cerowrt-devel] [Starlink] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Hi All,

I do appreciate this thread as well. As a test & measurement guy here are
my conclusions around network performance. Thanks in advance for any
comments.

Congestion can be mitigated the following ways
o) Size queues properly to minimize/negate bloat (easier said than done
with tech like WiFi)
o) Use faster links on the service side such that a queues' service rates
exceeds the arrival rate, no congestion even in bursts, if possible
o) Drop entries during oversubscribed states (queue processing can't "speed
up" like water flow through a constricted pipe, must drop)
o) Identify aggressor flows per congestion if possible
o) Forwarding planes can signal back the the sources "earlier" to minimize
queue build ups per a "control loop request" asking sources to pace their
writes
o) transport layers use techniques a la BBR
o) Use "home gateways" that support tech like FQ_CODEL

Latency can be mitigated the following ways
o) Mitigate or eliminate congestion, particularly around queueing delays
o) End host apps can use TCP_NOTSENT_LOWAT along with write()/select() to
reduce host sends of "better never than late" messages
o) Move servers closer to the clients per fundamental limit of the speed of
light (i.e. propagation delay of energy over the wave guides), a la CDNs
(Except if you're a HFT, separate servers across geography and make sure to
have exclusive user rights over the lowest latency links)

Transport control loop(s)
o) Transport layer control loops are non linear systems so network tooling
will struggle to emulate "end user experience"
o) 1/2 RTT does not equal OWD used to compute the bandwidth delay product,
imbalance and effects need to be measured
o) forwarding planes signaling congestion to sources wasn't designed in TCP
originally but the industry trend seems to be to moving towards this per
things like L4S

Photons, radio & antenna design
o) Find experts who have experience & knowledge, e.g. many do here
o) Photons don't really have mass nor size, at least per my limited
understanding of particle physics and QED though, I must admit, came from
reading things on the internet

Bob

On Mon, Sep 20, 2021 at 7:40 PM Vint Cerf  wrote:

> see https://mediatrust.com/
> v
>
>
> On Mon, Sep 20, 2021 at 10:28 AM Steve Crocker  wrote:
>
>> Related but slightly different: Attached is a slide some of my colleagues
>> put together a decade ago showing the number of DNS lookups involved in
>> displaying CNN's front page.
>>
>> Steve
>>
>>
>> On Mon, Sep 20, 2021 at 8:18 AM Valdis Klētnieks 
>> wrote:
>>
>>> On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:
>>> > what actually happens during a web page load,
>>>
>>> I'm pretty sure that nobody actually understands that anymore, in any
>>> more than handwaving levels.
>>>
>>> I have a nice Chrome extension called IPvFoo that actually tracks the IP
>>> addresses contacted during the load of the displayed page. I'll let you
>>> make
>>> a guess as to how many unique IP addresses were contacted during a load
>>> of https://www.cnn.com
>>>
>>> ...
>>>
>>>
>>> ...
>>>
>>>
>>> ...
>>>
>>>
>>> 145, at least half of which appeared to be analytics.  And that's only
>>> the
>>> hosts that were contacted by my laptop for HTTP, and doesn't count DNS,
>>> or
>>> load-balancing front ends, or all the back-end boxes.  As I commented
>>> over on
>>> NANOG, we've gotten to a point similar to that of AT long distance,
>>> where 60%
>>> of the effort of connecting a long distance phone call was the cost of
>>> accounting and billing for the call.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Starlink mailing list
>>> starl...@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/starlink
>>>
>> ___
>> Starlink mailing list
>> starl...@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/starlink
>>
>
>
> --
> Please send any postal/overnight deliveries to:
> Vint Cerf
> 1435 Woodhurst Blvd
> McLean, VA 22102
> 703-448-0965
>
> until further notice
>
>
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.
--- End Message ---
___
Cerowrt-devel mailing list

Re: [Cerowrt-devel] [Starlink] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Hi Bjørn,

I find, when possible, it's preferred to take telemetry data of actual
traffic (or reads and writes) vs a proxy. We had a case where TCP BE was
outperforming TCP w/VI because BE had the most engineering resources
assigned to it and engineers did a better job with BE. Using a proxy
protocol wouldn't have exercised the same logic paths (in this case it was
in the L2 driver) as TCP did. Hence, measuring actual TCP traffic (or
socket reads and socket writes) was needed to flush out the problem. Note:
I also find that network engineers tend to focus on the stack but it's the
e2e at the application level that impacts user experience. Send side bloat
can drive the OWD while the TCP stack's RTT may look fine. For WiFi test &
measurements, we've decided most testing should be using TCP_NOSENT_LOWAT
because it helps mitigate send side bloat which WiFi engineering doesn't
focus on per lack of ability to impact.

Also, I think OWD is under tested and two way based testing can give
incomplete and inaccurate information, particularly with respect to things
like an e2e transport's control loop.  A most obvious example is assuming
1/2 RTT is the same as OWD to/fro. For WiFi this assumption is most always
false. It also false for many residential internet connections where OWD
asymmetry is designed in.

Bob


On Tue, Oct 26, 2021 at 3:04 AM Bjørn Ivar Teigen  wrote:

> Hi Bob,
>
> My name is Bjørn Ivar Teigen and I'm working on modeling and measuring
> WiFi MAC-layer protocol performance for my PhD.
>
> Is it necessary to measure the latency using the TCP stream itself? I had
> a similar problem in the past, and solved it by doing the latency
> measurements using TWAMP running alongside the TCP traffic. The requirement
> for this to work is that the TWAMP packets are placed in the same queue(s)
> as the TCP traffic, and that the impact of measurement traffic is small
> enough so as not to interfere too much with your TCP results.
> Just my two cents, hope it's helpful.
>
> Bjørn
>
> On Tue, 26 Oct 2021 at 06:32, Bob McMahon 
> wrote:
>
>> Thanks Stuart this is helpful. I'm measuring the time just before the
>> first write() (of potentially a burst of writes to achieve a burst size)
>> per a socket fd's select event occurring when TCP_NOT_SENT_LOWAT being set
>> to a small value, then sampling the RTT and CWND and providing histograms
>> for all three, all on that event. I'm not sure the correctness of RTT and
>> CWND at this sample point. This is a controlled test over 802.11ax and
>> OFDMA where the TCP acks per the WiFi clients are being scheduled by the AP
>> using 802.11ax trigger frames so the AP is affecting the end/end BDP per
>> scheduling the transmits and the acks. The AP can grow the BDP or shrink it
>> based on these scheduling decisions.  From there we're trying to maximize
>> network power (throughput/delay) for elephant flows and just latency for
>> mouse flows. (We also plan some RF frequency stuff to per OFDMA) Anyway,
>> the AP based scheduling along with aggregation and OFDMA makes WiFi
>> scheduling optimums non-obvious - at least to me - and I'm trying to
>> provide insights into how an AP is affecting end/end performance.
>>
>> The more direct approach for e2e TCP latency and network power has been
>> to measure first write() to final read() and compute the e2e delay. This
>> requires clock sync on the ends. (We're using ptp4l with GPS OCXO
>> atomic references for that but this is typically only available in some
>> labs.)
>>
>> Bob
>>
>>
>> On Mon, Oct 25, 2021 at 8:11 PM Stuart Cheshire 
>> wrote:
>>
>>> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <
>>> make-wifi-f...@lists.bufferbloat.net> wrote:
>>>
>>> > Hi All,
>>> >
>>> > Sorry for the spam. I'm trying to support a meaningful TCP message
>>> latency w/iperf 2 from the sender side w/o requiring e2e clock
>>> synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to
>>> help with this. It seems that this event goes off when the bytes are in
>>> flight vs have reached the destination network stack. If that's the case,
>>> then iperf 2 client (sender) may be able to produce the message latency by
>>> adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled
>>> RTT.
>>> >
>>> > Does this seem reasonable?
>>>
>>> I’m not 100% sure what you’re asking, but I will try to help.
>>>
>>> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your
>>> endpoint as writable (e.g., via kqueue or epoll) until less than that
>>> threshold of data remains unsent. It won’t stop you writing more bytes if
>>> you want to, up to the socket send buffer size, but it won’t *ask* you for
>>> more data until the TCP_NOTSENT_LOWAT threshold is reached. In other words,
>>> the TCP implementation attempts to keep BDP bytes in flight +
>>> TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in
>>> flight is necessary to fill the network pipe and get good throughput. The

Re: [Cerowrt-devel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks for pointing out the congestion window. Not sure why it doesn't
increase. I think that takes a stack expert ;) The run below with rx window
clamp does seem to align with linux blocking the writes.

Yes, in the previous runt the worst cases were 5.121ms which does align
with the RTT.

As a side note: I wonder if WiFi AP folks can somehow better "schedule
aggregates" based on GSO "predictions." One of the challenges for WiFi is
to align aggregates with what TCP is feeding it. I'm not sure if an
intermediary last hop AP could keep the queue size based upon the e2e
source "big tcp" so-to-speak. This is all out of my areas of expertise but
it might be nice if the two non-linear control loops, being the AP &
802.11ax first/last link hop scheduling and e2e TCP's feedback loop could
somehow plugged together in a way to help with both e2e low latency and
throughput.

Here's a run with receive side window clamping set to 1024 bytes which I
think should force CWND not to grow. In this case it does look like
linux is blocking the writes as the TCP_NOTSENT_LOWAT select waits are sub
100 microseconds so the write must have blocked.

[root@localhost iperf2-code]# src/iperf -c 192.168.1.1%eth1 --trip-times -i
1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
WARN: option of --burst-size without --burst-period defaults --burst-period
to 1 second

Client connecting to 192.168.1.1, TCP port 5001 with pid 24601 via eth1 (1
flows)
Write buffer size: 4096 Byte
Bursting: 40.0 KByte every 1.00 seconds
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 4 bytes)
Enabled select histograms bin-width=0.100 ms, bins=1

[  1] local 192.168.1.4%eth1 port 46042 connected with 192.168.1.1 port
5001 (MSS=576) (prefetch=4) (trip-times) (sock=3) (ct=5.01 ms) on
2021-10-29 13:57:22 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
Cwnd/RTTNetPwr
[  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  10/0  0
 5K/10109 us  4
[  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,40:1,50:7,51:1
(5.00/95.00/99.7%=1/51/51,Outliers=0,obl/obu=0/0) (5.015
ms/1635541042.537251)
[  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/4941 us  8
[  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.015 ms/1635541043.465805)
[  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5036 us  8
[  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.013 ms/1635541044.602288)
[  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/4956 us  8
[  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.015 ms/1635541045.465820)
[  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5121 us  8
[  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.014 ms/1635541046.664221)
[  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5029 us  8
[  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.091 ms/1635541047.466021)
[  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/4930 us  8
[  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:9,2:1
(5.00/95.00/99.7%=1/2/2,Outliers=0,obl/obu=0/0) (0.121 ms/1635541048.466058)
[  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5096 us  8
[  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.015 ms/1635541049.465821)
[  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5086 us  8
[  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.015 ms/1635541050.466051)
[  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
 5K/5112 us  8
[  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:10
(5.00/95.00/99.7%=1/1/1,Outliers=0,obl/obu=0/0) (0.015 ms/1635541051.465915)
[  1] 0.00-10.02 sec   400 KBytes   327 Kbits/sec  100/0  0
 5K/6518 us  6
[  1] 0.00-10.02 sec S8(f)-PDF:
bin(w=100us):cnt(100)=1:90,2:1,40:1,50:7,51:1
(5.00/95.00/99.7%=1/50/51,Outliers=9,obl/obu=0/0) (5.015
ms/1635541042.537251)


[root@localhost iperf2-code]# src/iperf -s -i 1 -e -B 192.168.1.1%ap0
--tcp-rx-window-clamp 1024

Server listening on TCP port 5001 with pid 22772
Binding to local address 192.168.1.1 and iface ap0
Read buffer size:  128 KByte (Dist bin width=16.0 KByte)
TCP window size:  128 KByte (default)

[  1] local 192.168.1.1%ap0 port 5001 connected with 192.168.1.4 port 46042
(MSS=1448) (clamp=1024) (burst-period=1.00s) 

Re: [Cerowrt-devel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I'm confused. I don't see any blocking nor partial writes per the write()
at the app level with TCP_NOTSENT_LOWAT set at 4 bytes. The burst is 40K,
the write size is 4K and the watermark is 4 bytes. There are ten writes per
burst.

The S8 histograms are the times waiting on the select().  The first value
is the bin number (multiplied by 100usec bin width) and second the bin
count. The worst case time is at the end and is timestamped per unix epoch.

The second run is over a controlled WiFi link where a 99.7% point of 4-8ms
for a WiFi TX op arbitration win is in the ballpark. The first is 1G wired
and is in the 600 usec range. (No media arbitration there.)

 [root@localhost iperf2-code]# src/iperf -c 10.19.87.9 --trip-times -i 1 -e
--tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
WARN: option of --burst-size without --burst-period defaults --burst-period
to 1 second

Client connecting to 10.19.87.9, TCP port 5001 with pid 2124 (1 flows)
Write buffer size: 4096 Byte
Bursting: 40.0 KByte every 1.00 seconds
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 4 bytes)
Enabled select histograms bin-width=0.100 ms, bins=1

[  1] local 10.19.87.10%eth0 port 33166 connected with 10.19.87.9 port 5001
(MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=0.54 ms) on 2021-10-26
16:07:33 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
Cwnd/RTTNetPwr
[  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0  0
14K/5368 us  8
[  1] 0.00-1.00 sec S8-PDF: bin(w=100us):cnt(10)=1:1,2:5,3:2,4:1,11:1
(5.00/95.00/99.7%=1/11/11,Outliers=0,obl/obu=0/0) (1.089
ms/1635289653.928360)
[  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/569 us  72
[  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:1,3:4,4:1,7:1,8:1
(5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.736 ms/1635289654.928088)
[  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/312 us  131
[  1] 2.00-3.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:2,5:2,6:1
(5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.548 ms/1635289655.927776)
[  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/302 us  136
[  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,2:2,3:5,6:1
(5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.584 ms/1635289656.927814)
[  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/316 us  130
[  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:2,4:2,5:2,6:1
(5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.572 ms/1635289657.927810)
[  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/253 us  162
[  1] 5.00-6.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:2,3:4,5:1
(5.00/95.00/99.7%=1/5/5,Outliers=0,obl/obu=0/0) (0.417 ms/1635289658.927630)
[  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/290 us  141
[  1] 6.00-7.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:3,4:3,6:1
(5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.573 ms/1635289659.927771)
[  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/359 us  114
[  1] 7.00-8.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,3:4,4:3,6:1
(5.00/95.00/99.7%=1/6/6,Outliers=0,obl/obu=0/0) (0.570 ms/1635289660.927753)
[  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/349 us  117
[  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,3:5,4:1,7:1
(5.00/95.00/99.7%=1/7/7,Outliers=0,obl/obu=0/0) (0.608 ms/1635289661.927843)
[  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/347 us  118
[  1] 9.00-10.00 sec S8-PDF: bin(w=100us):cnt(10)=1:3,2:1,3:5,8:1
(5.00/95.00/99.7%=1/8/8,Outliers=0,obl/obu=0/0) (0.725 ms/1635289662.927861)
[  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0  0
14K/1519 us  27
[  1] 0.00-10.01 sec S8(f)-PDF:
bin(w=100us):cnt(100)=1:25,2:13,3:36,4:11,5:5,6:5,7:2,8:2,11:1
(5.00/95.00/99.7%=1/7/11,Outliers=0,obl/obu=0/0) (1.089
ms/1635289653.928360)

[root@localhost iperf2-code]# src/iperf -c 192.168.1.1 --trip-times -i 1 -e
--tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
WARN: option of --burst-size without --burst-period defaults --burst-period
to 1 second

Client connecting to 192.168.1.1, TCP port 5001 with pid 2131 (1 flows)
Write buffer size: 4096 Byte
Bursting: 40.0 KByte every 1.00 seconds
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 4 bytes)
Enabled select histograms bin-width=0.100 ms, bins=1

[  1] local 192.168.1.4%eth1 port 45518 connected with 192.168.1.1 port
5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.48 ms) on
2021-10-26 16:07:56 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
Cwnd/RTTNetPwr
[  1] 

Re: [Cerowrt-devel] [Bloat] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
This is linux. The code flow is burst writes until the burst size, take a
timestamp, call select(), take second timestamp and insert time delta into
histogram, await clock_nanosleep() to schedule the next burst. (actually,
the deltas, inserts into the histogram and user i/o are done in another
thread, i.e. iperf 2's reporter thread.)

I still must be missing something.  Does anything else need to be set to
reduce the skb size? Everything seems to be indicating 4K writes even when
gso_max_size is 2000 (I assume these are units of bytes?) There are ten
writes, ten reads and ten  RTTs for the bursts.  I don't see partial writes
at the app level.

[root@localhost iperf2-code]# ip link set dev eth1 gso_max_size 2000
[root@localhost iperf2-code]# ip -d link sh dev eth1
9: eth1:  mtu 1500 qdisc fq_codel state
UNKNOWN mode DEFAULT group default qlen 1000
link/ether 00:90:4c:40:04:59 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 1500 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size
2000 gso_max_segs 65535
[root@localhost iperf2-code]# uname -r
5.0.9-301.fc30.x86_64


It looks like RTT is being driven by WiFi TXOPs as doubling the write size
increases the aggregation by two but has no significant effect on the RTTs.

4K writes: tot_mpdus 328 tot_ampdus 209 mpduperampdu 2


8k writes:  tot_mpdus 317 tot_ampdus 107 mpduperampdu 3


[root@localhost iperf2-code]# src/iperf -c 192.168.1.1%eth1 --trip-times -i
1 -e --tcp-write-prefetch 4 -l 4K --burst-size=40K --histograms
WARN: option of --burst-size without --burst-period defaults --burst-period
to 1 second

Client connecting to 192.168.1.1, TCP port 5001 with pid 5145 via eth1 (1
flows)
Write buffer size: 4096 Byte
Bursting: 40.0 KByte every 1.00 seconds
TCP window size: 85.0 KByte (default)
Event based writes (pending queue watermark at 4 bytes)
Enabled select histograms bin-width=0.100 ms, bins=1

[  1] local 192.168.1.4%eth1 port 45680 connected with 192.168.1.1 port
5001 (MSS=1448) (prefetch=4) (trip-times) (sock=3) (ct=5.30 ms) on
2021-10-26 20:25:29 (PDT)
[ ID] IntervalTransferBandwidth   Write/Err  Rtry
Cwnd/RTTNetPwr
[  1] 0.00-1.00 sec  40.1 KBytes   329 Kbits/sec  11/0  0
14K/10091 us  4
[  1] 0.00-1.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:1,36:1,40:1,44:1,46:1,48:1,49:1,50:2,52:1
(5.00/95.00/99.7%=1/52/52,Outliers=0,obl/obu=0/0) (5.121
ms/1635305129.152339)
[  1] 1.00-2.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/4990 us  8
[  1] 1.00-2.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,39:1,45:1,49:5,50:1
(5.00/95.00/99.7%=1/50/50,Outliers=0,obl/obu=0/0) (4.991
ms/1635305130.153330)
[  1] 2.00-3.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/4904 us  8
[  1] 2.00-3.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:2,29:1,49:4,50:1,59:1,75:1
(5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.455
ms/1635305131.147353)
[  1] 3.00-4.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/4964 us  8
[  1] 3.00-4.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:4,50:2,59:1,65:1
(5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.460
ms/1635305132.146338)
[  1] 4.00-5.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/4970 us  8
[  1] 4.00-5.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,49:6,59:1,65:1
(5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.404
ms/1635305133.146335)
[  1] 5.00-6.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/4986 us  8
[  1] 5.00-6.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:2,48:1,49:1,50:4,59:1,64:1
(5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.395
ms/1635305134.146343)
[  1] 6.00-7.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/5059 us  8
[  1] 6.00-7.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:2,39:1,49:3,50:2,60:1,85:1
(5.00/95.00/99.7%=1/85/85,Outliers=0,obl/obu=0/0) (8.417
ms/1635305135.148343)
[  1] 7.00-8.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/5407 us  8
[  1] 7.00-8.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:2,40:1,49:4,50:1,59:1,75:1
(5.00/95.00/99.7%=1/75/75,Outliers=0,obl/obu=0/0) (7.428
ms/1635305136.147343)
[  1] 8.00-9.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/5188 us  8
[  1] 8.00-9.00 sec S8-PDF: bin(w=100us):cnt(10)=1:2,40:1,49:3,50:3,64:1
(5.00/95.00/99.7%=1/64/64,Outliers=0,obl/obu=0/0) (6.388
ms/1635305137.146284)
[  1] 9.00-10.00 sec  40.0 KBytes   328 Kbits/sec  10/0  0
14K/5306 us  8
[  1] 9.00-10.00 sec S8-PDF:
bin(w=100us):cnt(10)=1:2,39:1,49:2,50:2,51:1,60:1,65:1
(5.00/95.00/99.7%=1/65/65,Outliers=0,obl/obu=0/0) (6.422
ms/1635305138.146316)
[  1] 0.00-10.01 sec   400 KBytes   327 Kbits/sec  102/0  0
14K/5939 us  7
[  1] 0.00-10.01 sec S8(f)-PDF:
bin(w=100us):cnt(100)=1:19,29:1,36:1,39:3,40:3,44:1,45:1,46:1,48:2,49:33,50:18,51:1,52:1,59:5,60:2,64:2,65:3,75:2,85:1
(5.00/95.00/99.7%=1/65/85,Outliers=0,obl/obu=0/0) (8.417
ms/1635305135.148343)

[root@localhost 

Re: [Cerowrt-devel] [Make-wifi-fast] TCP_NOTSENT_LOWAT applied to e2e TCP msg latency

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks Stuart this is helpful. I'm measuring the time just before the first
write() (of potentially a burst of writes to achieve a burst size) per a
socket fd's select event occurring when TCP_NOT_SENT_LOWAT being set to a
small value, then sampling the RTT and CWND and providing histograms for
all three, all on that event. I'm not sure the correctness of RTT and CWND
at this sample point. This is a controlled test over 802.11ax and OFDMA
where the TCP acks per the WiFi clients are being scheduled by the AP using
802.11ax trigger frames so the AP is affecting the end/end BDP per
scheduling the transmits and the acks. The AP can grow the BDP or shrink it
based on these scheduling decisions.  From there we're trying to maximize
network power (throughput/delay) for elephant flows and just latency for
mouse flows. (We also plan some RF frequency stuff to per OFDMA) Anyway,
the AP based scheduling along with aggregation and OFDMA makes WiFi
scheduling optimums non-obvious - at least to me - and I'm trying to
provide insights into how an AP is affecting end/end performance.

The more direct approach for e2e TCP latency and network power has been to
measure first write() to final read() and compute the e2e delay. This
requires clock sync on the ends. (We're using ptp4l with GPS OCXO
atomic references for that but this is typically only available in some
labs.)

Bob


On Mon, Oct 25, 2021 at 8:11 PM Stuart Cheshire  wrote:

> On 21 Oct 2021, at 17:51, Bob McMahon via Make-wifi-fast <
> make-wifi-f...@lists.bufferbloat.net> wrote:
>
> > Hi All,
> >
> > Sorry for the spam. I'm trying to support a meaningful TCP message
> latency w/iperf 2 from the sender side w/o requiring e2e clock
> synchronization. I thought I'd try to use the TCP_NOTSENT_LOWAT event to
> help with this. It seems that this event goes off when the bytes are in
> flight vs have reached the destination network stack. If that's the case,
> then iperf 2 client (sender) may be able to produce the message latency by
> adding the drain time (write start to TCP_NOTSENT_LOWAT) and the sampled
> RTT.
> >
> > Does this seem reasonable?
>
> I’m not 100% sure what you’re asking, but I will try to help.
>
> When you set TCP_NOTSENT_LOWAT, the TCP implementation won’t report your
> endpoint as writable (e.g., via kqueue or epoll) until less than that
> threshold of data remains unsent. It won’t stop you writing more bytes if
> you want to, up to the socket send buffer size, but it won’t *ask* you for
> more data until the TCP_NOTSENT_LOWAT threshold is reached. In other words,
> the TCP implementation attempts to keep BDP bytes in flight +
> TCP_NOTSENT_LOWAT bytes buffered and ready to go. The BDP of bytes in
> flight is necessary to fill the network pipe and get good throughput. The
> TCP_NOTSENT_LOWAT of bytes buffered and ready to go is provided to give the
> source software some advance notice that the TCP implementation will soon
> be looking for more bytes to send, so that the buffer doesn’t run dry,
> thereby lowering throughput. (The old SO_SNDBUF option conflates both
> “bytes in flight” and “bytes buffered and ready to go” into the same
> number.)
>
> If you wait for the TCP_NOTSENT_LOWAT notification, write a chunk of n
> bytes of data, and then wait for the next TCP_NOTSENT_LOWAT notification,
> that will tell you roughly how long it took n bytes to depart the machine.
> You won’t know why, though. The bytes could depart the machine in response
> for acks indicating that the same number of bytes have been accepted at the
> receiver. But the bytes can also depart the machine because CWND is
> growing. Of course, both of those things are usually happening at the same
> time.
>
> How to use TCP_NOTSENT_LOWAT is explained in this video:
>
> 
>
> Later in the same video is a two-minute demo (time offset 42:00 to time
> offset 44:00) showing a “before and after” demo illustrating the dramatic
> difference this makes for screen sharing responsiveness.
>
> 
>
> Stuart Cheshire

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.
--- End Message ---

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Just an FYI,

iperf 2 uses a 4 usec delay for TCP and 100 usec delay for UDP to fill the
token bucket. We thought about providing a knob for this but decided not
to. We figured a busy wait CPU thread wasn't a big deal because of the
trend of many CPU cores. The threaded design works well for this. We also
support fq-pacing and isochronous traffic using clock_nanosleep() to
schedule the writes. We'll probably add Markov chain support but that's not
critical and may not affect actionable engineering. We found isoch as a
useful traffic profile, at least for our WiFi testing. I'm going to add
support for TCP_NOTSENT_LOWAT for select()/write() based transmissions. I'm
doubtful this is very useful as event based scheduling based on times seems
better. We'll probably use it for unit testing WiFi aggregation and see if
it helps there or not. I'll see if it aligns with the OWD measurements.

On queue depth, we use two techniques. The most obvious is to measure the
end to end delay and use rx histograms, getting all the samples without
averaging. The second, internal for us only, is using network telemetry and
mapping all the clock domains to the GPS domain. Any moment in time the
end/end path can be inspected to where every packet is.

Our automated testing is focused around unit tests and used to
statistically monitor code changes (which come at a high rate and apply to
a broad range of chips) - so the requirements can be very different from a
network or service provider.

Agreed that the amount of knobs and reactive components are a challenge.
And one must assume non-linearity which becomes obvious after a few direct
measurements (i.e. no averaging.) The challenge of statistical;y
reproducible is always there. We find Montec Carlo techniques can be useful
only when they are proven to be statistically reproducible.

Bob


On Sat, Jul 17, 2021 at 4:29 PM Aaron Wood  wrote:

> On Mon, Jul 12, 2021 at 1:32 PM Ben Greear 
> wrote:
>
>> UDP is better for getting actual packet latency, for sure.  TCP is
>> typical-user-experience-latency though,
>> so it is also useful.
>>
>> I'm interested in the test and visualization side of this.  If there were
>> a way to give engineers
>> a good real-time look at a complex real-world network, then they have
>> something to go on while trying
>> to tune various knobs in their network to improve it.
>>
>
> I've always liked the smoke-ping visualization, although a single graph is
> only really useful for a single pair of endpoints (or a single segment,
> maybe).  But I can see using a repeated set of graphs (Tufte has some
> examples), that can represent an overview of pairwise collections of
> latency+loss:
> https://www.edwardtufte.com/bboard/images/0003Cs-8047.GIF
> https://www.edwardtufte.com/tufte/psysvcs_p2
>
> These work for understanding because the tiled graphs are all identically
> constructed, and the reader first learns how to read a single tile, and
> then learns the pattern of which tiles represent which measurements.
>
> Further, they are opinionated.  In the second link above, the y axis is
> not based on the measured data, but standardized expected values, which (I
> think) is key to quick readability.  You never need to read the axes.  Much
> like setting up gauges such that "nominal" is always at the same indicator
> position for all graphs (e.g. straight up).  At a glance, you can see if
> things are "correct" or not.
>
> That tiling arrangement wouldn't be great for showing interrelationships
> (although it may give you a good historical view of correlated behavior).
> One thought is to overlay a network graph diagram (graph of all network
> links) with small "sparkline" type graphs.
>
> For a more physical-based network graph, I could see visualizing the queue
> depth for each egress port (max value over a time of X, or percentage of
> time at max depth).
>
> Taken together, the timewise correlation could be useful (which peers are
> having problems communicating, and which ports between them are impacted?).
>
> I think getting good data about queue depth may be the hard part,
> especially catching transients and the duty cycle / pulse-width of the load
> (and then converting that to a number).  Back when I uncovered the iperf
> application-level pacing granularity was too high 5 years ago, I called it
> them "millibursts", and maybe dtaht pointed out that link utilization is
> always 0% or 100%, and it's just a matter of the PWM of the packet rate
> that makes it look like something in between.
> https://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html
>
>
>
> I'll let others try to figure out how build and tune the knobs, but the
>> data acquisition and
>> visualization is something we might try to accomplish.  I have a feeling
>> I'm not the
>> first person to think of this, howeverprobably someone already has
>> done such
>> a thing.
>>
>> Thanks,
>> Ben
>>
>> On 7/12/21 1:04 PM, Bob McMahon wrote:
>> > I believe end 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks for this. I plan to purchase the second volume to go with my copy of
volume 1. There is (always) more to learn and your expertise is very
helpful.

Bob

PS.  As a side note, I've added support for TCP_NOTSENT_LOWAT in iperf 2.1.4
 and it's proving
interesting per WiFi/BT latency testing including helping to mitigate
sender side bloat.
*--tcp-write-prefetch **n*[kmKM]Set TCP_NOTSENT_LOWAT on the socket and use
event based writes per select() on the socket.I'll probably add measuring
the select() delays to see if that correlates to things like RF
arbitrations, etc.


On Wed, Jul 21, 2021 at 4:20 PM Leonard Kleinrock  wrote:

> Just a few comments following David Reed's insightful comments re the
> history of the ARPANET and its approach to flow control.  I have attached
> some pages from my Volume II which provide an understanding of how we
> addressed flow control and its implementation in the ARPANET.
>
> The early days of the ARPANET design and evaluation involved detailed
> design of what we did call “Flow Control”.  In my "Queueing Systems, Volume
> II: Computer Applications”, John Wiley, 1976, I documented much of what we
> designed and evaluated for the ARPANET, and focused on performance,
> deadlocks, lockups and degradations due to flow control design.  Aspects of
> congestion control were considered, but this 2-volume book was mostly about
> understanding congestion.Of interest are the many deadlocks that we
> discovered in those early days as we evaluated and measured the network
> behavior.  Flow control was designed into that early network, but it had a
> certain ad-hoc flavor and I point out the danger of requiring flows to
> depend upon the acquisition of multiple tokens that were allocated from
> different portions of the network at the same time in a distributed
> fashion.  The attached relevant sections of the book address these issues;
>  I thought it would be of value to see what we were looking at back then.
>
> On a related topic regarding flow and congestion control (as triggered by
> David’s comment* "**at most one packet waiting for each egress link in
> the bottleneck path.”*), in 1978, I published a paper
> 
>  in
> which I extended the notion of Power (the ratio of throughput to response
> time) that had been introduced by Giessler, et a
> l
> and I pointed out the amazing properties that emerged when Power is
> optimized, e.g., that one should keep each hop in the pipe “just full”,
> i.e., one message per hop.  As it turns out, and as has been discussed in
> this email chain, Jaffe
>  showed
> in 1981 that this optimization was not decentralizable and so no one
> pursued this optimal operating point (notwithstanding the fact that I
> published other papers on this issue, for example in 1979
> 
>  and
> in 1981 ).  So this
> issue of Power lay dormant for decades until Van Jacobsen, et al,
> resurrected the idea with their BBR flow control design in 2016
>  when they showed that
> indeed one could decentralize power.  Considerable research has since
> followed their paper including another by me in 2018
> .
> (This was not the first time that a publication challenging the merits of a
> new idea negatively impacted that idea for decades - for example, the 1988
> book “Perceptrons”
> 
>  by
> Minsky and Papert discouraged research into neural networks for many years
> until that idea was proven to have merit.)  But the story is not over as
> much  work has yet to be done to develop the algorithms that can properly
> deal with congestion in the sense that this email chain continues to
> discuss it.
>
> Best,
> Len
>
>
>
>
>
>
>
> On Jul 13, 2021, at 10:49 AM, David P. Reed  wrote:
>
> Bob -
>
> On Tuesday, July 13, 2021 1:07pm, "Bob McMahon" 
> said:
>
> "Control at endpoints benefits greatly from even small amounts of
> information supplied by the network about the degree of congestion present
> on the path."
>
> Agreed. The ECN mechanism seems like a shared thermostat in a building.
> It's basically an on/off where everyone is trying to set the temperature.
> It does affect, in a non-linear manner, but still an effect. Better than a
> thermostat set at infinity or 0 Kelvin 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Ok, adding support for TCP_WINDOW_CLAMP and TCP_NOTSENT_LOWAT into iperf 2
seems useful for TCP WiFi latency related testing.  These option names are
quite obfuscated. I can't see many but some ultimate networking geeks
knowing what these actually do.

Here are some proposed command/option names which wouldn't pass any "parser
police" (Parser police was the internal discussion list we used at Cisco to
review router commands. The Cisco cli is a disaster even with pp, but less
so than what could have been.)

{"*tcp-rx-window-clamp*", required_argument, , 1},
{"*tcp-not-sent-low-watermark*", required_argument, , 1},

I'd for sure like to rename "tcp-not-sent-low-watermark" to something more
intuitive. (My daughter, trained in linguistics, is having a field day
laughing at this "nerd language" that is beyond human comprehension.) This
cli option sets the socket option and causes the use of select for writes
(vs a write spin loop.)

Thanks in advance for any suggestions here,
Bob

On Wed, Jul 14, 2021 at 6:27 PM Holland, Jake  wrote:

> From: Bob McMahon via Bloat 
> > Date: Wed,2021-07-14 at 11:38 AM
> > One challenge I faced with iperf 2 was around flow control's effects on
> > latency. I find if iperf 2 rate limits on writes then the end/end
> > latencies, RTT look good because the pipe is basically empty, while rate
> > limiting reads to the same value fills the window and drives the RTT up.
> > One might conclude, from a network perspective, the write side is
> > better.  But in reality, the write rate limiting is just pushing the
> > delay into the application's logic, i.e. the relevant bytes may not be
> > in the pipe but they aren't at the receiver either, they're stuck
> > somewhere in the "tx application space."
> >
> > It wasn't obvious to me how to address this. We added burst measurements
> > (burst xfer time, and bursts/sec) which, I think, helps.
> ...
> >>> I find the assumption that congestion occurs "in network" as not always
> >>> true. Taking OWD measurements with read side rate limiting suggests
> that
> >>> equally important to mitigating bufferbloat driven latency using
> congestion
> >>> signals is to make sure apps read "fast enough" whatever that means. I
> >>> rarely hear about how important it is for apps to prioritize reads over
> >>> open sockets. Not sure why that's overlooked and bufferbloat gets all
> the
> >>> attention. I'm probably missing something.
>
> Hi Bob,
>
> You're right that the sender generally also has to avoid sending
> more than the receiver can handle to avoid delays in a message-
> reply cycle on the same TCP flow.
>
> In general, I think of failures here as application faults rather
> than network faults.  While important for user experience, it's
> something that an app developer can solve.  That's importantly
> different from network buffering.
>
> It's also somewhat possible to avoid getting excessively backed up
> in the network because of your own traffic.  Here bbr usually does
> a decent job of keeping the queues decently low.  (And you'll maybe
> find that some of the bufferbloat measurement efforts are relying
> on the self-congestion you get out of cubic, so if you switch them
> to bbr you might not get a good answer on how big the network buffers
> are.)
>
> In general, anything along these lines has to give backpressure to
> the sender somehow.  What I'm guessing you saw when you did receiver-
> side rate limiting was that the backpressure had to fill bytes all
> the way back to a full receive kernel buffer (making a 0 rwnd for
> TCP) and a full send kernel buffer before the send writes start
> failing (I think with ENOBUFS iirc?), and that's the first hint the
> sender has that it can't send more data right now.  The assumption
> that the receiver can receive as fast as the sender can send is so
> common that it often goes unstated.
>
> (If you love to suffer, you can maybe get the backpressure to start
> earlier, and with maybe a lower impact to your app-level RTT, if
> you try hard enough from the receive side with TCP_WINDOW_CLAMP:
> https://man7.org/linux/man-pages/man7/tcp.7.html#:~:text=tcp_window_clamp
> But you'll still be living with a full send buffer ahead of the
> message-response.)
>
> But usually the right thing to do if you want receiver-driven rate
> control is to send back some kind of explicit "slow down, it's too
> fast for me" feedback at the app layer that will make the sender send
> slower.  For instance most ABR players will shift down their bitrate
> if they're failing to render video fast enough just as well as if the
> network isn't feeding the video segments fast enough, like if they're
> CPU-bound from something else churning on the machine.  (RTP-based
> video players are supposed to send feedback with this same kind of
> "slow down" capability, and sometimes they do.)
>
> But what you can't fix from the endpoints no matter how hard you
> try is the buffers in the network that get filled by other