> On May 17, 2019, at 09:36, Dave Taht <[email protected]> wrote:
>
> Jason is on the bloat list, nick feamster used to be, but isn't.
> Sometimes I wish I could drag in various folk at the FCC, FTC, or some
> other regulatory body more often. BITAG. Places like that. Having some
> co-ordinated "Here are our comments on your thing" might be better
> than the general chaos on the bloat list, and, since google is so
> ineffective at indexing mailing lists these days some way to raise it
> in the search results.
>
> On Fri, May 17, 2019 at 12:01 AM Jonathan Foulkes
> <[email protected]> wrote:
>>
>> Thanks for sharing Dave.
>>
>> A good paper, but there are few gaps worthy of mentioning on this list:
>>
>> Testing when there is an AQM present means the test must adapt to the
>> challenge of smaller cwnd existing for any one stream, therefore it will
>> take many more streams to saturate a line with cwnd = 30 than if the cwnd is
>> allowed to grow to >1,000
>> In general, the impact of cwnd relative to saturation and impact on delay
>> was not visited, and yet it’s critical. One of the reasons for spiky delays
>> on high speed lines is the ginormous cwnds hogging the line with their
>> 800ms+ RTT’s
Good point, I note that the detailed dslreports results also contain
information about the average cwnd, and I admit that so far I have never looked
at those numbers; will do in the future
>>
>> Asymmetry of provisioned upload relative to download, at some point, the
>> ack-stream can be held up by either lack of capacity or bloat in the uplink.
>> So even though a link can deliver 300Mbps down, a bloated uplink of 5mbps
>> might never allow that level to be reached.
IIRC a back of the envelop estimate assuming MTU~1500 and one ACK every two
full MSS packets gives the following percentage of ACK traffic for a given
amount of reverse traffic (I assume a cable link here for overhead, but that
ratio does not depend too much on the actual per-packet-overhead):
100 * ((18+(20+20+12))/(1500+18)) = 4.61% ~. 1/20 ~ 5%
Techniques like ACK filtering/thinning and GSO/GRO can reduce this number
considerably. So the 300 Mbps link needs to have at least 300*0.05 = 15 Mbps
uplink to allow saturation with bog-standard TCP-Reno flows. Most asymmetries I
have seen in the wild seem to keep this in mind, but I would not be amazed to
see something like 300/5 in the wild ;)
>> There are ISPs provisioning truly crazy asymmetric service.
+1; or rather -1 as while this is true I do not want to endorse that
>>
>> They do make a good point about the local network, WiFi specifically being
>> the new bottleneck, which is why we included an iperf instance that can be
>> started on the IQrouter to help run client to server tests that help spot
>> local network capacity limits, typically on WiFi.
Enlightened as so often. I note the official German speedtest will ask
for lan/wlan connection and will only consider tests via lan as oficial (their
desktop app even will not run over wifi at all, which I find obnoxious).
>>
>> Regarding their point about ‘Cross traffic’ impact on measurements, Cake’s
>> per-host / per-target fairness also complicates AQM-enabled testing from
>> client devices. Which is why we make the built-in speed test the arbiter of
>> true line capacity, as it factors for ALL traffic flowing through the
>> router.
+1. There is a toy project going on in the openwrt forum trying to
collect traffic, CPU load & frequency, and RTT data for concurrently running
saturating loads; the idea being the same as yours, measuring the
ingress/egress traffic directly at the router and just using the speedtest(s)
as a load generator(s). But that project has not gone very far. (It statred
from the question what shaper performance one can expect from different SoCs)
>> But, as you mention, that is also a challenge from a CPU resource standpoint
>> on higher speeds.
Yes this is why monitoring the CPU load during the test seems like a
great idea.
>>
>> The biggest gap in this paper is not paying sufficient attention to latency
>> as a critical metric, and one that is controllable by an AQM. Bufferbloat
>> metrics have more impact on end-user experience than +/- 50Mbps on a 100mbps
>> baseline.
>> I was rather miffed they do not even mention the DSLreports.com speedtest,
>> or the fast.com test, as those are the two that provide a bufferbloat metric.
I might add that https://www.thinkbroadband.com/speedtest also does a
few things well, a) they do run a single stream and a 6 stream test in sequence
and report the numbers for both and b) they also do a latency-under-load test
(click the Analysis button in the first results graph) . Unfortunately the
latency-under-load grpahs is inaccessible from the main and final results
page...
>
> I keep hoping government will get it.
>
> I note that I'm in general quite pleased with the overall trendlines
> on dslreports (
> http://www.dslreports.com/speedtest/results/bufferbloat?up=1 ). A lot
> of the bloat on cablemodems disappearing is due to uplink speeds
> getting much better, and holding buffer sizes constant, of course,
> rather than the docsis 3.1 deployment or our efforts, but I do tend to
> look at the long fuzzy blue data on the left of the green 30ms line
> and think that's mostly us.
>
> The thing that scares though, is that since that test is exaustively
> used by those tuning their links with fq_codel, is that that dataset
> is now quite polluted. So what does reality actually look like? On
> this trip I encountered network after network essentially in a state
> of congestion collapse.
>
> The other big problems with dslreports is that the test does not run
> for long enough at this higher rates,
After a free registration, one can configure the test duration (as can
be done for fast.com) in
https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803
we collect recommendations how to configure the test for better bufferbloat
assessment icludingsetting the duration to 30 seconds (which at a point in the
past was the maximum the test allowed, but I believe the limit was increased
since then; but since dslreports has to pay for their bandwidth I believe
recommending something shortish is the right thing to do, and in my expereience
30 seconds is already much better than the typical ~10 second durations for
other speedttests).
> and that, without seeing the raw
> data (just the inverse of what's being displayed, what dslreports is
> throwing out), I worry. It's a cosmic background bufferbloat detector
> that it's needed. Still dsl sucks, still there's data at +1 second
> (With a cutoff of 4 sec that is an artifact of the test), those really
> worry me.
>
>> The industry as whole MUST pay attention and socialize the relevancy of
>> managed latencies as being critical to customer satisfaction and good
>> application performance. And that starts with tests that clearly grade that
>> critical aspect.
>
> One can wish. I think only the bufferbloat issue breaking out
> thoroughly into the business press, in places like the economist or
> cto or fortune, will make a difference here.
In the EU getting the european regulators on board and convinced that
bofferbloat is important and to be avoided would help a lot; as they are
responsible for the reference official speedtests in europe.
Best Regards
Sebastian
>
>> Cheers,
>>
>> Jonathan
>>
>>> On May 15, 2019, at 3:58 AM, Dave Taht <[email protected]> wrote:
>>>
>>> If it helps any: Nick Feamster and Jason Livingood just published "
>>> Internet Speed Measurement: Current Challenges and Future
>>> Recommendations " ( https://arxiv.org/pdf/1905.02334.pdf ) a few days
>>> ago, and outlines quite a few problems going forward at higher speeds.
>>> I do wish the document had pointed out more clearly that router based
>>> measurements have problems also, with weaker cpus unable to source
>>> enough traffic for an accurate measurement, but I do hope this
>>> document has impact, and it's a good read, regardless.
>>>
>>> Still, somehow getting it right at lower speeds is always on my mind.
>>> I'd long ago hoped that DSL devices would adopt BQL, and that
>>> cablemodems would also, thus moving packet processing a little higher
>>> on the stack so more advanced algorithms like cake could take hold.
>>>
>>> On Wed, May 15, 2019 at 9:32 AM Sebastian Moeller <[email protected]> wrote:
>>>>
>>>> Hi All,
>>>>
>>>>
>>>> I believe the following to be relevant to this discussion:
>>>> https://apenwarr.ca/log/20180808
>>>> Where he discusses a similar idea including implementation albeit aimed at
>>>> lower bandwidth and sans the automatic bandwidth tracking.
>>>>
>>>>
>>>>> On May 15, 2019, at 01:34, David P. Reed <[email protected]> wrote:
>>>>>
>>>>>
>>>>> Ideally, it would need to be self-configuring, though... I.e., something
>>>>> like the IQRouter auto-measuring of the upstream bandwidth to tune the
>>>>> shaper.
>>>>
>>>> @Jonathan from your experience how tricky is it to get reliable speedtest
>>>> endpoints and how reliable are they in practice. And do you do any
>>>> sanitization, like take another measure immediate if the measured rate
>>>> differs from the last by more than XX% or something like that?
>>>>
>>>>
>>>>>
>>>>> Sure, seems like this is easy to code because there are exactly two ports
>>>>> to measure, they can even be labeled physically "up" and "down" to
>>>>> indicate their function.
>>>>
>>>> IMHO the real challenge is automated measurements over the internet at
>>>> Gbps speeds. It is not hard to get some test going (by e.g. tapping into
>>>> ookla's fast net of confederated measurement endpoints) but getting
>>>> something where the servers can reliably saturate 1Gbps+ seems somewhat
>>>> trickier (last time I looked one required a 1Gbps connection to the server
>>>> to participate in speedtest.net, obviously not really suited for measuring
>>>> Gbps speeds).
>>>> In the EU there exists a mandate for national regulators to establish
>>>> and/or endorse an anointed "official" speedtests, untended to keep ISP
>>>> marketing honest, that come with stricter guarantees (e.g. the official
>>>> German speedtest, breitbandmessung.de will only admit tests if the servers
>>>> are having sufficient bandwidth reserves to actually saturate the link;
>>>> the enduser is required to select the speed-tier giving them a strong hint
>>>> about the required rates I believe).
>>>> For my back-burner toy project "per-packet-overhead estimation on
>>>> arbitrary link technology" I am currently facing the same problem, I need
>>>> a traffic sink and source that can reliably saturate my link so I can
>>>> measure maximum achievable goodput, so if anybody in the list has ideas, I
>>>> am all ears/eyes.
>>>>
>>>>>
>>>>> For reference, the GL.iNet routers are tiny and nicely packaged, and run
>>>>> OpenWrt; they do have one with Gbit ports[0], priced around $70. I very
>>>>> much doubt it can actually push a gigabit, though, but I haven't had a
>>>>> chance to test it. However, losing the WiFi, and getting a slightly
>>>>> beefier SoC in there will probably be doable without the price going
>>>>> over $100, no?
>>>>>
>>>>> I assume the WiFi silicon is probably the most costly piece of
>>>>> intellectual property in the system. So yeah. Maybe with the right parts
>>>>> being available, one could aim at $50 or less, without sales channel
>>>>> markup. (Raspberry Pi ARM64 boards don't have GigE, and I think that
>>>>> might be because the GigE interfaces are a bit pricey. However, the ARM64
>>>>> SoC's available are typically Celeron-class multicore systems. I don't
>>>>> know why there aren't more ARM64 systems on a chip with dual GigE, but I
>>>>> suspect searching for them would turn up some).
>>>>
>>>> The turris MOX (https://www.turris.cz/en/specification/) might be a decent
>>>> startimg point as it comes with one Gbethernet port and both a SGMII and a
>>>> PCIe signals routed on a connector, they also have a 4 and an 8 port
>>>> switch module, but for our purposes it might be possible to just create a
>>>> small single Gb ethernet port board to get started.
>>>>
>>>> Best Regards
>>>> Sebastian
>>>>
>>>>>
>>>>> -Toke
>>>>>
>>>>> [0] https://www.gl-inet.com/products/gl-ar750s/
>>>>> _______________________________________________
>>>>> Cerowrt-devel mailing list
>>>>> [email protected]
>>>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>>>
>>>> _______________________________________________
>>>> Bloat mailing list
>>>> [email protected]
>>>> https://lists.bufferbloat.net/listinfo/bloat
>>>
>>>
>>>
>>> --
>>>
>>> Dave Täht
>>> CTO, TekLibre, LLC
>>> http://www.teklibre.com
>>> Tel: 1-831-205-9740
>>> _______________________________________________
>>> Bloat mailing list
>>> [email protected]
>>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
> --
>
> Dave Täht
> CTO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-831-205-9740
_______________________________________________
Bloat mailing list
[email protected]
https://lists.bufferbloat.net/listinfo/bloat
