Hi David,
On Jul 26, 2014, at 01:26 , David Lang <[email protected]> wrote: > But I think that what we are seeing from teh results of the bufferbloat work > is that a properly configured network doesn't degrade badly as it gets busy. > > Individual services will degrade as they need more bandwidth than is > available, but that sort of degredation is easy for the user to understand. > > The current status-quo is where good throughput at 80% utilization may be > 80Mb, at 90% utilization it may be 85Mb, at 95% utilization it is 60Mb, and > at 100% utilization it pulses between 10Mb and 80Mb averaging around 20Mb and > latency goes from 10ms to multiple seconds over this range. > > With BQL and fw_codel, 80% utilization would still be 80Mb, 90% utilization > would be 89Mb, 95% utilization would be 93Mb with latency only going to 20ms > > so there is a real problem to solve in the current status-quo, and the > question is if there is a way to quantify the problem and test for it in ways > that are repeatable, meaningful and understandable. > > This is a place to avoid letting perfect be the enemy of good enough. > > If you ask even relatively technical people about the quality of a network > connection, they will talk to you about bandwidth and latency. > > But if you talk to a networking expert, they don't even mention that, they > talk about signal strength, waveform distortion, bit error rates, error > correction mechanisms, signal regeneration, and probably many other things > that I don't know enough to even mention :-) > > > Everyone is already measuring peak bandwidth today, and that is always going > to be an important factor, so it will stay around. > > So we need to show the degredation of the network, and I think that either > ping(loaded)-ping(unloaded) or ping(loaded)/ping(unloaded) will give us > meaningful numbers that people can understand and talk about, while still > being meaningful in the real world. Maybe we should follow Neil and Martin’s lead and consider either ping(unloaded)-ping(loaded) or ping(unloaded)/ping(loaded) and call the whole thing quality estimator or factor (as negative quality or a factor < 0 intuitively shows a degradation). Also my bet is on the difference not on the ratio, why should people with bad latency to begin with (satellite?) be more tolerant to further degradation? I would assume that on a high latency link if at all the “budget” for further degradation might be smaller than on a low latency link (reasoning: there might be a fixed latency budget for acceptable latency for voip). > > which of the two is more useful is something that we would need to get a > bunch of people with different speed lines to report and see which is > affected less by line differences and distance to target. Or make sure we always measure against the closest target (which with satellite might still be far away)? Best Regards sebastian > > David Lang > > On Fri, 25 Jul 2014, [email protected] wrote: > >> I think what is being discussed is "how to measure the quality of one >> endpoint's experience of the entire Internet over all time or over a >> specific interval of time". >> Yet the systems that are built on top of the Internet transport do not have >> any kind of uniform dependence on the underlying transport behavior in terms >> of their quality. Even something like VoIP's quality as experienced by two >> humans talking over it has a dependency on the Internet's behavior, but one >> that is hardly simple. >> As an extreme, if one endpoint experiences a direct DDoS attack or is >> indirectly affected by one somewhere in the path, the quality of the >> experience might be dramatically reduced. >> So any attempt to define a delta-Q that has meaning in terms of user >> experience appears pointless and even silly - the endpoint experience is >> adequate under a very wide variety of conditions, but degrades terribly >> under certain kinds of conditions. >> As a different point, let's assume that the last-mile is 80% utilized, but >> the latency variation in that utilization is not larger than 50 msec. This >> is a feasible-to-imagine operating point, but it requires a certain degreee >> of tight control that may be very hard to achieve over thousands of >> independent application services through that point, so its feasibility is >> contingent on lots of factors. Then if the 20% capacity is far larger than >> 64 kb/sec we know that toll-quality audio can be produced with a small >> endpoint "jitter buffer". There's no "delta-Q" there at all - quality is >> great. >> So the point is: a single number or even a single "morphism" (whatever that >> is) to a specific algebraic domain element (a mapping to a semi-lattice with >> non-Abelian operators?) does not allow one to define a "measure" of an >> endpoint of the Internet that can be used to compute "quality" of all >> applications. >> Or in purely non-abstract terms: if there were a delta-Q it would be useless >> for most network applications, but might be useful for a single network >> application. >> So I submit that delta-Q is a *metaphor* and not a very useful one at that. >> It's probably as useful as providing a "funkiness" measure for an Internat >> access point. We can certainly talk about and make claims about the >> relative "funkiness" of different connections and different providers. We >> might even claim that cable providers make funkier network providers than >> cellular providers. >> But to what end? >> >> >> On Friday, July 25, 2014 5:13pm, "David Lang" <[email protected]> said: >> >> >> >>> On Fri, 25 Jul 2014, Martin Geddes wrote: >>> > So what is ΔQ and how do you "compute" it (to the extent it is a >>> "computed" >>> > thing)? >>> don't try to reduce it to a single number, we have two numbers that seem to >>> matter >>> 1. throughput (each direction) >>> 2. latency under load >>> Currently the speed test sites report throughput in each direction and ping >>> time >>> while not under load >>> If they could just add a ping time under load measurement, then we could >>> talk >>> meaningfully about either the delta or ratio of the ping times as the >>> "bufferbloat factor" >>> no, it wouldn't account for absolutly every nuance, but it would come pretty >>> close. >>> If a connection has good throughput and a low bufferbloat factor, it should >>> be >>> good for any type of use. >>> If it has good throughput, but a horrid bufferbloat factor, then you need to >>> artifically limit your traffic to stay clear of saturating the bandwith >>> (sacraficing throughput) >>> David Lang >>> > Starting point: the only observable effect of a network is to lose and >>> > delay data -- i.e. to "attenuate quality" by adding the toxic effects of >>> > time to distributed computations. ΔQ is a *morphism* that relates the >>> > "quality attenuation" that the network imposes to the application >>> > performance, and describes the trading spaces at all intermediate layers >>> > of >>> > abstraction. It is shown in the attached graphic. >>> > >>> > Critically, it frames quality as something that can only be lost >>> > ("attenuated"), both by the network and the application. Additionally, it >>> > is stochastic, and works with random variables and distributions. >>> > >>> > At its most concrete level, it is the individual impairment encountered by >>> > every packet when the network in operation. But we don't want to have to >>> > track every packet - 1:1 scale maps are pretty useless. So we need to >>> > abstract that in order to create a model that has value. >>> > >>> > Next abstraction: an improper random variable. This unifies loss and delay >>> > into a single stochastic object. >>> > Next abstraction: received transport, which is a CDF where we are >>> > interested in the properties of the "tail". >>> > >>> > Next abstraction, that joins network performance and application QoE (as >>> > relates to performance): relate the CDF to the application through a >>> > Quality Transport Agreement. This "stochastic contract" is both necessary >>> > and sufficient to deliver the application outcome. >>> > >>> > Next concretisation towards QoE: offered load of demand, as a CDF. >>> > Next concretisation towards QoE: breach hazard metric, which abstracts the >>> > application performance. Indicates the likelihood of the QTA contract >>> > being >>> > broken, and how badly. >>> > Final concretisation: the individual application performance encountered >>> > by >>> > every user. Again, a 1:1 map that isn't very helpful. >>> > >>> > So as you can see, it's about as far away from a single point average >>> > metric as you can possibly get. A far richer model is required in order to >>> > achieve robust performance engineering. >>> > >>> > It is "computed" using multi-point measurements to capture the >>> > distribution. The G/S/V charts you see are based on processing that data >>> > to >>> > account for various issues, including clock skew. >>> > >>> > I hope that helps. We need to document more of this in public, which is an >>> > ongoing process. >>> > >>> > Martin >>> > >>> > On 25 July 2014 16:58, Sebastian Moeller <[email protected]> wrote: >>> > >>> >> Hi Martin, >>> >> >>> >> thanks for the pointers, >>> >> >>> >> >>> >> On Jul 25, 2014, at 16:25 , Martin Geddes <[email protected]> >>> wrote: >>> >> >>> >>> You may find the following useful background reading on the state of >>> the >>> >> art in network measurement, and a primer on ΔQ (which is the >>> property we >>> >> wish to measure). >>> >>> >>> >>> First, start with this presentation: Network performance >>> optimisation >>> >> using high-fidelity measures >>> >>> Then read this one to decompose ΔQ into G, S and V: >>> Fundamentals of >>> >> network performance engineering >>> >>> Then read this one to get a bit more sense on what ΔQ is >>> about: >>> >> Introduction to ΔQ and Network Performance Science (extracts) >>> >>> >>> >>> Then read these essays: >>> >>> >>> >>> Foundation of Network Science >>> >>> How to do network performance chemistry >>> >>> How to X-ray a telecoms network >>> >>> There is no quality in averages: IPX case study >>> >> >>> >> All of this makes intuitively sense, but it is a bit light on >>> how >>> >> deltaQ is to be computed ;). >>> >> As far as I understand it also has not much bearing on my home >>> >> network; the only one under my control. Now, following the buffer bloat >>> >> discussion for some years, I have internalized the idea that bandwidth >>> >> alone does not suffice to describe the quality of my network connection. >>> I >>> >> think that the latency increase under load (for unrelated flows) is the >>> >> best of all the bad single number measures of network dynamics/quality. >>> I >>> >> should be related to what I understood deltaQ to depend on (as packet >>> loss >>> >> for non real time flows will cause an increase in latency). I think >>> that >>> >> continuous measurements make a to n of sense for ISPs, >>> backbone-operators, >>> >> mobile carriers … but at home, basically, I operate as my own >>> network >>> >> quality monitor ;) (that is I try to pin point and debug (transient) >>> >> anomalies). >>> >> >>> >>> >>> >>> Martin >>> >>> >>> >>> For fresh thinking about telecoms sign up for my free newsletter or >>> >> visit the Geddes Think Tank. >>> >>> LinkedIn Twitter Mobile: +44 7957 499219 Skype: mgeddes >>> >>> Martin Geddes Consulting Ltd, Incorporated in Scotland, number >>> SC275827 >>> >> VAT Number: 859 5634 72 Registered office: 17-19 East London Street, >>> >> Edinburgh, EH7 4BN >>> >>> >>> >>> >>> >>> >>> >>> On 25 July 2014 15:17, Sebastian Moeller <[email protected]> >>> wrote: >>> >>> Hi Neil, >>> >>> >>> >>> >>> >>> On Jul 25, 2014, at 14:24 , Neil Davies <[email protected]> >>> wrote: >>> >>> >>> >>>> Rich >>> >>>> >>> >>>> I have a deep worry over this style of single point measurement - >>> and >>> >> hence speed - as an appropriate measure. >>> >>> >>> >>> But how do you propose to measure the (bottleneck) link >>> capacity >>> >> then? It turns out for current CPE and CMTS/DSLAM equipment one >>> typically >>> >> can not relay on good QoE out of the box, since typically these devices >>> do >>> >> not use their (largish) buffers wisely. Instead the current remedy is to >>> >> take back control over the bottleneck link by shaping the actually sent >>> >> traffic to stay below the hardware link capacity thereby avoiding >>> feeling >>> >> the consequences of the over-buffering. But to do this is is quite >>> helpful >>> >> to get an educated guess what the bottleneck links capacity actually is. >>> >> And for that purpose a speediest seems useful. >>> >>> >>> >>> >>> >>>> We know, and have evidence, that throughput/utilisation is not a >>> good >>> >> proxy for the network delivering suitable quality of experience. We work >>> >> with organisation (Telco’s, large system integrators etc) where we >>> spend a >>> >> lot of time having to “undo” the consequences of >>> “maximising speed”. Just >>> >> like there is more to life than work, there is more to QoE than speed. >>> >>>> >>> >>>> For more specific comments see inline >>> >>>> >>> >>>> On 25 Jul 2014, at 13:09, Rich Brown >>> <[email protected]> wrote: >>> >>>> >>> >>>>> Neil, >>> >>>>> >>> >>>>> Thanks for the note and the observations. My thoughts: >>> >>>>> >>> >>>>> 1) I note that speedof.me does seem to overstate the speed >>> results. >>> >> At my home, it reports 5.98mbps down, and 638kbps up, while >>> >> betterspeedtest.sh shows 5.49/0.61 mbps. (speedtest.net gives numbers >>> >> similar to the betterspeedtest.net script.) >>> >>>>> >>> >>>>> 2) I think we're in agreement about the peak upload rate that >>> you >>> >> point out is too high. Their measurement code runs in the browser. It >>> seems >>> >> likely that the browser pumps out a few big packets before getting flow >>> >> control information, thus giving the impression that they can send at a >>> >> higher rate. This comports with the obvious decay that ramps toward the >>> >> long-term rate. >>> >>>> >>> >>>> I think that its simpler than that, it is measuring the rate at >>> which >>> >> it can push packets out the interface - its real time rate is precisely >>> >> that - it can not be the rate being reported by the far end, it can >>> never >>> >> exceed the limiting link. The long term average (if it is like other >>> speed >>> >> testers we’ve had to look into) is being measured at the TCP/IP SDU >>> level >>> >> by measuring the difference in time between the first and last >>> timestamps >>> >> of data stream and dividing that into the total data sent. Their >>> >> “over-estimate” is because there are packets buffered in the >>> CPE that have >>> >> left the machine but not arrived at the far end. >>> >>> >>> >>> Testing from an openwrt router located at a >>> >> high-symmetric-bandwidth location shows that speedof.me does not scale >>> >> higher than ~ 130 Mbps server to client and ~15Mbps client to server (on >>> >> the same connection I can get 130Mbps S2C and ~80Mbps C2S, so the >>> asymmetry >>> >> in the speedof.me results is not caused by my local environment). >>> >>> @Rich and Dave, this probably means that for the upper end >>> of >>> >> fiber and cable and VDSL connections speed of.me is not going to be a >>> >> reliable speed measure… Side note www.sppedtest.net shows ~100Mbps >>> S2C >>> >> and ~100Mbps C2S, so might be better suited to high-upload links... >>> >>> >>> >>>> >>> >>>>> >>> >>>>> 3) But that long-term speed should be at or below the >>> theoretical >>> >> long-term rate, not above it. >>> >>>> >>> >>>> Agreed, but in this case knowing the sync rate already defines >>> that >>> >> maximum. >>> >>> >>> >>> I fully agree, but for ADSL the sync rate also contains a lot >>> of >>> >> encapsulation, so the maximum achievable TCP rate is at best ~90% of >>> link >>> >> rate. Note for cerowrt’s SQM system the link rate is exactly the >>> right >>> >> number to start out with at that system can take the encapsulation into >>> >> account. But even then it is somewhat unintuitive to deduce the expected >>> >> good-put from the link rate. >>> >>> >>> >>>> >>> >>>>> >>> >>>>> Two experiments for you to try: >>> >>>>> >>> >>>>> a) What does betterspeedtest.sh show? (It's in the latest >>> CeroWrt, in >>> >> /usr/lib/CeroWrtScripts, or get it from github: >>> >> https://github.com/richb-hanover/CeroWrtScripts ) >>> >>>>> >>> >>>>> b) What does www.speedtest.net show? >>> >>>>> >>> >>>>> I will add your question (about the inaccuracy) to the note >>> that I >>> >> want to send out to speedof.me this weekend. I will also ask that they >>> >> include min/max latency measurements to their test, and an option to >>> send >>> >> for > 10 seconds to minimize any effect of PowerBoost… >>> >>> >>> >>> I think they do already, at least for the download >>> bandwidth; >>> >> they start with 128Kb and keep doubling the file size until a file takes >>> >> longer than 8 seconds to transfer, they only claim to report the numbers >>> >> from that last transferred file, so worst case with a stable link and a >>> >> bandwidth > 16kbps ;), it has taken at least 12 seconds (4 plus 8) of >>> >> measuring before the end of the plot, so the bandwidth of at least the >>> last >>> >> half of the download plot should be representative even assuming power >>> >> boost. Caveat, I assume that power boost will not be reset by the >>> transient >>> >> lack of data transfer between the differently sized files (but since it >>> >> should involve the same IPs and port# why should power boost reset >>> itself?). >>> >>> >>> >>> Best Regards >>> >>> Sebastian >>> >>> >>> >>> >>> >>> >>> >>>>> >>> >>>>> Best regards, >>> >>>>> >>> >>>>> Rich >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Jul 25, 2014, at 5:10 AM, Neil Davies >>> <[email protected]> >>> >> wrote: >>> >>>>> >>> >>>>>> Rich >>> >>>>>> >>> >>>>>> You may want to check how accurate they are to start. >>> >>>>>> >>> >>>>>> I just ran a “speed test” on my line (which I >>> have complete control >>> >> and visibility over the various network elements) and it reports an >>> average >>> >> “speed” (in the up direction) that is in excess of the >>> capacity of the >>> >> line, it reports the maximum rate at nearly twice the best possible rate >>> of >>> >> the ADSL connection. >>> >>>>>> >>> >>>>>> Doesn’t matter how pretty it is, if its not >>> accurate it is of no >>> >> use. This is rather ironic as the web site claims it is the >>> “smartest and >>> >> most accurate”! >>> >>>>>> >>> >>>>>> Neil >>> >>>>>> >>> >>>>>> <speedof_me_14-07-25.png> >>> >>>>>> >>> >>>>>> PS pretty clear to me what mistake they’ve made in >>> the measurement >>> >> process - its to do with incorrect inference and hence missing the >>> >> buffering effects. >>> >>>>>> >>> >>>>>> On 20 Jul 2014, at 14:19, Rich Brown >>> <[email protected]> >>> >> wrote: >>> >>>>>> >>> >>>>>>> Doc Searls ( >>> >> >>> http://blogs.law.harvard.edu/doc/2014/07/20/the-cliff-peronal-clouds-need-to-climb/) >>> >> mentioned in passing that he uses a new speed test website. I checked it >>> >> out, and it was very cool… >>> >>>>>>> >>> >>>>>>> www.speedof.me is an all-HTML5 website that seems to >>> make accurate >>> >> measurements of the up and download speeds of your internet connection. >>> >> It’s also very attractive, and the real-time plots of the speed >>> show >>> >> interesting info. (screen shot at: http://richb-hanover.com/speedof-me/) >>> >>>>>>> >>> >>>>>>> Now if we could get them to a) allow longer/bigger >>> tests to >>> >> circumvent PowerBoost, and b) include a latency measurement so people >>> could >>> >> point out their bufferbloated equipment. >>> >>>>>>> >>> >>>>>>> I'm going to send them a note. Anything else I should >>> add? >>> >>>>>>> >>> >>>>>>> Rich >>> >>>>>>> _______________________________________________ >>> >>>>>>> Bloat mailing list >>> >>>>>>> [email protected] >>> >>>>>>> https://lists.bufferbloat.net/listinfo/bloat >>> >>>>>> >>> >>>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> Bloat mailing list >>> >>>> [email protected] >>> >>>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >>> >>> _______________________________________________ >>> >>> Bloat mailing list >>> >>> [email protected] >>> >>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> >>> >> >>> >> >>> >_______________________________________________ >>> Cerowrt-devel mailing list >>> [email protected] >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> [email protected] >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel > _______________________________________________ > Bloat mailing list > [email protected] > https://lists.bufferbloat.net/listinfo/bloat _______________________________________________ Bloat mailing list [email protected] https://lists.bufferbloat.net/listinfo/bloat
