The starlink list was not originally cc´d and yet since I think this debate concerns that also, I have added the cc back. Carry on!
On Sat, Sep 30, 2023 at 8:20 AM Sebastian Moeller via LibreQoS <libre...@lists.bufferbloat.net> wrote: > > Hi Mike, > > [I took the liberty to remove some individual address from the Cc, as I > assume most/all already be covered by the lists] > > > > On Sep 30, 2023, at 16:41, Mike Conlow <mcon...@cloudflare.com> wrote: > > > > First, a thank you to Dave, and lots of you all, for longtime shepherding > > of this community and efforts to make the Internet better. > > > > As I read this thread and think about the coming debate in the U.S., two > > things come to mind: > > > > 1. Ofcom is considering a net neutrality "clarification". The first topic > > in the consultation is whether ISPs should be allowed to offer "premium > > quality retail plans". It doesn't specify the technical implementation, but > > there would be different plans for "users who mainly stream" vs "people who > > use high quality virtual reality applications". Apparently ISPs feel the > > existing NN rules are not clear on whether this is allowed. > > [SM] Not sure this is not simply an attempt of using regulatory > divergence from the EU (IMHO for no good reason or outcome)... Also und er > the existing EU rules ISPs are arguably already free to treat the whole class > of latency sensitive VR to lower delay than bulk streaming as long as they do > son consistently and not based on commercial relationships with the senders... > During the covid19 lock downs the EU offered clarification on the regulation > that really drove home the do not discriminate inside of a specific traffic > class, and define classes by purely technical not economical parameters. That > said, I always like to look at data and hence am happy to the the UK > apparently prepping to run that experiment (I am also happy not to live there > right now not having to prticipate in said experiment*). > > > *) Other than that the british islands offer a lot of really great places I > certainly would like to live at, but I digress. > > > > > The question I'm thinking about is do we want an Internet where end user > > plans are divided up this way? > > [SM] Personally, I consider internet access infrastructure and do not > think this looks like a good way forward. > > > And if not, is a NN regulation the right place to put those rules? > > [SM] Could well be, but depends on the framing, no? > > > > > 2. To the point in the PS of the below email, I would agree things are > > mostly working in the EU, and in the US. But things are broken in Germany > > to the point where consumers have a degraded Internet experience because > > their ISP won't provision enough interconnection. > > [SM] This a very peculiar case of the local incumbent Deutsche > Telekom (DTAG) (all in all a pretty competent ISP that runs a tight ship in > its network and tends to follow regulations to the letter (not however > necessarily to the intent, but they are not different from other corporations > of similar size)). DTAG is large enough to qualify as tier 1 (T1) ISP that > is, to the best of our knowledge they do not pay anybody for transit and peer > with all other T1-ISPs, they also have a relative large share of eye-balls in > one of Europes larger and profitable markets. They (as did AT&T and Verizon > in the US and probably other ISPs in similar positions as well) that most of > their users traffic was within network (e.g. from German companies > hosted/homed by DTAG) or via important partners like Google that have decent > peering links (unclear whether/if Google actually is charged for that) but > that there is a considerable number of services that reach DTAG eye-balls via > their transit, that is essentially via one of the other T1-ISPs (I simplify > here, I have no insight in the actual bisiness relationships between all > players). And now DTAG basically instructed its generally capable network > team to simply manage the size of the peering links with the big > transit-providers carefully so that they never fully clogg, but clearly see > increased packet loss and queueing delay during prime time. That in turn is > clearly a competition problem if streaming service A judders/jitters/and > buffers jumps between quality tiers while streaming service B smoothly and > boringly just streams at the desired resolution. Now Telekom is happy to > offer service A a product they call "internet transit" but that is priced > pretty high (I have seen some comparative numbers for transit pricing in > Germany I am not permitted to share or reveal more about) so high in fact > that no content provider that can afford more than a single transit provider > would use for anything but reaching DTAG eye-balls or closely related ones > (like in the past SwissCom). > This behaviour is not s secret but evades regulatory action, because it does > not openly violate the EU regulation which in the BEREC interpretation does > not really cover the interconnection side. DTAG is careful enough to not > purposefully target specific potential customers but simply treats all > traffic coming in/out via "other transit than its own" as "has to tolerate > overheated links during primetime". > > > > Are NN rules the right place to address this > > [SM] They could well be the actual text of the 2015/2120 does not > make a distinction between access and interconnection. But this is a tricky > field and will directly affect parts of larger ISP's core business so I do > not see this happen in the EU anytime soon, unless ISPs like Telekom clearly > abuses this in a way that is too obvious... ATM it is mostly telecom, but I > believe any of the big old monopoly incumbents likely is big enough in its > home market to pull of a stunt like this, so there is the potential of > someone over doing it... > > > and make sure it doesn't happen in the US? Or is one bad actor across the > > EU and US the cost of doing business and the Internet ecosystem and > > "market" are mostly solving the issue? > > [SM] As happy as I am to diss DTAG for that behavior (I am also happy > to praise it in ears where it shows exemplary behavior) DTAG is not alone in > that business acumen, I think that some of the big US telco's dod/do exactly > the same, but unlike telekom I have no evidence. > > > Regards > Sebastian > > P.S.: I was a customer at DTAG for several years and I did not notice the > conscious under-peering with the other T1 ISPs in my day to day usage, so > while the issue clearly and measurably exists it is not an issue that normal > users will encounter often and are also unlikely to properly root-cause (the > blame will likely land by my example service A above). > > > > > > > > > > On Sat, Sep 30, 2023 at 8:19 AM Sebastian Moeller via Rpm > > <r...@lists.bufferbloat.net> wrote: > > Hi Frantisek, > > > > > On Sep 30, 2023, at 14:00, Frantisek Borsik via Rpm > > > <r...@lists.bufferbloat.net> wrote: > > > > > > Back then in 2015, when NN was enacted by Wheeler & CO, there was a great > > > body of work (IMHO) done on this subject by Martin Geddes: > > > https://www.martingeddes.com/1261-2/ > > > > > > But let's pick one report written by his colleagues and published by > > > Ofcom (UK telecoms regulator): > > > > > > • You cannot conflate ‘equality of [packet] treatment’ with > > > delivering equality of [user application] outcomes. Only the latter > > > matters, as ordinary users don’t care what happened to the packets in > > > transit. Yet the relevant academic literature fixates on the local > > > operation of the mechanisms (including Traffic Management), not their > > > global aggregate effect. > > > > [SM] The EU laid out pretty clear why they mandated the NN > > regulations in eu regulation 2015/2120: > > > > [...] > > (8) > > When providing internet access services, providers of those services should > > treat all traffic equally, without discrimination, restriction or > > interference, independently of its sender or receiver, content, application > > or service, or terminal equipment. According to general principles of Union > > law and settled case-law, comparable situations should not be treated > > differently and different situations should not be treated in the same way > > unless such treatment is objectively justified. > > (9) > > The objective of reasonable traffic management is to contribute to an > > efficient use of network resources and to an optimisation of overall > > transmission quality responding to the objectively different technical > > quality of service requirements of specific categories of traffic, and thus > > of the content, applications and services transmitted. Reasonable traffic > > management measures applied by providers of internet access services should > > be transparent, non-discriminatory and proportionate, and should not be > > based on commercial considerations. The requirement for traffic management > > measures to be non-discriminatory does not preclude providers of internet > > access services from implementing, in order to optimise the overall > > transmission quality, traffic management measures which differentiate > > between objectively different categories of traffic. Any such > > differentiation should, in order to optimise overall quality and user > > experience, be permitted only on the basis of objectively different > > technical quality of service requirements (for example, in terms of > > latency, jitter, packet loss, and bandwidth) of the specific categories of > > traffic, and not on the basis of commercial considerations. Such > > differentiating measures should be proportionate in relation to the purpose > > of overall quality optimisation and should treat equivalent traffic > > equally. Such measures should not be maintained for longer than necessary. > > (10) > > Reasonable traffic management does not require techniques which monitor the > > specific content of data traffic transmitted via the internet access > > service. > > (11) > > Any traffic management practices which go beyond such reasonable traffic > > management measures, by blocking, slowing down, altering, restricting, > > interfering with, degrading or discriminating between specific content, > > applications or services, or specific categories of content, applications > > or services, should be prohibited, subject to the justified and defined > > exceptions laid down in this Regulation. Those exceptions should be subject > > to strict interpretation and to proportionality requirements. Specific > > content, applications and services, as well as specific categories thereof, > > should be protected because of the negative impact on end-user choice and > > innovation of blocking, or of other restrictive measures not falling within > > the justified exceptions. Rules against altering content, applications or > > services refer to a modification of the content of the communication, but > > do not ban non-discriminatory data compression techniques which reduce the > > size of a data file without any modification of the content. Such > > compression enables a more efficient use of scarce resources and serves the > > end-users’ interests by reducing data volumes, increasing speed and > > enhancing the experience of using the content, applications or services > > concerned. > > (12) > > Traffic management measures that go beyond such reasonable traffic > > management measures may only be applied as necessary and for as long as > > necessary to comply with the three justified exceptions laid down in this > > Regulation. > > [...] > > > > There really is little IMHO that can be brought against these, all pretty > > fair and reasonable. What it does is accept that internet access is > > essential infrastructure and that hence access needs to be as well > > regulated as access to water, electricity, gas, streets, ... . Yes this has > > some consequences of what ISPs can and can not do. But this is normal "cost > > of business". I for one am quite happy about this regulation existing as > > locally it did away with some (not all) shenanigans of some ISPs that were > > clearly not operating in the interest of their paying eye-balls. > > > > There is a whole cottage industry of consultants that decry the EU's > > decision and try to lobby against it, but honestly reading these mostly > > makes me think harsher regulation might be required (on consultans about > > how much they are allowed to massage the facts ;) ). > > > > Regards > > Sebastian > > > > P.S.: Of course if we look close enough we surely can find corner-cases > > where either the EU regulations or the translation into national law result > > in less desirable outcomes, but "nothing is perfect" and all in all the > > regulations seem to be "good enough". With the caveat that explicitly > > excluding ISP interconnect from the regulations BEREC essentially pointed > > the way for ISPs wanting to monetize their eye-balls twice to do so via > > interconnects, but that only works for the 800 pound gorillas and generally > > is not a game smaller ISPs can play. I do understand why BEREC wants to > > stay out of the interconnection issue, as this is rather complicated and > > the market seems to generally work okay-ish (that is not badly enough to > > make intervention a hot-button issue for voters and hence politicians). > > > > > > > > > > > > All the best, > > > > > > Frank > > > > > > Frantisek (Frank) Borsik > > > > > > > > > > > > https://www.linkedin.com/in/frantisekborsik > > > > > > Signal, Telegram, WhatsApp: +421919416714 > > > > > > iMessage, mobile: +420775230885 > > > > > > Skype: casioa5302ca > > > > > > frantisek.bor...@gmail.com > > > > > > > > > > > > On Fri, Sep 29, 2023 at 6:15 PM dan via Rpm <r...@lists.bufferbloat.net> > > > wrote: > > > ok, lots and lots of great comments here for sure. > > > > > > bandwidth abundance: Not for most people and ISPs. The 'carriers' > > > aren't carrying to many places at affordable rates. I've pulled quotes > > > from Lumen and Zayo at over $5k/month/gig. We typically pay 900-1400 for > > > a gig of service. This isn't abundance and it's widespread and it leaves > > > only major providers that can afford/amortize out 100G transits etc. > > > My answer to this is one that Dave and I have bounced back and forth is > > > the idea of micro IXs in every municipality and having that somehow tied > > > into access to the ROW in counties etc. Not fully hashed out, but the > > > fiber is in the ground, it could be sold, but the carriers are not well > > > incentivised to sell it. It takes the better part of a year to get a DIA > > > within 100ft of a Lumen hut sometimes... Heck, it could even be a > > > government program to get an μIX with x feet of every school, city hall, > > > and library. I don't care how it's done but this would get abundance > > > NEAR end users and open up essentially every town to competition. > > > > > > monopoly. This is a historical thing for most cable and DSL incumbents. > > > They have enjoyed virtual monopolies with cable owning population centers > > > and DSL owning the outskirts and there is no product darwinism here where > > > customer satisfaction is a pressure. That may not be the future but it > > > definitely is the past. These companies may have to shift into customer > > > satisfaction as a major part instead of a minor part of their corporate > > > culture to fend off fttx and ultra-modern wisps. > > > > > > Starlink is not offering significant competition to major carriers. > > > Starlink's 1.5 million customers are at LEAST 90% pulled from other > > > satellite services and small ISPs. Spectrum and Comcast's losses to > > > starlink are measured in decimal points. > > > > > > Only fttx and ultra-modern wireless tech really threatens these > > > incumbents. Typical wisps aren't putting a dent in these guys, just > > > scraping the paint off their bumper. We're pulling customers at the > > > scale of 'dozens' for example. Spectrum's management doesn't know we > > > exist we're such a small threat to them. > > > > > > But to further the point here, these fttx and ultra-modern wisps can only > > > exist in places where there is adequate carrier services to start with. > > > In areas where they spend the money and do the build but there aren't > > > good carrier services, those fiber services suck and the wISPs start to > > > claw back even with inferior technology. We've pulled quite a few > > > customers off fttx deployments because of this sort of situation. > > > > > > > > > On Fri, Sep 29, 2023 at 7:28 AM Rich Brown <richb.hano...@gmail.com> > > > wrote: > > > Thank you Jonathan for this clear description of the issues and their > > > history. I wonder if there's a fourth one - privacy. > > > > > > Rosenworcel's talk > > > https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf also points out > > > that ISPs might want to monetize our traffic patterns and location data. > > > (This is less of an issue in the EU, but the US remains a Wild West in > > > this regard.) > > > > > > I am hopeful that the FCC will include this in their NPRM (which must be > > > available by now but I haven't looked...) > > > > > > - Rich Brown > > > > > > > On Sep 29, 2023, at 12:54 AM, Jonathan Morton via Rpm > > > > <r...@lists.bufferbloat.net> wrote: > > > > > > > >> On 29 Sep, 2023, at 1:19 am, David Lang via Bloat > > > >> <bloat@lists.bufferbloat.net> wrote: > > > >> > > > >> Dave T called out earlier that the rise of bittorrent was a large part > > > >> of the inital NN discussion here in the US. But a second large portion > > > >> was a money grab from ISPs thinking that they could hold up large paid > > > >> websites (netflix for example) for additional fees by threatening to > > > >> make their service less useful to their users (viewing their users as > > > >> an asset to be marketed to the websites rather than customers to be > > > >> satisfied by providing them access to the websites) > > > >> > > > >> I don't know if a new round of "it's not fair that Netflix doesn't pay > > > >> us for the bandwidth to service them" would fall flat at this point or > > > >> not. > > > > > > > > I think there were three more-or-less separate concerns which have, > > > > over time, fallen under the same umbrella: > > > > > > > > > > > > 1: Capacity-seeking flows tend to interfere with latency-sensitive > > > > flows, and the "induced demand" phenomenon means that increases in link > > > > rate do not in themselves solve this problem, even though they may be > > > > sold as doing so. > > > > > > > > This is directly addressed by properly-sized buffers and/or AQM, and > > > > even better by FQ and SQM. It's a solved problem, so long as the > > > > solutions are deployed. It's not usually necessary, for example, to > > > > specifically enhance service for latency-sensitive traffic, if FQ does > > > > a sufficiently good job. An increased link rate *does* enhance service > > > > quality for both latency-sensitive and capacity-seeking traffic, > > > > provided FQ is in use. > > > > > > > > > > > > 2: Swarm traffic tends to drown out conventional traffic, due to > > > > congestion control algorithms which try to be more-or-less fair on a > > > > per-flow basis, and the substantially larger number of parallel flows > > > > used by swarm traffic. This also caused subscribers using swarm > > > > traffic to impair the service of subscribers who had nothing to do with > > > > it. > > > > > > > > FQ on a per-flow basis (see problem 1) actually amplifies this effect, > > > > and I think it was occasionally used as an argument for *not* deploying > > > > FQ. ISPs' initial response was to outright block swarm traffic where > > > > they could identify it, which was then softened to merely throttling it > > > > heavily, before NN regulations intervened. Usage quotas also showed up > > > > around this time, and were probably related to this problem. > > > > > > > > This has since been addressed by several means. ISPs may use FQ on a > > > > per-subscriber basis to prevent one subscriber's heavy traffic from > > > > degrading service for another. Swarm applications nowadays tend to > > > > employ altruistic congestion control which deliberately compensates for > > > > the large number of flows, and/or mark them with one or more of the > > > > Least Effort class DSCPs. Hence, swarm applications are no longer as > > > > damaging to service quality as they used to be. Usage quotas, however, > > > > still remain in use as a profit centre, to the point where an > > > > "unlimited" service is a rare and precious specimen in many > > > > jurisdictions. > > > > > > > > > > > > 3: ISPs merged with media distribution companies, creating a conflict > > > > of interest in which the media side of the business wanted the internet > > > > side to actively favour "their own" media traffic at the expense of > > > > "the competition". Some ISPs began to actively degrade Netflix > > > > traffic, in particular by refusing to provision adequate peering > > > > capacity at the nodes through which Netflix traffic predominated, or by > > > > zero-rating (for the purpose of usage quotas) traffic from their own > > > > media empire while refusing to do the same for Netflix traffic. > > > > > > > > **THIS** was the true core of Net Neutrality. NN regulations forced > > > > ISPs to carry Netflix traffic with reasonable levels of service, even > > > > though they didn't want to for purely selfish and greedy commercial > > > > reasons. NN succeeded in curbing an anti-competitive and > > > > consumer-hostile practice, which I am perfectly sure would resume just > > > > as soon as NN regulations were repealed. > > > > > > > > And this type of practice is just the sort of thing that technologies > > > > like L4S are designed to support. The ISPs behind L4S actively do not > > > > want a technology that works end-to-end over the general Internet. > > > > They want something that can provide a domination service within their > > > > own walled gardens. That's why L4S is a NN hazard, and why they > > > > actively resisted all attempts to displace it with SCE. > > > > > > > > > > > > All of the above were made more difficult to solve by the monopolistic > > > > nature of the Internet service industry. It is actively difficult for > > > > Internet users to move to a truly different service, especially one > > > > based on a different link technology. When attempts are made to > > > > increase competition, for example by deploying a publicly-funded > > > > network, the incumbents actively sabotage those attempts by any means > > > > they can. Monopolies are inherently customer-hostile, and arguments > > > > based on market forces fail in their presence. > > > > > > > > - Jonathan Morton > > > > > > > > _______________________________________________ > > > > Rpm mailing list > > > > r...@lists.bufferbloat.net > > > > https://lists.bufferbloat.net/listinfo/rpm > > > > > > _______________________________________________ > > > Rpm mailing list > > > r...@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > > > _______________________________________________ > > > Rpm mailing list > > > r...@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/rpm > > > > _______________________________________________ > > Rpm mailing list > > r...@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/rpm > > _______________________________________________ > LibreQoS mailing list > libre...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/libreqos -- Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html Dave Täht CSO, LibreQos _______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
