Re: COVID-19 vs. peering wars

[Mark Tinka]

On 5/Apr/20 15:52, Radu-Adrian Feurdean wrote:

> There was not regulation for that.
> There were some politicians crying out and loud in the media that streaming 
> platforms should reduce bit-rates. Netflix took the opportunity to try 
> (sooner than initially scheduled) new compression schemes/algorithms on their 
> platform. Further, they took the opportunity to say "everything is OK, the 
> new stuff will be deployed full-scale around Europe". In parallel, other 
> streaming platforms took their measures, some of them as simple as "default 
> is one level lower" (720p instead of 1080p, or even 480p instead of 720p). 
> That went as far as platforms that would never be named explicitly by any 
> "responsible" politician (like pornhub and sorts).
> Chances are the results of the "bitrate reduction" will end up in the US 
> pretty soon. Netflix are also insisting on the fact that it's not a quality 
> reduction, just new compression allowing for lower bitrates over the wire.
>
> The French regulator is even very decent in this respect, the official 
> message being : "situation is overall good, in the rare cases and places 
> where there are issues operators will do heir job to fix the issues".
>
> In general, there are no new issues, just probably more people realising the 
> issues that already existed for some time.

Curious - if anyone is actively monitoring streaming traffic trends,
have you seen a "decrease" in that graph even though you are sure the
same number of eyeballs (or even more) are online? Especially for those
VoD providers who are implementing new compression algorithms, and not
necessarily reducing resolution?

Mark.

Re: COVID-19 vs. peering wars

[Radu-Adrian Feurdean]

On Fri, Mar 20, 2020, at 20:31, Matthew Petach wrote:
> Netflix, Amazon Prime, Youtube, Hulu, and other video
> streaming services cut their bit rates down?
> 
> https://www.bbc.com/news/technology-51968302
> https://arstechnica.com/tech-policy/2020/03/netflix-and-youtube-cut-streaming-quality-in-europe-to-handle-pandemic/
> 
> It seems that perhaps the fingers, and the regulatory
> hammer, are being pointed in the wrong direction at

There was not regulation for that.
There were some politicians crying out and loud in the media that streaming 
platforms should reduce bit-rates. Netflix took the opportunity to try (sooner 
than initially scheduled) new compression schemes/algorithms on their platform. 
Further, they took the opportunity to say "everything is OK, the new stuff will 
be deployed full-scale around Europe". In parallel, other streaming platforms 
took their measures, some of them as simple as "default is one level lower" 
(720p instead of 1080p, or even 480p instead of 720p). That went as far as 
platforms that would never be named explicitly by any "responsible" politician 
(like pornhub and sorts).
Chances are the results of the "bitrate reduction" will end up in the US pretty 
soon. Netflix are also insisting on the fact that it's not a quality reduction, 
just new compression allowing for lower bitrates over the wire.

The French regulator is even very decent in this respect, the official message 
being : "situation is overall good, in the rare cases and places where there 
are issues operators will do heir job to fix the issues".

In general, there are no new issues, just probably more people realising the 
issues that already existed for some time.

Peering-wise, BAU, nothing new. Only thing is one of the 4 majors ISPs, ~21% 
market share, over 98% IPv6 deployment on fixed (and 0% on mobile) mono-homed 
to Cogent and de-peering HE. They are not peering as a general rule.

Re: COVID-19 vs. peering wars

[Bradley Huffaker]

Regardless of the possible gain from “solving” peering. 
You are talking about renegotiating thousands of individual 
agreements between hundreds of individual organizations, 
all while everyone is in lockdown.

or

You ask a handful of companies to make changes to their own systems. 
Good luck with the peering, I believe the bit rates have already been changed. 

Bradley 

> On Mar 21, 2020, at 4:31 AM, Matthew Petach  wrote:
> 
> 
> 
> I'm curious; 
> would people say that fixing peering inefficiencies could have 
> a bigger impact on service performance than asking that 
> Netflix, Amazon Prime, Youtube, Hulu, and other video
> streaming services cut their bit rates down?
> 
> https://www.bbc.com/news/technology-51968302 
> <https://www.bbc.com/news/technology-51968302>
> https://arstechnica.com/tech-policy/2020/03/netflix-and-youtube-cut-streaming-quality-in-europe-to-handle-pandemic/
>  
> <https://arstechnica.com/tech-policy/2020/03/netflix-and-youtube-cut-streaming-quality-in-europe-to-handle-pandemic/>
> 
> It seems that perhaps the fingers, and the regulatory
> hammer, are being pointed in the wrong direction at
> the moment.  ^_^;
> 
> Matt
> staying safely under the saran-wrap blanket for the next few weeks
> 
> 
> 
> 
> On Fri, Mar 20, 2020 at 9:31 AM Adam Thompson  <mailto:athomp...@merlin.mb.ca>> wrote:
> Every large ISP does this (or rather, doesn't) at every IX in Canada.  Bell 
> isn't unique by any stretch.
> 
> It's not in their economic interest to peer at a local IX, because from their 
> perspective, the IX takes away business (Managed L2 point-to-point circuits, 
> at the very least) from them.
> 
> Don't expect the dominant wireline ISP(s) in any region to join local IXes 
> anytime soon, sadly, no matter how much it would benefit their customers.  
> After all, the customer is always free to purchase service to the IX and join 
> the IX, right???  *grumble*
> 
> In my local case, if BellMTS joined MBIX, un-cached DNS resolution times 
> could potentially drop by 15msec.  That's HUGE.  But the end-user experience 
> is not their primary goal.  Their primary goal is profit, as always.
> 
> -Adam Thompson
>  Founding member, MBIX (once upon a time)
> 
> Adam Thompson
> Consultant, Infrastructure Services
> MERLIN
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athomp...@merlin.mb.ca <mailto:athomp...@merlin.mb.ca>
> www.merlin.mb.ca <http://www.merlin.mb.ca/>
> 
> > -Original Message-
> > From: NANOG mailto:nanog-boun...@nanog.org>> On 
> > Behalf Of Sadiq Saif
> > Sent: Friday, March 20, 2020 9:38 AM
> > To: nanog@nanog.org <mailto:nanog@nanog.org>
> > Subject: Re: COVID-19 vs. peering wars
> > 
> > On Fri, 20 Mar 2020, at 10:31, Steve Mikulasik via NANOG wrote:
> > >
> > > In Canada the CRTC really needs to get on Canadian ISPs about peering
> > > very liberally at IXs in each province. I know of one major
> > > institution right now that would have a major work from home issue
> > > resolved if one big ISP would peer with one big tier 1 in the IX they
> > > are both located at in the same province. Instead traffic needs to
> > > flow across the country or to the USA to get back to the same city.
> > 
> > **cough** Bell Canada **cough**.
> > 
> > --
> >   Sadiq Saif
> >   https://sadiqsaif.com/ <https://sadiqsaif.com/>
> 

RE: COVID-19 vs. peering wars

[Adam Thompson]

Worldwide, I don’t know.

In Canada, peering is pretty messed up, i.e. it simply doesn’t happen at scale. 
 At all.  Even where it should.  The overwhelmingly vast majority of Canadian 
traffic, even when nominally in-country, still transits the USA somewhere.

If we had “ideal” full-mesh peering (i.e. setting aside all commercial 
considerations) at, say, regional IXes, including various popular CDNs, then 
service would take a giant step for the better for everyone who isn’t a big-4 
(Bell, Telus, Shaw or Rogers) customer.  Which admittedly would be an 
improvement for “only” about 30%-40% of the population… negligible, really, 
we’re only a country of 10M after all :-/.

FYI, we have 4 big ISPs because none of them cover the entire country: they 
all* descend from local/regional monopolies or duopolies.   *Mostly, that’s an 
approximation.
-Adam

Adam Thompson
Consultant, Infrastructure Services
[[MERLIN LOGO]]<https://www.merlin.mb.ca/>
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca<mailto:athomp...@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

From: Matthew Petach 
Sent: Friday, March 20, 2020 2:31 PM
To: Adam Thompson 
Cc: Sadiq Saif ; nanog@nanog.org
Subject: Re: COVID-19 vs. peering wars



I'm curious;
would people say that fixing peering inefficiencies could have
a bigger impact on service performance than asking that
Netflix, Amazon Prime, Youtube, Hulu, and other video
streaming services cut their bit rates down?

https://www.bbc.com/news/technology-51968302
https://arstechnica.com/tech-policy/2020/03/netflix-and-youtube-cut-streaming-quality-in-europe-to-handle-pandemic/

It seems that perhaps the fingers, and the regulatory
hammer, are being pointed in the wrong direction at
the moment.  ^_^;

Matt
staying safely under the saran-wrap blanket for the next few weeks




On Fri, Mar 20, 2020 at 9:31 AM Adam Thompson 
mailto:athomp...@merlin.mb.ca>> wrote:
Every large ISP does this (or rather, doesn't) at every IX in Canada.  Bell 
isn't unique by any stretch.

It's not in their economic interest to peer at a local IX, because from their 
perspective, the IX takes away business (Managed L2 point-to-point circuits, at 
the very least) from them.

Don't expect the dominant wireline ISP(s) in any region to join local IXes 
anytime soon, sadly, no matter how much it would benefit their customers.  
After all, the customer is always free to purchase service to the IX and join 
the IX, right???  *grumble*

In my local case, if BellMTS joined MBIX, un-cached DNS resolution times could 
potentially drop by 15msec.  That's HUGE.  But the end-user experience is not 
their primary goal.  Their primary goal is profit, as always.

-Adam Thompson
 Founding member, MBIX (once upon a time)

Adam Thompson
Consultant, Infrastructure Services
MERLIN
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca<mailto:athomp...@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca>

> -Original Message-
> From: NANOG mailto:nanog-boun...@nanog.org>> On 
> Behalf Of Sadiq Saif
> Sent: Friday, March 20, 2020 9:38 AM
> To: nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: Re: COVID-19 vs. peering wars
>
> On Fri, 20 Mar 2020, at 10:31, Steve Mikulasik via NANOG wrote:
> >
> > In Canada the CRTC really needs to get on Canadian ISPs about peering
> > very liberally at IXs in each province. I know of one major
> > institution right now that would have a major work from home issue
> > resolved if one big ISP would peer with one big tier 1 in the IX they
> > are both located at in the same province. Instead traffic needs to
> > flow across the country or to the USA to get back to the same city.
>
> **cough** Bell Canada **cough**.
>
> --
>   Sadiq Saif
>   https://sadiqsaif.com/

Re: COVID-19 vs. peering wars

[Matthew Petach]

I'm curious;
would people say that fixing peering inefficiencies could have
a bigger impact on service performance than asking that
Netflix, Amazon Prime, Youtube, Hulu, and other video
streaming services cut their bit rates down?

https://www.bbc.com/news/technology-51968302
https://arstechnica.com/tech-policy/2020/03/netflix-and-youtube-cut-streaming-quality-in-europe-to-handle-pandemic/

It seems that perhaps the fingers, and the regulatory
hammer, are being pointed in the wrong direction at
the moment.  ^_^;

Matt
staying safely under the saran-wrap blanket for the next few weeks




On Fri, Mar 20, 2020 at 9:31 AM Adam Thompson 
wrote:

> Every large ISP does this (or rather, doesn't) at every IX in Canada.
> Bell isn't unique by any stretch.
>
> It's not in their economic interest to peer at a local IX, because from
> their perspective, the IX takes away business (Managed L2 point-to-point
> circuits, at the very least) from them.
>
> Don't expect the dominant wireline ISP(s) in any region to join local IXes
> anytime soon, sadly, no matter how much it would benefit their customers.
> After all, the customer is always free to purchase service to the IX and
> join the IX, right???  *grumble*
>
> In my local case, if BellMTS joined MBIX, un-cached DNS resolution times
> could potentially drop by 15msec.  That's HUGE.  But the end-user
> experience is not their primary goal.  Their primary goal is profit, as
> always.
>
> -Adam Thompson
>  Founding member, MBIX (once upon a time)
>
> Adam Thompson
> Consultant, Infrastructure Services
> MERLIN
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athomp...@merlin.mb.ca
> www.merlin.mb.ca
>
> > -Original Message-
> > From: NANOG  On Behalf Of Sadiq Saif
> > Sent: Friday, March 20, 2020 9:38 AM
> > To: nanog@nanog.org
> > Subject: Re: COVID-19 vs. peering wars
> >
> > On Fri, 20 Mar 2020, at 10:31, Steve Mikulasik via NANOG wrote:
> > >
> > > In Canada the CRTC really needs to get on Canadian ISPs about peering
> > > very liberally at IXs in each province. I know of one major
> > > institution right now that would have a major work from home issue
> > > resolved if one big ISP would peer with one big tier 1 in the IX they
> > > are both located at in the same province. Instead traffic needs to
> > > flow across the country or to the USA to get back to the same city.
> >
> > **cough** Bell Canada **cough**.
> >
> > --
> >   Sadiq Saif
> >   https://sadiqsaif.com/
>
>

RE: COVID-19 vs. peering wars

[Adam Thompson]

Every large ISP does this (or rather, doesn't) at every IX in Canada.  Bell 
isn't unique by any stretch.

It's not in their economic interest to peer at a local IX, because from their 
perspective, the IX takes away business (Managed L2 point-to-point circuits, at 
the very least) from them.

Don't expect the dominant wireline ISP(s) in any region to join local IXes 
anytime soon, sadly, no matter how much it would benefit their customers.  
After all, the customer is always free to purchase service to the IX and join 
the IX, right???  *grumble*

In my local case, if BellMTS joined MBIX, un-cached DNS resolution times could 
potentially drop by 15msec.  That's HUGE.  But the end-user experience is not 
their primary goal.  Their primary goal is profit, as always.

-Adam Thompson
 Founding member, MBIX (once upon a time)

Adam Thompson
Consultant, Infrastructure Services
MERLIN
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athomp...@merlin.mb.ca
www.merlin.mb.ca

> -Original Message-
> From: NANOG  On Behalf Of Sadiq Saif
> Sent: Friday, March 20, 2020 9:38 AM
> To: nanog@nanog.org
> Subject: Re: COVID-19 vs. peering wars
> 
> On Fri, 20 Mar 2020, at 10:31, Steve Mikulasik via NANOG wrote:
> >
> > In Canada the CRTC really needs to get on Canadian ISPs about peering
> > very liberally at IXs in each province. I know of one major
> > institution right now that would have a major work from home issue
> > resolved if one big ISP would peer with one big tier 1 in the IX they
> > are both located at in the same province. Instead traffic needs to
> > flow across the country or to the USA to get back to the same city.
> 
> **cough** Bell Canada **cough**.
> 
> --
>   Sadiq Saif
>   https://sadiqsaif.com/

Re: COVID-19 vs. peering wars

[Sadiq Saif]

On Fri, 20 Mar 2020, at 10:31, Steve Mikulasik via NANOG wrote:
>  
> In Canada the CRTC really needs to get on Canadian ISPs about peering 
> very liberally at IXs in each province. I know of one major institution 
> right now that would have a major work from home issue resolved if one 
> big ISP would peer with one big tier 1 in the IX they are both located 
> at in the same province. Instead traffic needs to flow across the 
> country or to the USA to get back to the same city.

**cough** Bell Canada **cough**.

-- 
  Sadiq Saif
  https://sadiqsaif.com/

RE: COVID-19 vs. peering wars

[Steve Mikulasik via NANOG]

In Canada the CRTC really needs to get on Canadian ISPs about peering very 
liberally at IXs in each province. I know of one major institution right now 
that would have a major work from home issue resolved if one big ISP would peer 
with one big tier 1 in the IX they are both located at in the same province. 
Instead traffic needs to flow across the country or to the USA to get back to 
the same city.


From: NANOG  On Behalf Of Matthew Petach
Sent: Thursday, March 19, 2020 4:15 PM
To: Mike Bolitho 
Cc: NANOG 
Subject: COVID-19 vs. peering wars

CAUTION: This email originated from outside of Civeo.
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.



On Thu, Mar 19, 2020 at 10:27 AM Mike Bolitho 
mailto:mikeboli...@gmail.com>> wrote:
Restoration:

The repair or returning to service of one or more telecommunications services 
that have experienced a service outage or are unusable for any reason, 
including a damaged or impaired telecommunications facility. Such repair or 
returning to service may be done by patching, rerouting, substitution of 
component parts or pathways, and other means, as determined necessary by a 
service vendor.

https://www.cisa.gov/sites/default/files/publications/OEC%20TSP%20Operations%20Guide%20Final%2012062016_FINAL%20508C.pdf

My understanding, and what we did while I worked for a Tier I ISP, was that 
even for degraded circuits we had to do everything in our power to restore to 
full operations. If capacity is an issue and causes TSP coded DIA circuits to 
be unusable then that falls under the "any reason" clause of that line.

- Mike Bolitho

If you're going to bang that drum, the place you're going to get the most 
buck-for-your-bang is using it to force better cooperation between ISPs.

It appears that baking cakes was not sufficient to get recalcitrant players to 
work together.

https://www.flickr.com/photos/mpetach/4031195041

Perhaps a global pandemic may be sufficient to have government begin to 
*compel* networks to interconnect at locations at which they share common 
peering infrastructure?

If you're worried about congestion and performance, that would be the place to 
start pushing.

Matt
staying safely at home away from the flame-fest that may ensue from this.   ^_^;

Re: COVID-19 vs. peering wars

[Matt Erculiani]

Interesting thought, Matt.

I've emailed both of my Senators to inform them of this issue and its
potential impact on the resiliency of the internet (the most infamous
culprit being an operator of root DNS servers, to name a specific example).
I would encourage every NANOG member who cares about this issue to do the
same.

It may be a shot in the dark, but it's a start I guess...

-Matt

On Thu, Mar 19, 2020 at 4:15 PM Matthew Petach 
wrote:

>
>
> On Thu, Mar 19, 2020 at 10:27 AM Mike Bolitho 
> wrote:
>
>> *Restoration:*
>>
>> *The repair or returning to service of one or more telecommunications
>> services that have experienced a service outage or are unusable for any
>> reason, including a damaged or impaired telecommunications facility. Such
>> repair or returning to service may be done by patching, rerouting,
>> substitution of component parts or pathways, and other means, as determined
>> necessary by a service vendor.*
>>
>>
>> https://www.cisa.gov/sites/default/files/publications/OEC%20TSP%20Operations%20Guide%20Final%2012062016_FINAL%20508C.pdf
>>
>>
>> My understanding, and what we did while I worked for a Tier I ISP, was
>> that even for degraded circuits we had to do everything in our power to
>> restore to full operations. If capacity is an issue and causes TSP coded
>> DIA circuits to be unusable then that falls under the "any reason" clause
>> of that line.
>>
>> - Mike Bolitho
>>
>
> If you're going to bang that drum, the place you're going to get the most
> buck-for-your-bang is using it to force better cooperation between ISPs.
>
> It appears that baking cakes was not sufficient to get recalcitrant
> players to work together.
>
> https://www.flickr.com/photos/mpetach/4031195041
>
> Perhaps a global pandemic may be sufficient to have government begin to
> *compel* networks to interconnect at locations at which they share common
> peering infrastructure?
>
> If you're worried about congestion and performance, that would be the
> place to start pushing.
>
> Matt
> staying safely at home away from the flame-fest that may ensue from this.
>   ^_^;
>
>

-- 
Matt Erculiani
ERCUL-ARIN