Re: S.Korea broadband firm sues Netflix after traffic surge

[Justin Streiner]

On Wed, Oct 20, 2021 at 3:41 PM Matthew Walster  wrote:
The user initiates the connection to the CDN. The user is paying for a
level of access to the internet via the BT network, with varying tiers of
speed at particular costs. They are advertised as "Unlimited broadband: With
no data caps or download limits, you can do as much as you like online." on
their website. Many CDNs bring the data closer to the customer, either
embedded within their network, or meeting at various PoPs/IXPs around the
country.

Seems pretty disingenuous to now say the called party has to pay as well,
in stark contrast to decades of precedent with their telephone product,
just because their customers are actually using what they were sold.

All in all, this raises an interesting question. Is British Telecom running
> their networks so hot, that just keeping the lights on requires capacity
> upgrades or are they just looking for freebies?


What happened is pretty clear, and not just for BT or SK.  Those providers,
as a business decision, built their business models around a certain level
of oversubscription that would strike a balance between customers not being
unhappy and squeezing as much headroom out of the network before upgrades
are required (beefier routers/switches, fatter pipes, more peering/transit,
etc).  That business model got upended when that acceptable level of
oversubscription changed.  Video streaming was the puddle of
gasoline/petrol on the floor, and the change in user traffic patterns was
the lit match.

By asking content providers to hand over money to those carriers in
exchange for (better) access to their customers, many of the ISPs could in
fact be triple-dipping because they already get revenue from their
customers and some also get various government subsidies to provide certain
types of service or services in certain areas.

Definitely doesn't pass the sniff test.

Thank you
jms

>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jared Brown]

I don't know what they are putting in the water in Korea, but strange things 
are reported from there.

In addition to the SK Telecom shenanigans, apparently KT can't tell the 
difference between a DDoS and a routing error.

https://en.yna.co.kr/view/AEN20211025006253320


- Jared

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/21/21 20:28, Fred Baker wrote:

I’m not sure I disagree, but let throw in a point of consideration. 
Historically, as you note, the caller pays the toll. However, the 
caller also CHOSE to call, even though the called party might find the 
call irritating. With a CDN, the network is out there hoping to be 
called, and the user makes that choice, “calling” the CDN. Could it 
not be accurately said that the user originated the “call”?


We can't seem to dump our telco ways, can we :-)?

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

Uh, that is what is being said… The user originated the call, so the CDN 
shouldn’t have to pay the user’s ISP to
deliver the replies to the users’ requests.

Owen


> On Oct 21, 2021, at 11:28 , Fred Baker  wrote:
> 
> I’m not sure I disagree, but let throw in a point of consideration. 
> Historically, as you note, the caller pays the toll. However, the caller also 
> CHOSE to call, even though the called party might find the call irritating. 
> With a CDN, the network is out there hoping to be called, and the user makes 
> that choice, “calling” the CDN. Could it not be accurately said that the user 
> originated the “call”?
> 
> Sent using a machine that autocorrects in interesting ways...
> 
>> On Oct 20, 2021, at 12:43 PM, Matthew Walster  wrote:
>> 
>> 
>> Seems pretty disingenuous to now say the called party has to pay as well, in 
>> stark contrast to decades of precedent with their telephone product, just 
>> because their customers are actually using what they were sold.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Thu, 21 Oct 2021, 19:28 Fred Baker,  wrote:

> I’m not sure I disagree, but let throw in a point of consideration.
> Historically, as you note, the caller pays the toll. However, the caller
> also CHOSE to call, even though the called party might find the call
> irritating. With a CDN, the network is out there hoping to be called, and
> the user makes that choice, “calling” the CDN. Could it not be accurately
> said that the user originated the “call”?
>

Yes. The caller is the person at home, the called party is the CDN. Without
the CDN, it then becomes a trunk call.

I'm deliberately ignoring the fact that the connection initiation goes in
one direction and the bulk of the traffic flow goes the other way --
because it wouldn't go at all if it wasn't called up to be requested.
Like... The speaking clock I guess. My analogy long since broke down.

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Fred Baker]

I’m not sure I disagree, but let throw in a point of consideration. 
Historically, as you note, the caller pays the toll. However, the caller also 
CHOSE to call, even though the called party might find the call irritating. 
With a CDN, the network is out there hoping to be called, and the user makes 
that choice, “calling” the CDN. Could it not be accurately said that the user 
originated the “call”?

Sent using a machine that autocorrects in interesting ways...

> On Oct 20, 2021, at 12:43 PM, Matthew Walster  wrote:
> 
> 
> Seems pretty disingenuous to now say the called party has to pay as well, in 
> stark contrast to decades of precedent with their telephone product, just 
> because their customers are actually using what they were sold.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Thu, 21 Oct 2021 at 17:43, Owen DeLong  wrote:

> > On Oct 21, 2021, at 06:30 , Allen McKinley Kitchen (gmail) <
> allenmckinleykitc...@gmail.com> wrote:
> > I totally agree that this is not a perfect analogy. But I have some
> sympathy for both parties in this debate.
>
> Close enough on the analogy, but under the circumstances, I fail to
> understand the need for sympathy for the eyeball
> providers.
>

For the most part, with the advent of adaptive streaming, any congestion
should (not quite that simple at large scale, of course) mean the quality
of the viewed content should only degrade rather than stop working. That
implies that the eyeball networks can wait until links are a lot more
utilised before being upgraded, because there isn't this catastrophe of
streaming media ceasing to work once available bandwidth is exhausted.

This is the UK, though, and so many ISPs have been doing that for a long
time already. In fact, there have been several marketing campaigns over the
years differentiating services solely on "we're faster during the internet
rush hour" and similar.

I'll be quiet now, though, I'm not advancing the discussion in a useful way
any longer :P

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 21, 2021, at 06:30 , Allen McKinley Kitchen (gmail) 
>  wrote:
> 
> 
> 
> ..Allen
> 
>> On Oct 20, 2021, at 15:43, Matthew Walster  wrote, 
>> among other things:
>> 
>> Seems pretty disingenuous to now say the called party has to pay as well, in 
>> stark contrast to decades of precedent with their telephone product, just 
>> because their customers are actually using what they were sold.
> 
> Let me add a (perhaps naïve) perspective:
> 
> When speaking of calling party costs, there are two costs in that model: the 
> cost of access to the network (recurring telephone line charges) and the 
> cost, if metered, of making an actual call.
> 
> My analogy looks something like this: as the manager of a telecom system 
> where hundreds of clinical dictators were calling at all hours, while I did 
> not pay for any individual incoming calls, it was a necessary business 
> expense for me to have sufficient connection to the network – that is to say, 
> a sufficient number of telephone lines. The cost for dozens of T1 lines was 
> not a small budget item.
> 
> So, in one sense, I as a called party did indeed find it necessary to pay 
> substantial fees for the privilege of being called.
> 
> And when I instituted toll-free calling, of course I paid significantly more.

Sure… However, sufficient connection is represented by Netflix investing in 
sufficient PNIs or peering connections
to attach to the eyeball network in question. There is significant expense in 
that and Netflix has done so. Netflix
is also willing (in most cases) to provide equipment at their expense which can 
be placed in the eyeball providers’
networks to further reduce the demands on the provider network as well as 
reducing Netflix costs and improving
the UX for customers that are mutual among the network in question and Netflix.

> I totally agree that this is not a perfect analogy. But I have some sympathy 
> for both parties in this debate.

Close enough on the analogy, but under the circumstances, I fail to understand 
the need for sympathy for the eyeball
providers.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 20, 2021, at 11:53 , Jared Brown  wrote:
> 
> Not to be outdone, British Telecom joins the cephalopod games:
> 
> “Every Tbps (terabit-per-second) of data consumed over and above current 
> levels costs about £50m,” says Marc Allera, the chief executive of BT’s 
> consumer division. “In the last year alone we’ve seen 4Tbps of extra usage 
> and the cost to keep up with that growth is huge.”
> 
> “When the rules were created 25 years ago I don’t think anyone would have 
> envisioned four or five companies would be driving 80% of the traffic on the 
> world’s internet. They aren’t making a contribution to the services they are 
> being carried on; that doesn’t feel right.”
> 
> “A lot of the principles of net neutrality are incredibly valuable, we are 
> not trying to stop or marginalise players but there has to be more effective 
> coordination of demand than there is today”
> 
> https://www.theguardian.com/business/2021/oct/10/squid-games-success-reopens-debate-over-who-should-pay-for-rising-internet-traffic-netflix
> 
> 
> For reference British Telecom has about 10 million broadband subscribers, so 
> apparently those £200m capacity upgrades are stinging.

Back of the napkin, unless I got something very wrong, £200M across 10M 
subscribers is £20 per subscriber as a one-time cost.

Spread across a (very rapid) deprecation schedule of 5 years, that amounts to 
33 pence per subscriber per month.

Hey, BT, charge your customers an extra 50 pence per month and enjoy pocketing 
that £81.6M surplus.

> All in all, this raises an interesting question. Is British Telecom running 
> their networks so hot, that just keeping the lights on requires capacity 
> upgrades or are they just looking for freebies?

Whether or not they are running that hot, the math above makes me think that 
they are being whiney wankers.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Allen McKinley Kitchen (gmail)]

..Allen

> On Oct 20, 2021, at 15:43, Matthew Walster  wrote, among 
> other things:
> 
> Seems pretty disingenuous to now say the called party has to pay as well, in 
> stark contrast to decades of precedent with their telephone product, just 
> because their customers are actually using what they were sold.

Let me add a (perhaps naïve) perspective:

When speaking of calling party costs, there are two costs in that model: the 
cost of access to the network (recurring telephone line charges) and the cost, 
if metered, of making an actual call.

My analogy looks something like this: as the manager of a telecom system where 
hundreds of clinical dictators were calling at all hours, while I did not pay 
for any individual incoming calls, it was a necessary business expense for me 
to have sufficient connection to the network – that is to say, a sufficient 
number of telephone lines. The cost for dozens of T1 lines was not a small 
budget item.

So, in one sense, I as a called party did indeed find it necessary to pay 
substantial fees for the privilege of being called.

And when I instituted toll-free calling, of course I paid significantly more.

I totally agree that this is not a perfect analogy. But I have some sympathy 
for both parties in this debate.

Blessings..

..Allen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Wed, 20 Oct 2021 at 19:53, Jared Brown  wrote:

> “When the rules were created 25 years ago I don’t think anyone would have
> envisioned four or five companies would be driving 80% of the traffic on
> the world’s internet. They aren’t making a contribution to the services
> they are being carried on; that doesn’t feel right.”
>

In the UK, for regular residential geographic telephone numbers, only one
side pays for the call -- the calling party, the one that dials the number.

The user initiates the connection to the CDN. The user is paying for a
level of access to the internet via the BT network, with varying tiers of
speed at particular costs. They are advertised as "Unlimited broadband: With
no data caps or download limits, you can do as much as you like online." on
their website. Many CDNs bring the data closer to the customer, either
embedded within their network, or meeting at various PoPs/IXPs around the
country.

Seems pretty disingenuous to now say the called party has to pay as well,
in stark contrast to decades of precedent with their telephone product,
just because their customers are actually using what they were sold.

All in all, this raises an interesting question. Is British Telecom running
> their networks so hot, that just keeping the lights on requires capacity
> upgrades or are they just looking for freebies?


Taking advantage of a situation and jumping on the bandwagon, some would
say. After decades of chronic underinvestment in UK broadband, they're
finally starting to offer competitive products, and aren't used to having
to pay for it -- though as it happens, it would appear the public purse is
picking up the costs anyway:
https://www.ispreview.co.uk/index.php/2020/08/govs-1bn-helps-493600-uk-premises-get-gigabit-broadband.html

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jared Brown]

Not to be outdone, British Telecom joins the cephalopod games:

“Every Tbps (terabit-per-second) of data consumed over and above current levels 
costs about £50m,” says Marc Allera, the chief executive of BT’s consumer 
division. “In the last year alone we’ve seen 4Tbps of extra usage and the cost 
to keep up with that growth is huge.”

“When the rules were created 25 years ago I don’t think anyone would have 
envisioned four or five companies would be driving 80% of the traffic on the 
world’s internet. They aren’t making a contribution to the services they are 
being carried on; that doesn’t feel right.”

“A lot of the principles of net neutrality are incredibly valuable, we are not 
trying to stop or marginalise players but there has to be more effective 
coordination of demand than there is today”

https://www.theguardian.com/business/2021/oct/10/squid-games-success-reopens-debate-over-who-should-pay-for-rising-internet-traffic-netflix


For reference British Telecom has about 10 million broadband subscribers, so 
apparently those £200m capacity upgrades are stinging.

All in all, this raises an interesting question. Is British Telecom running 
their networks so hot, that just keeping the lights on requires capacity 
upgrades or are they just looking for freebies?


- Jared

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 19, 2021, at 08:47 , Tom Beecher  wrote:
> 
> Vs. an ISP that is causing the problem or trying to run a protection racket 
> against content providers, I think it wouldn’t be hard for the content
> provider to supply appropriate messaging inserted at the front end of 
> playback to explain the situation to their mutual customers. Instead of the
> typical FBI notice, imagine the movie starting with an ad that explains how 
> the ISP is trying to increase consumer costs by forcing Netflix to
> pass along additional fees paid to the ISP to deliver content the customer 
> has already paid said same ISP to deliver.
> 
> Wouldn't be hard, but doubtful it would be effective. 
> 
> Consumers already get the same message on a few TV channels during the annual 
> carriage dispute-a-palooza, with both sides telling them to call the other 
> one to complain. It clearly doesn't work. 

I don’t think that’s as commonplace in S. Korea as it is here.

It appears that the Netflix Verizon notices had the desired effect.

> Outside of our sphere, nobody cares about this stuff. They just want their 
> thing to work. 

Agreed… The trick is which side is able to convince the users that the other is 
the one preventing that.

Owen

> 
> On Mon, Oct 18, 2021 at 9:37 PM Owen DeLong via NANOG  > wrote:
> 
> 
> > On Oct 18, 2021, at 14:48 , Jay Hennigan  > > wrote:
> > 
> > On 10/18/21 07:02, Josh Luthman wrote:
> > 
> >>Netflix, as an example, has even been willing to bear most of the cost
> >>with peering or bringing servers to ISPs to reduce the ISP's costs and
> >>improve the ISP customer's experience.
> > 
> > Netflix doesn't do those things because it cares about the ISP's costs and 
> > the ISP customers' experience.
> > 
> > Netflix does these things because Netflix cares about Netflix's costs and 
> > Netflix's customers' experience.
> 
> Of course, that doesn’t change the fact that it does lower the ISP’s costs 
> and improve the ISP customers’ experience.
> 
> >>It's about time Netflix played
> >>chicken with one of these ISPs and stopped offering service  (or
> >>offered
> >>limited service) to the ISPs that try to extort them and other content
> >>providers:
> > 
> > Then Netflix would risk losing those customers, especially if the ISP in 
> > question is a cable company or offers its own video streaming services.
> 
> Vs. an ISP that is causing the problem or trying to run a protection racket 
> against content providers, I think it wouldn’t be hard for the content
> provider to supply appropriate messaging inserted at the front end of 
> playback to explain the situation to their mutual customers. Instead of the
> typical FBI notice, imagine the movie starting with an ad that explains how 
> the ISP is trying to increase consumer costs by forcing Netflix to
> pass along additional fees paid to the ISP to deliver content the customer 
> has already paid said same ISP to deliver.
> 
> Somehow, I don’t see the ISP doing well against such a PR onslaught.
> 
> > Also, by peering and bringing servers to ISPs, Netflix improves its 
> > customers' experience and reduces Netflix's costs because they no longer 
> > need to pay a transit provider to deliver content.
> 
> Where the ISP in question isn’t trying to force them to pay transit costs 
> within said eyeball network, sure. But in SK’s case, it looks like they’re 
> trying to force Netflix to pay to reach their eyeballs, even though the 
> eyeballs in question are already paying them to deliver Netflix (and other) 
> content.
> 
> >>Sorry, your service provider does not believe in net
> >>neutrality and has imposed limitations on your Netflix experience.
> > 
> > They actually did pretty much exactly that with Verizon back in 2014.
> > 
> > https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-over-slow-data-speeds/
> >  
> > 
> 
> It appears to have worked out fairly well for them, too.
> 
> Owen
> 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Tom Beecher]

>
> Vs. an ISP that is causing the problem or trying to run a protection
> racket against content providers, I think it wouldn’t be hard for the
> content
> provider to supply appropriate messaging inserted at the front end of
> playback to explain the situation to their mutual customers. Instead of the
> typical FBI notice, imagine the movie starting with an ad that explains
> how the ISP is trying to increase consumer costs by forcing Netflix to
> pass along additional fees paid to the ISP to deliver content the customer
> has already paid said same ISP to deliver.
>

Wouldn't be hard, but doubtful it would be effective.

Consumers already get the same message on a few TV channels during the
annual carriage dispute-a-palooza, with both sides telling them to call the
other one to complain. It clearly doesn't work.

Outside of our sphere, nobody cares about this stuff. They just want their
thing to work.

On Mon, Oct 18, 2021 at 9:37 PM Owen DeLong via NANOG 
wrote:

>
>
> > On Oct 18, 2021, at 14:48 , Jay Hennigan  wrote:
> >
> > On 10/18/21 07:02, Josh Luthman wrote:
> >
> >>Netflix, as an example, has even been willing to bear most of the
> cost
> >>with peering or bringing servers to ISPs to reduce the ISP's costs
> and
> >>improve the ISP customer's experience.
> >
> > Netflix doesn't do those things because it cares about the ISP's costs
> and the ISP customers' experience.
> >
> > Netflix does these things because Netflix cares about Netflix's costs
> and Netflix's customers' experience.
>
> Of course, that doesn’t change the fact that it does lower the ISP’s costs
> and improve the ISP customers’ experience.
>
> >>It's about time Netflix played
> >>chicken with one of these ISPs and stopped offering service  (or
> >>offered
> >>limited service) to the ISPs that try to extort them and other
> content
> >>providers:
> >
> > Then Netflix would risk losing those customers, especially if the ISP in
> question is a cable company or offers its own video streaming services.
>
> Vs. an ISP that is causing the problem or trying to run a protection
> racket against content providers, I think it wouldn’t be hard for the
> content
> provider to supply appropriate messaging inserted at the front end of
> playback to explain the situation to their mutual customers. Instead of the
> typical FBI notice, imagine the movie starting with an ad that explains
> how the ISP is trying to increase consumer costs by forcing Netflix to
> pass along additional fees paid to the ISP to deliver content the customer
> has already paid said same ISP to deliver.
>
> Somehow, I don’t see the ISP doing well against such a PR onslaught.
>
> > Also, by peering and bringing servers to ISPs, Netflix improves its
> customers' experience and reduces Netflix's costs because they no longer
> need to pay a transit provider to deliver content.
>
> Where the ISP in question isn’t trying to force them to pay transit costs
> within said eyeball network, sure. But in SK’s case, it looks like they’re
> trying to force Netflix to pay to reach their eyeballs, even though the
> eyeballs in question are already paying them to deliver Netflix (and other)
> content.
>
> >>Sorry, your service provider does not believe in net
> >>neutrality and has imposed limitations on your Netflix experience.
> >
> > They actually did pretty much exactly that with Verizon back in 2014.
> >
> >
> https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-over-slow-data-speeds/
>
> It appears to have worked out fairly well for them, too.
>
> Owen
>
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 18, 2021, at 14:48 , Jay Hennigan  wrote:
> 
> On 10/18/21 07:02, Josh Luthman wrote:
> 
>>Netflix, as an example, has even been willing to bear most of the cost
>>with peering or bringing servers to ISPs to reduce the ISP's costs and
>>improve the ISP customer's experience.
> 
> Netflix doesn't do those things because it cares about the ISP's costs and 
> the ISP customers' experience.
> 
> Netflix does these things because Netflix cares about Netflix's costs and 
> Netflix's customers' experience.

Of course, that doesn’t change the fact that it does lower the ISP’s costs and 
improve the ISP customers’ experience.

>>It's about time Netflix played
>>chicken with one of these ISPs and stopped offering service  (or
>>offered
>>limited service) to the ISPs that try to extort them and other content
>>providers:
> 
> Then Netflix would risk losing those customers, especially if the ISP in 
> question is a cable company or offers its own video streaming services.

Vs. an ISP that is causing the problem or trying to run a protection racket 
against content providers, I think it wouldn’t be hard for the content
provider to supply appropriate messaging inserted at the front end of playback 
to explain the situation to their mutual customers. Instead of the
typical FBI notice, imagine the movie starting with an ad that explains how the 
ISP is trying to increase consumer costs by forcing Netflix to
pass along additional fees paid to the ISP to deliver content the customer has 
already paid said same ISP to deliver.

Somehow, I don’t see the ISP doing well against such a PR onslaught.

> Also, by peering and bringing servers to ISPs, Netflix improves its 
> customers' experience and reduces Netflix's costs because they no longer need 
> to pay a transit provider to deliver content.

Where the ISP in question isn’t trying to force them to pay transit costs 
within said eyeball network, sure. But in SK’s case, it looks like they’re 
trying to force Netflix to pay to reach their eyeballs, even though the 
eyeballs in question are already paying them to deliver Netflix (and other) 
content.

>>Sorry, your service provider does not believe in net
>>neutrality and has imposed limitations on your Netflix experience.
> 
> They actually did pretty much exactly that with Verizon back in 2014.
> 
> https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-over-slow-data-speeds/

It appears to have worked out fairly well for them, too.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jay Hennigan]

On 10/18/21 07:02, Josh Luthman wrote:


Netflix, as an example, has even been willing to bear most of the cost
with peering or bringing servers to ISPs to reduce the ISP's costs and
improve the ISP customer's experience.


Netflix doesn't do those things because it cares about the ISP's costs 
and the ISP customers' experience.


Netflix does these things because Netflix cares about Netflix's costs 
and Netflix's customers' experience.



It's about time Netflix played
chicken with one of these ISPs and stopped offering service  (or
offered
limited service) to the ISPs that try to extort them and other content
providers:


Then Netflix would risk losing those customers, especially if the ISP in 
question is a cable company or offers its own video streaming services.


Also, by peering and bringing servers to ISPs, Netflix improves its 
customers' experience and reduces Netflix's costs because they no longer 
need to pay a transit provider to deliver content.



Sorry, your service provider does not believe in net
neutrality and has imposed limitations on your Netflix experience.


They actually did pretty much exactly that with Verizon back in 2014.

https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-over-slow-data-speeds/

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mike Hammett]

"at some point it just doesn't matter and becomes marketing hype." 

There is A LOT of hype over increasing broadband speeds, so much so to the 
point where immense oversubscription is the only practical way forward, then 
people piss and moan that ISPs didn't build enough to keep up with non-existent 
(at the time) demand. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Michael Thomas"  
To: nanog@nanog.org 
Sent: Sunday, October 10, 2021 3:13:50 PM 
Subject: Re: S.Korea broadband firm sues Netflix after traffic surge 




On 10/10/21 12:57 PM, Mark Tinka wrote: 





On 10/10/21 21:33, Matthew Petach wrote: 






If you sell a service for less than it costs to provide, simply 
based on the hopes that people won't actually *use* it, that's 
called "gambling", and I have very little sympathy for businesses 
that gamble and lose. 


You arrived at the crux of the issue, quickly, which was the basis of my 
initial response last week - infrastructure is dying. And we simply aren't 
motivated enough to figure it out. 

When you spend 25+ years sitting in a chair waiting for the phone to ring or 
the door to open, for someone to ask, "How much for 5Mbps?", your misfortune 
will never be your own fault. 




Isn't that what Erlang numbers are all about? My suspicion is that after about 
100Mbs most people wouldn't notice the difference in most cases. My ISP is 
about 25Mbs on a good day (DSL) and it serves our needs fine and have never run 
into bandwidth constraints. Maybe if we were streaming 4k all of the time it 
might be different, but frankly the difference for 4k isn't all that big. It's 
sort of like phone screen resolution: at some point it just doesn't matter and 
becomes marketing hype. 
Mike 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Blake Hudson]

Imagine it's 2021. Over a decade ago the world started a transition from 
captive audio broadcast media from a single source towards unicast 
streaming from multiple sources. You operate an ISP network that was 
designed for a past era and you have been slow to keep up with your 
competitors or with the changing trends. Customers are not happy. Your 
customers don't understand. People don't understand. You are a cog in 
the machine that is causing resistance and see an opportunity to get 
paid twice for a single job. You won't get out of the way once paid, in 
fact you'll grasp at your position even harder to ensure that you will 
continue to get paid. You are SK Telecom.



On 10/18/2021 9:02 AM, Josh Luthman wrote:
Imagine it's 2008 and your AP is pushing out 3 mbps. Customers are all 
happy.  Suddenly, Netflix demands 10x what you're offering.  Customers 
are not happy.


Customers don't understand.  People don't understand. There are a 
million cogs in the machine and if the path of least resistance is to 
turn left, an ISP is going to turn left.


Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Fri, Oct 1, 2021 at 10:19 AM Blake Hudson  wrote:



On 10/1/2021 8:48 AM, Sean Donelan wrote:
> South Korean Internet service provider SK Broadband has sued
Netflix
> to pay for costs from increased network traffic and maintenance
work
> because of a surge of viewers to the U.S. firm's content, an SK
> spokesperson said on Friday.
> [...]
> Last year, Netflix had brought its own lawsuit on whether it had
any
> obligation to pay SK for network usage, arguing Netflix's duty ends
> with creating content and leaving it accessible. It said SK's
expenses
> were incurred while fulfilling its contractual obligations to
Internet
> users, and delivery in the Internet world is "free of charge as a
> principle", according to court documents.
> [...]
>
>

https://www.reuters.com/business/media-telecom/skorea-broadband-firm-sues-netflix-after-traffic-surge-squid-game-2021-10-01/

>
>

I'll never understand over how ISPs see content providers as the
enemy
(or a rival). The content is why ISPs have customers. Don't get upset
when your customer uses the service that you sold them (in a way
that is
precisely in accordance with the expected usage)!

Netflix, as an example, has even been willing to bear most of the
cost
with peering or bringing servers to ISPs to reduce the ISP's costs
and
improve the ISP customer's experience. It's about time Netflix played
chicken with one of these ISPs and stopped offering service (or
offered
limited service) to the ISPs that try to extort them and other
content
providers: Sorry, your service provider does not believe in net
neutrality and has imposed limitations on your Netflix experience.
For a
better Netflix experience, consider exploring one of these other
nearby
internet providers: x, y, z.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Josh Luthman]

Imagine it's 2008 and your AP is pushing out 3 mbps. Customers are all
happy.  Suddenly, Netflix demands 10x what you're offering.  Customers are
not happy.

Customers don't understand.  People don't understand.  There are a million
cogs in the machine and if the path of least resistance is to turn left, an
ISP is going to turn left.

Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Fri, Oct 1, 2021 at 10:19 AM Blake Hudson  wrote:

>
>
> On 10/1/2021 8:48 AM, Sean Donelan wrote:
> > South Korean Internet service provider SK Broadband has sued Netflix
> > to pay for costs from increased network traffic and maintenance work
> > because of a surge of viewers to the U.S. firm's content, an SK
> > spokesperson said on Friday.
> > [...]
> > Last year, Netflix had brought its own lawsuit on whether it had any
> > obligation to pay SK for network usage, arguing Netflix's duty ends
> > with creating content and leaving it accessible. It said SK's expenses
> > were incurred while fulfilling its contractual obligations to Internet
> > users, and delivery in the Internet world is "free of charge as a
> > principle", according to court documents.
> > [...]
> >
> >
> https://www.reuters.com/business/media-telecom/skorea-broadband-firm-sues-netflix-after-traffic-surge-squid-game-2021-10-01/
> >
> >
>
> I'll never understand over how ISPs see content providers as the enemy
> (or a rival). The content is why ISPs have customers. Don't get upset
> when your customer uses the service that you sold them (in a way that is
> precisely in accordance with the expected usage)!
>
> Netflix, as an example, has even been willing to bear most of the cost
> with peering or bringing servers to ISPs to reduce the ISP's costs and
> improve the ISP customer's experience. It's about time Netflix played
> chicken with one of these ISPs and stopped offering service  (or offered
> limited service) to the ISPs that try to extort them and other content
> providers: Sorry, your service provider does not believe in net
> neutrality and has imposed limitations on your Netflix experience. For a
> better Netflix experience, consider exploring one of these other nearby
> internet providers: x, y, z.
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Tom Beecher]

I agree with you generally.

It's not impossible, but probably unlikely for an individual to be sued for
contents of cookie data or similar small fragments like that.

I do believe it's orders of more magnitude more likely for the 'average'
residential consumer to attract a suit from the MPAA/RIAA/etc because there
is a torrent stream emanating from their connection, and I have little
faith that any provider would go out of their way to jump in front and say
'no no, that's our tech'.

On Tue, Oct 12, 2021 at 5:15 PM Matthew Petach 
wrote:

>
>
> On Tue, Oct 12, 2021 at 2:01 PM Tom Beecher  wrote:
>
>> I think it would be absolutely *stunning* for content providers
>>> to turn the model on its head; use a bittorrent like model for
>>> caching and serving content out of subscribers homes at
>>> recalcitrant ISPs, so that data doesn't come from outside,
>>> it comes out of the mesh within the eyeball network, with
>>> no clear place for the ISP to stick a $$$ bill to.
>>>
>>
>> I'm familiar with some work and ideas that have gone into such a thing,
>> and I'm personally very much against it for non-technical reasons.
>>
>> Given how far the law lags behind technology, the last thing anyone
>> should be ok with is a 3rd party storing bits on ANYTHING in their house,
>> or transmitting those bits from a network connection that is registered to
>> them.
>>
>
> *chortle*
>
> So, I take it you steadfastly block *all* cookies from being stored
> or transmitted from your browser at home?
>
> Oh, wait.  You meant it's OK to let some third parties
> store and transmit bits from your devices, but only
> the ones you like and support, and as long as they're
> small bits, and you're sure there's nothing harmful or
> illegal in them.
>
> So, that means you check each cookie to make sure
> there's nothing in them that could be illegal?
>
> You sure someone hasn't tucked something like
> the DeCSS algorithm, or the RSA algorithm into
> a cookie in your browser, like this?
>
> https://commons.wikimedia.org/wiki/File:Munitions_T-shirt_(front).jpg
>
> https://www.cafepress.com/+,954530397?utm_medium=cpc_source=pla-google_campaign=7979505756-d-c_content=83814261273-adid-395151690662_term=pla-1396845372217-pid-954530397=Cj0KCQjw5JSLBhCxARIsAHgO2SeM10JbFgeus96hEedn0d0m2Kkz6Z91-frlEIUh-3ZD2w89j8EUmCsaAvnAEALw_wcB
>
> The fact of the matter is, every one of us allows
> third parties to store data on all our devices, all
> the time, and send it back out on the network,
> completely unsupervised by us, even though
> it could contain data which is illegal to cross
> certain arbitrary political boundaries.
>
> I understand where you're coming from, I really
> do.
>
> But I don't think people stop and think about just
> how completely that ship has sailed, from a legal
> standpoint.  You could have been asked by a random
> website to store code which is illegal to export in a
> cookie which is then offered back up to any other
> website in whatever jurisdiction around the globe
> that asks for it, and you'll be completely unaware
> of it, because we've all gotten past the point of "ask
> me about every cookie" being a workable setting on
> any of our devices.
>
> Go ahead.  Turn off all cookie support on all your devices
> for 24 hours.  Don't let any of that third party data in or out
> of your home during that time.
>
> Let me know how well that turns out.
>
> Bonus points if you enforce it on your family/spouse/SO/partner
> at the same time, and they're still talking to you at the end of the
> 24 hours.  ;-P
>
> Matt
>
>
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[scott]

On 10/13/21 2:39 AM, Doug Barton wrote:

On the cookie issue, I have had very good luck with this in Firefox:

https://addons.mozilla.org/en-US/firefox/addon/cookie-autodelete/
-



Nice, I have the settings to delete all history and cookies when I close 
the browser as well as remove them all the time while I am using it.



I don't want to leave Firefox because of NoScript.  That stops a lot of 
snooping.  Too bad it doesn't work for other browsers like Vivaldi.  I 
would switch in a heartbeat because the dirty stuff Mozilla Location 
Services does is ugly.



scott




hope this helps,

Doug


On 10/12/21 6:26 AM, scott wrote:


On 10/12/21 9:15 PM, Matthew Petach wrote:


So, I take it you steadfastly block *all* cookies from being stored
or transmitted from your browser at home?
--\



I used to when Firefox had the "ask me every time" for cookies. They 
got rid of that, so now I clear them out all the time. Many times a 
day and every time I close the browser... :)


Then I found out about Mozilla Location Services, how they made it so 
we can't block that and realized they only blocked others and not 
themselves from feasting on our data.


https://en.wikipedia.org/wiki/Mozilla_Location_Services

https://location.services.mozilla.com

Bastards!

scott

Re: S.Korea broadband firm sues Netflix after traffic surge

[Doug Barton]

On the cookie issue, I have had very good luck with this in Firefox:

https://addons.mozilla.org/en-US/firefox/addon/cookie-autodelete/

hope this helps,

Doug


On 10/12/21 6:26 AM, scott wrote:


On 10/12/21 9:15 PM, Matthew Petach wrote:


So, I take it you steadfastly block *all* cookies from being stored
or transmitted from your browser at home?
--\



I used to when Firefox had the "ask me every time" for cookies. They got 
rid of that, so now I clear them out all the time.  Many times a day and 
every time I close the browser... :)


Then I found out about Mozilla Location Services, how they made it so we 
can't block that and realized they only blocked others and not 
themselves from feasting on our data.


https://en.wikipedia.org/wiki/Mozilla_Location_Services

https://location.services.mozilla.com

Bastards!

scott

Re: S.Korea broadband firm sues Netflix after traffic surge

[scott]

On 10/12/21 9:15 PM, Matthew Petach wrote:


So, I take it you steadfastly block *all* cookies from being stored
or transmitted from your browser at home?
--\



I used to when Firefox had the "ask me every time" for cookies. They got 
rid of that, so now I clear them out all the time.  Many times a day and 
every time I close the browser... :)


Then I found out about Mozilla Location Services, how they made it so we 
can't block that and realized they only blocked others and not 
themselves from feasting on our data.


https://en.wikipedia.org/wiki/Mozilla_Location_Services

https://location.services.mozilla.com

Bastards!

scott

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Tue, Oct 12, 2021 at 2:01 PM Tom Beecher  wrote:

> I think it would be absolutely *stunning* for content providers
>> to turn the model on its head; use a bittorrent like model for
>> caching and serving content out of subscribers homes at
>> recalcitrant ISPs, so that data doesn't come from outside,
>> it comes out of the mesh within the eyeball network, with
>> no clear place for the ISP to stick a $$$ bill to.
>>
>
> I'm familiar with some work and ideas that have gone into such a thing,
> and I'm personally very much against it for non-technical reasons.
>
> Given how far the law lags behind technology, the last thing anyone should
> be ok with is a 3rd party storing bits on ANYTHING in their house, or
> transmitting those bits from a network connection that is registered to
> them.
>

*chortle*

So, I take it you steadfastly block *all* cookies from being stored
or transmitted from your browser at home?

Oh, wait.  You meant it's OK to let some third parties
store and transmit bits from your devices, but only
the ones you like and support, and as long as they're
small bits, and you're sure there's nothing harmful or
illegal in them.

So, that means you check each cookie to make sure
there's nothing in them that could be illegal?

You sure someone hasn't tucked something like
the DeCSS algorithm, or the RSA algorithm into
a cookie in your browser, like this?

https://commons.wikimedia.org/wiki/File:Munitions_T-shirt_(front).jpg
https://www.cafepress.com/+,954530397?utm_medium=cpc_source=pla-google_campaign=7979505756-d-c_content=83814261273-adid-395151690662_term=pla-1396845372217-pid-954530397=Cj0KCQjw5JSLBhCxARIsAHgO2SeM10JbFgeus96hEedn0d0m2Kkz6Z91-frlEIUh-3ZD2w89j8EUmCsaAvnAEALw_wcB

The fact of the matter is, every one of us allows
third parties to store data on all our devices, all
the time, and send it back out on the network,
completely unsupervised by us, even though
it could contain data which is illegal to cross
certain arbitrary political boundaries.

I understand where you're coming from, I really
do.

But I don't think people stop and think about just
how completely that ship has sailed, from a legal
standpoint.  You could have been asked by a random
website to store code which is illegal to export in a
cookie which is then offered back up to any other
website in whatever jurisdiction around the globe
that asks for it, and you'll be completely unaware
of it, because we've all gotten past the point of "ask
me about every cookie" being a workable setting on
any of our devices.

Go ahead.  Turn off all cookie support on all your devices
for 24 hours.  Don't let any of that third party data in or out
of your home during that time.

Let me know how well that turns out.

Bonus points if you enforce it on your family/spouse/SO/partner
at the same time, and they're still talking to you at the end of the
24 hours.  ;-P

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Tom Beecher]

>
> I think it would be absolutely *stunning* for content providers
> to turn the model on its head; use a bittorrent like model for
> caching and serving content out of subscribers homes at
> recalcitrant ISPs, so that data doesn't come from outside,
> it comes out of the mesh within the eyeball network, with
> no clear place for the ISP to stick a $$$ bill to.
>

I'm familiar with some work and ideas that have gone into such a thing, and
I'm personally very much against it for non-technical reasons.

Given how far the law lags behind technology, the last thing anyone should
be ok with is a 3rd party storing bits on ANYTHING in their house, or
transmitting those bits from a network connection that is registered to
them.



On Mon, Oct 11, 2021 at 4:06 PM Matthew Petach 
wrote:

>
>
> On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka  wrote:
>
>> However, in an era where content is making a push to get as close to the
>> eyeballs as possible, kit getting cheaper and faster because of merchant
>> silicon, and abundance of aggregated capacity at exchange points, can we
>> leverage the shorter, faster links to change the model?
>>
>> Mark.
>>
>
> I think it would be absolutely *stunning* for content providers
> to turn the model on its head; use a bittorrent like model for
> caching and serving content out of subscribers homes at
> recalcitrant ISPs, so that data doesn't come from outside,
> it comes out of the mesh within the eyeball network, with
> no clear place for the ISP to stick a $$$ bill to.
>
> Imagine you've got a movie; you slice it into 1,000
> encrypted chunks; you make part of your license
> agreement for customers a requirement that they
> will allow you to use up to 20GB of disk space on
> their computer and to serve up data chunks into
> the network in return for a slightly cheaper monthly
> subscription cost to your service.  You put 1 slice
> of that movie on each of 1,000 customers in a
> network; then you replicate that across the next
> thousand customers, and again for the next
> thousand, until you've got enough replicas of
> each shard to handle a particular household
> going offline.  Your library is still safe from
> piracy, no household has more than 1/1000th of a
> movie locally, and they don't have the key to decrypt
> it anyhow; but they've got 1/1000th of 4, different
> movies, and when someone in that ISP wants to watch the
> movie, the chunks are being fetched from other households
> within the eyeball network.  The content provider would have
> shard servers in region, able to serve up any missing shards that
> can't be fetched locally within the ISP--but the idea would be that
> as the number of subscribers within an ISP goes up, instead of the
> ISP seeing a large, single-point-source increase in traffic, what they
> see is an overall increase in east-west traffic among their users.
>
> Because the "serving of shards to others" happens primarily while the
> user is actively streaming content, you have a natural bell curve; during
> peak streaming times, you have more nodes active to serve up shards,
> handling the increased demand; at lower demand times, when fewer
> people are active, and there's fewer home-nodes to serve shards, the
> content network's shard servers can pick up the slack...but that'll
> generally
> only happen during lower traffic times, when the traffic won't be
> competing
> and potentially causing pain for the ISP.
>
> Really, it seems like a win-win scenario.
>
> I'm confident we'll see a content network come out with a model like this
> within the next 5 years, at which point the notion of blackmailing content
> networks for additional $$$s will be a moot point, because the content
> will
> be distributed and embedded within every major eyeball network already,
> whether they like it or not, on their customer's devices.
>
> Let's check back in 2026, and see if someone's become fantastically
> successful doing this or not.  ;)
>
> Thanks!
>
> Matt
>
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Tue, 12 Oct 2021, 02:24 Owen DeLong,  wrote:

>
> A 4K 2 hour movie is about 40GB. Most modern smart TVs around 32GB of RAM
> and can probably devote about 20GB of that to buffering a stream, so yeah,
> that should actually be doable.
>

Most users are not streaming 4K, it's a very small fraction of streams.
Even those that do tend to have only about 4-8GB of flash, which is about
half used by the operating system alone. RAM-wise, you're definitely off by
at least an order of magnitude there.

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Tue, Oct 12, 2021 at 8:16 AM Jared Brown  wrote:

> Mark Tinka wrote:

[...]

>
> > But I doubt that
> > will work, unless someone can think up a clever way to modify BitTorrent
> > to suit today's network architectures.
>   Unless network topology is somehow exposed, this isn't possible. All
> anybody can do is use latency, IP and ASN information as a proxy.
>
>   Nothing is stopping a BitTorrent client from being selective about its
> peers. The current peer selection algorithm optimizes for throughput, not
> adjecency or topology.
>

Thank you to everyone who pointed out this has
already been tried in the past--I wasn't aware of
it, but it stands to reason.  By the time I think of
something as a good idea, there's a high probability
it's already been done somewhere.  ;)

In terms of exposing network topology, remember the
clients get their information on what chunks to fetch
from whom from the tracker.  As each client connects
to the tracker to report what chunks it has, the tracker
can build a mapping of client IP to ASN, coupled with
latency.  For added fanciness, a traceroute towards the
client's public IP can be performed, and then clusters
can be mapped of clients with the highest numbers
of common elements in the traceroute path back,
which would give you a measure of network topological
"closeness".

That is, if the traceroutes to client A and client B are the
same for 12 out of 15 hops, but the traceroutes to client A
and client C are only the same for 8 out of 15 hops, we have
a good hint that client A and client B are probably topologically
closer than client A and client C, and therefore when client A makes
a request for chunks of movie 1, if both client B and client C have
relevant chunks, we would provide client A with client B's information
preferentially over client C's information.

That way, the tracker can help cluster data transfers in roughly
topological "closeness".  Over time, you can build up a more and
more accurate topological map as you collect path information
from each tracker back to each client.  For added points, since
we're talking about subscription-based content delivery, associate
each client's IP address(es) with the subscriber login information
and now you have a mapping of where that subscriber watches
content from, over time.  Knowing their viewing history would
allow you to get an idea of what they're likely to watch next, and
where in the network they're likely to watch it, and you can nudge
your pre-seeding of chunks of the next-most-likely-to-be-watched
content to other clients topologically near to where the subscriber
is most likely going to be connected when they want to watch that.

...but perhaps I'm getting a bit too far into the "creepy" factor at
this point.   ^_^;


> - Jared
>
>
Thanks!

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 12, 2021, at 09:04 , Jared Brown  wrote:
> 
> Doug Barton wrote:
>> One incentive I haven't seen anyone mention is that ISPs don't want to 
>> charge customers what it really costs to provide them access. 
>  For the sake of argument, let's assume this is true.
> 
>  For this to work, I am really trying hard to ignore inconvenient facts like:
> 
>  "South Korea’s SK Telecom (SKT) has reported operating revenues of 
>   KRW4.818 trillion (USD4.2 billion) for the quarter ended 30 June 2021,
>   up 4.7% year-on-year, with it saying that the increase was ‘due to 
>   continued solid growth trends in all business areas’.
> 
>   SKT’s operating income in Q2 2021 totalled KRW397 billion, up 10.8% on
>   an annualised basis..."
> 
>   
> https://www.commsupdate.com/articles/2021/08/12/sk-telecom-reports-revenue-increase-in-2q21-as-5g-subscriber-numbers-rise/
> 
>  Nevertheless, let's go with the hypothesis that service is provided below 
> cost.
> 
>  Providing access is mostly fixed costs, as there are very few consumables in 
> running a network.
> 
>  IP transit costs aren't an issue, since Netflix will do settlement free 
> peering.
> 
>  This leaves the internal network of SK Telecom as the problem and cost 
> center.
> 
>  There would have been no marginal cost if SK Telecom's own network was 
> capable of handling the traffic 
>  of its customers.
> 
>  So basically SK Telecom is mad at Netflix for forcing equipment upgrades 
> faster than budgeted.
> 
>  Should Netflix have to pay for SK Telecom sucking at traffic planning and 
> budgeting?

Apparently SK Telecom thinks so.

Not surprising since they certainly aren’t the first eyeball ISP to put forth 
this notion.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 12, 2021, at 08:13 , Jared Brown  wrote:
> 
> Mark Tinka wrote:
>> Someone can correct me if I'm wrong, but the way I know BitTorrent to
>> work is the file is downloaded to disk, unarchived and then listed as
>> ready to watch.
>  That's not how it works. Several streaming BitTorrent clients specifically 
> request blocks in order so that you can start watching immediately.
>  Not that you need a special client, it works pretty well with the standard 
> client as well on a well seeded torrent, as blocks are generally requested 
> more or less in order.
> 
>> It also assumes the device has all the necessary apps
>> and codecs needed to render the file.
>  Well, yes. Or you could just stream content that is guaranteed to be 
> compatible with the device used.
> 
>> On the other hand, BitTorrent could just make an Apple
>> TV/PS4/PS5/Xbox/whatever-device-you-use app as well.
>  They could, and they might even have, I forget, but there is little demand 
> for such a thing as a centralized CDN strategy works better.
> 
>> But I doubt that
>> will work, unless someone can think up a clever way to modify BitTorrent
>> to suit today's network architectures.
>  Unless network topology is somehow exposed, this isn't possible. All anybody 
> can do is use latency, IP and ASN information as a proxy.

Well, latency + measurements of e2e bandwidth and possibly IP+ASN information.

However, in reality, if you are trying to optimize your ability to receive 
data, latency + e2e bandwidth are pretty good assuming they don’t
vary greatly (which is, admittedly a problem, they do, and worse, any 
client-level continuous measurement at scale will affect the experiment
in a very negative way).

>  Nothing is stopping a BitTorrent client from being selective about its 
> peers. The current peer selection algorithm optimizes for throughput, not 
> adjecency or topology.

Is that bad?

It might be suboptimal for the eyeball ISP, but it seems to me that it’s 
probably optimal for everyone else involved.


Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 12, 2021, at 06:45 , Mark Tinka  wrote:
> 
> 
> 
> On 10/11/21 22:57, Matthew Walster wrote:
> 
>> Ignoring for the moment that P2P is inherently difficult to stream with 
>> (you're usually downloading chunks in parallel, and with devices like Smart 
>> TVs etc you don't really have the storage to do so anyway) 
>> there's also the problem that things like BitTorrent don't know network 
>> topology and therefore only really increases the cross-sectional bandwidth 
>> required.
>> 
>> Not to mention that it has been tried before, and didn't work then either.
> 
> Yeah, and people also want to click a title and start watching immediately.
> 
> Someone can correct me if I'm wrong, but the way I know BitTorrent to work is 
> the file is downloaded to disk, unarchived and then listed as ready to watch. 
> It also assumes the device has all the necessary apps and codecs needed to 
> render the file.

It can work this way and usually does if the format of the file being 
transmitted is an archive (e.g. rar, tar) or compressed file (e.g. gzip) or 
both (e.g. zip).

OTOH, since MPEG is already a compressed format and little is gained by further 
attempting to compress it (usually the opposite), it can be sent as a stream. 
BitTorrent can handle streams and there are clients that will start playback as 
soon as a sufficient fraction of the early portion of the stream has arrived 
while continuing to download the remainder of the
file in the background.

Also, BitTorrent isn’t the only form of chunked stream transfer available these 
days, it’s just the straw man most people use to talk about multiple-source 
chunked transfer of a common file.

> On the other hand, BitTorrent could just make an Apple 
> TV/PS4/PS5/Xbox/whatever-device-you-use app as well. But I doubt that will 
> work, unless someone can think up a clever way to modify BitTorrent to suit 
> today's network architectures.

BitTorrent and a number of other peer-to-peer chunked transfers are already 
well suited to today’s network architectures and work just fine. The problem is 
that there’s a bit of history of eyeball ISPs becoming hostile and using things 
like DPI to find ways to interfere with them when they become “too successful”.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 18:33, Sabri Berisha wrote:


Yes, let's go back to 2003. The ISP I worked for at that time was one of
the first in the country (if not the first) to host Akamai's caching servers.

Ten years later I worked on a project where Akamai caching was embedded in
subscriber management routers. It was announced, but never productized. This
concept would have brought caching as close to the subscriber as possible.

Today, with the widespread use of HTTPS, something like this is just not
feasible.


Yes, the utility of Squid and similar local caching servers became less 
helpful as objects got more dynamic. This exacerbated as traffic shifted 
over to tcp/443.


Then the CDN's started shipping content closer and closer to eyeballs, 
and that has generally become the norm over the past decade.


I'm not sure anyone still using Squid & Friends is seeing net gains with 
that model, particularly with a major CDN likely being close by.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Sabri Berisha]

- On Oct 11, 2021, at 12:58 AM, Mark Tinka mark@tinka.africa wrote:

Hi,

> However, in an era where content is making a push to get as close to the
> eyeballs as possible, kit getting cheaper and faster because of merchant
> silicon, and abundance of aggregated capacity at exchange points, can we
> leverage the shorter, faster links to change the model?

Yes, let's go back to 2003. The ISP I worked for at that time was one of 
the first in the country (if not the first) to host Akamai's caching servers.

Ten years later I worked on a project where Akamai caching was embedded in
subscriber management routers. It was announced, but never productized. This
concept would have brought caching as close to the subscriber as possible.

Today, with the widespread use of HTTPS, something like this is just not
feasible.

Thanks,

Sabri

 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Xavier Beaudouin via NANOG]

Hello,

>  Providing access is mostly fixed costs, as there are very few consumables in
>  running a network.
> 
>  IP transit costs aren't an issue, since Netflix will do settlement free 
> peering.
> 
>  This leaves the internal network of SK Telecom as the problem and cost 
> center.

Even with that. Netflix can also host at their point of presence their proxies.
Remember the proxies can push up to 400Gbps of stuff with about 500W of power.

So even if they have issues in their network this is their job to make the 
things
done correctly.

Paying just the electricity bill for a bunch of server is less expensive than
buying several 100G long haul links. That's a point and this is their problems 
if
those guys didn't wanted a cache in their network.

Regards,
Xavier

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jared Brown]

Doug Barton wrote:
> One incentive I haven't seen anyone mention is that ISPs don't want to 
> charge customers what it really costs to provide them access. 
  For the sake of argument, let's assume this is true.

  For this to work, I am really trying hard to ignore inconvenient facts like:

  "South Korea’s SK Telecom (SKT) has reported operating revenues of 
   KRW4.818 trillion (USD4.2 billion) for the quarter ended 30 June 2021,
   up 4.7% year-on-year, with it saying that the increase was ‘due to 
   continued solid growth trends in all business areas’.

   SKT’s operating income in Q2 2021 totalled KRW397 billion, up 10.8% on
   an annualised basis..."

   
https://www.commsupdate.com/articles/2021/08/12/sk-telecom-reports-revenue-increase-in-2q21-as-5g-subscriber-numbers-rise/
  
  Nevertheless, let's go with the hypothesis that service is provided below 
cost.

  Providing access is mostly fixed costs, as there are very few consumables in 
running a network.

  IP transit costs aren't an issue, since Netflix will do settlement free 
peering.

  This leaves the internal network of SK Telecom as the problem and cost center.

  There would have been no marginal cost if SK Telecom's own network was 
capable of handling the traffic 
  of its customers.

  So basically SK Telecom is mad at Netflix for forcing equipment upgrades 
faster than budgeted.

  Should Netflix have to pay for SK Telecom sucking at traffic planning and 
budgeting?


- Jared

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 17:39, Jared Brown wrote:


   Since we aren't talking about random pirated content, but p2p streaming from a 
major content provider it would obviously be point & click.


Yes, in which case Jane + Thatho don't need to worry about device 
compatibility, especially if the device is a major brand like Apple TV, 
an established gaming console or an established smart TV.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jared Brown]

Mark Tinka wrote:
>> Well, yes. Or you could just stream content that is guaranteed to be 
>> compatible with the device used.
>
> People on this list would bother to check compatibility.
>
> Jane + Thatho just point & click.
  Since we aren't talking about random pirated content, but p2p streaming from 
a major content provider it would obviously be point & click.


- Jared

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 17:13, Jared Brown wrote:

That's not how it works. Several streaming BitTorrent clients 
specifically request blocks in order so that you can start watching 
immediately.

   Not that you need a special client, it works pretty well with the standard 
client as well on a well seeded torrent, as blocks are generally requested more 
or less in order.


Thanks. I wasn't aware there were torrent clients that streamed straight 
to video. The last time I used torrents, it was to download files for 
later consumption.


I found Usenet feeds more reliable.



Well, yes. Or you could just stream content that is guaranteed to be compatible 
with the device used.


People on this list would bother to check compatibility.

Jane + Thatho just point & click.



They could, and they might even have, I forget, but there is little demand for 
such a thing as a centralized CDN strategy works better.


Which is why I'm not sure it would work, in today's Internet.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jared Brown]

Mark Tinka wrote:
> Someone can correct me if I'm wrong, but the way I know BitTorrent to
> work is the file is downloaded to disk, unarchived and then listed as
> ready to watch.
  That's not how it works. Several streaming BitTorrent clients specifically 
request blocks in order so that you can start watching immediately.
  Not that you need a special client, it works pretty well with the standard 
client as well on a well seeded torrent, as blocks are generally requested more 
or less in order.

> It also assumes the device has all the necessary apps
> and codecs needed to render the file.
  Well, yes. Or you could just stream content that is guaranteed to be 
compatible with the device used.

> On the other hand, BitTorrent could just make an Apple
> TV/PS4/PS5/Xbox/whatever-device-you-use app as well.
  They could, and they might even have, I forget, but there is little demand 
for such a thing as a centralized CDN strategy works better.

> But I doubt that
> will work, unless someone can think up a clever way to modify BitTorrent
> to suit today's network architectures.
  Unless network topology is somehow exposed, this isn't possible. All anybody 
can do is use latency, IP and ASN information as a proxy.

  Nothing is stopping a BitTorrent client from being selective about its peers. 
The current peer selection algorithm optimizes for throughput, not adjecency or 
topology.


- Jared

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 22:57, Matthew Walster wrote:

Ignoring for the moment that P2P is inherently difficult to stream 
with (you're usually downloading chunks in parallel, and with devices 
like Smart TVs etc you don't really have the storage to do so anyway) 
there's also the problem that things like BitTorrent don't know 
network topology and therefore only really increases the 
cross-sectional bandwidth required.


Not to mention that it has been tried before, and didn't work then either.


Yeah, and people also want to click a title and start watching immediately.

Someone can correct me if I'm wrong, but the way I know BitTorrent to 
work is the file is downloaded to disk, unarchived and then listed as 
ready to watch. It also assumes the device has all the necessary apps 
and codecs needed to render the file.


On the other hand, BitTorrent could just make an Apple 
TV/PS4/PS5/Xbox/whatever-device-you-use app as well. But I doubt that 
will work, unless someone can think up a clever way to modify BitTorrent 
to suit today's network architectures.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 14:20, Jason Iannone wrote:

Isn't this a problem with legacy peering agreements in today's 
internet? The same thing happened between Netflix, Level3, and Verizon 
a few years ago. The legacy concept of settlement-free peering is 
based on traffic forwarding parity. If what I forward to you roughly 
matches what you forward to me, we can agree that we have no reason to 
charge each other for access. The concept works fine when content and 
eyeballs are evenly distributed between providers. This doesn't work 
in today's divergent content and eyeball networks.


If Netflix agreed to settlement-free peering under the legacy 
definition, then as far as the letter of the law goes, Netflix is in 
the wrong.


Indeed.

Traffic ratios to determine peering partners(hips) is, I think, archaic, 
and a cop out for not having to deal with peering requests. There are 
many networks with whom peering would add value, despite not matching 
traffic ratios. As you say, BigContent are such networks.


Specifically for us, geographic network scope is the most important, as 
I don't want to help you build your backbone on the back of mine, for 
free. There may be some volume requirements, but certainly not made on 
the basis of directional ratios.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 22:05, Matthew Petach wrote:



Let's check back in 2026, and see if someone's become fantastically
successful doing this or not.  ;)


I have to say, your idea is quite fantastical. I'm not sure I have 
enough brain cells to consider how it will work, remembering that vCPE's 
were all the rage in 2011, and ten years later, they seem to have 
fizzled out without a real-world deployment of note :-).


At any rate, with the current state-of-the-art, deploying Metro edge 
caches is within the realms of possibility. However, it's such a rich 
solution, that it will only likely ever work in a select group of cities 
around the world.


For the rest of us, a nearby data centre pumping cached content across 
fibre links all the way into homes is as good as we shall get. However, 
what this network looks like in 2021 vs. 2006, perhaps, allows us to 
reconsider the model.


One example that comes to mind is VoD-only ISP's, whose raison d'être is 
to deliver VoD content from local caches, with no infrastructure to 
support access to the global Internet. I'd be keen to build something 
like that, particularly in a world where the traditional infrastructure 
operator is a dying species.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jason Iannone]

Isn't this a problem with legacy peering agreements in today's internet?
The same thing happened between Netflix, Level3, and Verizon a few years
ago. The legacy concept of settlement-free peering is based on traffic
forwarding parity. If what I forward to you roughly matches what you
forward to me, we can agree that we have no reason to charge each other for
access. The concept works fine when content and eyeballs are evenly
distributed between providers. This doesn't work in today's divergent
content and eyeball networks.

If Netflix agreed to settlement-free peering under the legacy definition,
then as far as the letter of the law goes, Netflix is in the wrong.

SK's in kind of a pickle when it comes to peering regulation, but it looks
like they need 1:1.8 for settlment-free peering.
https://35v.peeringasia.com/files/Internet.Regulation.in.Korea.pdf

ATT Peering Policy
https://ecfsapi.fcc.gov/file/6518398337.pdf
Peer must maintain a balanced traffic ratio between its network and AS7018.
In particular:
- No more than 2.0:1.0 ratio of traffic flowing in either direction, on
average
- Balanced time of day traffic distribution currently as measured by peak
to average traffic levels

Verizon Peering Policy
https://www.verizon.com/business/terms/peering/
Traffic Exchange Ratio. The ratio of the aggregate amount of traffic
exchanged between the Requester and the Verizon Business Internet Network
with which it seeks to interconnect shall be roughly balanced and shall not
exceed 1.8:1.

Lumen Peering Policy
https://www.lumen.com/en-us/about/legal/peering-policy.html
The backbone cost burden associated with settlement-free peering traffic
exchange should be equitably shared. Regardless of the direction or type of
traffic exchanged between the networks, the routing practices and location
of interconnection points should be such that each party bears a reasonably
equal share of backbone costs.




On Mon, Oct 11, 2021 at 9:27 PM Owen DeLong via NANOG 
wrote:

>
>
> On Oct 11, 2021, at 13:57 , Matthew Walster  wrote:
>
>
>
> On Mon, 11 Oct 2021 at 21:05, Matthew Petach 
> wrote:
>
>> I think it would be absolutely *stunning* for content providers
>> to turn the model on its head; use a bittorrent like model for
>> caching and serving content out of subscribers homes at
>> recalcitrant ISPs, so that data doesn't come from outside,
>> it comes out of the mesh within the eyeball network, with
>> no clear place for the ISP to stick a $$$ bill to.
>>
>
> Ignoring for the moment that P2P is inherently difficult to stream with
> (you're usually downloading chunks in parallel, and with devices like Smart
> TVs etc you don't really have the storage to do so anyway) there's also the
> problem that things like BitTorrent don't know network topology and
> therefore only really increases the cross-sectional bandwidth required.
>
>
> A 4K 2 hour movie is about 40GB. Most modern smart TVs around 32GB of RAM
> and can probably devote about 20GB of that to buffering a stream, so yeah,
> that should actually be doable.
>
> While torrent-like distribution isn’t particularly good for the eyeball
> provider, it can be good for getting content to eyeballs under some
> circumstances regardless of how bad it is for said network.
>
> Unfortunately, it’s not good at knowing how bad it’s being for the network
> and it’s also not good at detecting the circumstances when it’s good for
> the end user or not.
>
> Not to mention that it has been tried before, and didn't work then either.
>
>
> Yep.
>
> Owen
>
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 13:57 , Matthew Walster  wrote:
> 
> 
> 
> On Mon, 11 Oct 2021 at 21:05, Matthew Petach  > wrote:
> I think it would be absolutely *stunning* for content providers 
> to turn the model on its head; use a bittorrent like model for 
> caching and serving content out of subscribers homes at 
> recalcitrant ISPs, so that data doesn't come from outside, 
> it comes out of the mesh within the eyeball network, with 
> no clear place for the ISP to stick a $$$ bill to.
> 
> Ignoring for the moment that P2P is inherently difficult to stream with 
> (you're usually downloading chunks in parallel, and with devices like Smart 
> TVs etc you don't really have the storage to do so anyway) there's also the 
> problem that things like BitTorrent don't know network topology and therefore 
> only really increases the cross-sectional bandwidth required.

A 4K 2 hour movie is about 40GB. Most modern smart TVs around 32GB of RAM and 
can probably devote about 20GB of that to buffering a stream, so yeah, that 
should actually be doable.

While torrent-like distribution isn’t particularly good for the eyeball 
provider, it can be good for getting content to eyeballs under some 
circumstances regardless of how bad it is for said network.

Unfortunately, it’s not good at knowing how bad it’s being for the network and 
it’s also not good at detecting the circumstances when it’s good for the end 
user or not.

> Not to mention that it has been tried before, and didn't work then either.

Yep.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 13:05 , Matthew Petach  wrote:
> 
> 
> 
> On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka  wrote:
> However, in an era where content is making a push to get as close to the 
> eyeballs as possible, kit getting cheaper and faster because of merchant 
> silicon, and abundance of aggregated capacity at exchange points, can we 
> leverage the shorter, faster links to change the model?
> 
> Mark.
> 
> I think it would be absolutely *stunning* for content providers 
> to turn the model on its head; use a bittorrent like model for 
> caching and serving content out of subscribers homes at 
> recalcitrant ISPs, so that data doesn't come from outside, 
> it comes out of the mesh within the eyeball network, with 
> no clear place for the ISP to stick a $$$ bill to.

I worked for a company that tried this model several years back.
It did not go well. Users were not happy. Eyeball ISPs were very
not happy (and started actively mitigating users who participated
through a variety of nefarious means), and content providers got
a bit hot under the collar about it (the experiment was a content
aggregator for lack of a better term).

> Imagine you've got a movie; you slice it into 1,000 
> encrypted chunks; you make part of your license 
> agreement for customers a requirement that they 
> will allow you to use up to 20GB of disk space on 
> their computer and to serve up data chunks into 
> the network in return for a slightly cheaper monthly 
> subscription cost to your service.  You put 1 slice 
> of that movie on each of 1,000 customers in a 
> network; then you replicate that across the next 
> thousand customers, and again for the next 
> thousand, until you've got enough replicas of 
> each shard to handle a particular household 
> going offline.  Your library is still safe from 
> piracy, no household has more than 1/1000th of a 
> movie locally, and they don't have the key to decrypt 
> it anyhow; but they've got 1/1000th of 4, different 
> movies, and when someone in that ISP wants to watch the 
> movie, the chunks are being fetched from other households 
> within the eyeball network.  The content provider would have 
> shard servers in region, able to serve up any missing shards that 
> can't be fetched locally within the ISP--but the idea would be that 
> as the number of subscribers within an ISP goes up, instead of the 
> ISP seeing a large, single-point-source increase in traffic, what they 
> see is an overall increase in east-west traffic among their users.

That’s a pretty close description of exactly what we did.

> Because the "serving of shards to others" happens primarily while the 
> user is actively streaming content, you have a natural bell curve; during 
> peak streaming times, you have more nodes active to serve up shards, 
> handling the increased demand; at lower demand times, when fewer 
> people are active, and there's fewer home-nodes to serve shards, the 
> content network's shard servers can pick up the slack...but that'll generally 
> only happen during lower traffic times, when the traffic won't be competing 
> and potentially causing pain for the ISP.

It’s a beautiful theory. We expected this to be the case, too. Turns out, not
so much. There’s an awful lot of long tail out there that is poorly accounted
for in this. Yes, it does help with some of the most popular content, but there
are pitfalls there, too.

> Really, it seems like a win-win scenario.

We thought so too, until we learned otherwise. Today, with more symmetrical
connections and larger uplinks, it might be more feasible. Back then (Around
2005), it was a good idea on paper.

> I'm confident we'll see a content network come out with a model like this 
> within the next 5 years, at which point the notion of blackmailing content 
> networks for additional $$$s will be a moot point, because the content will 
> be distributed and embedded within every major eyeball network already,
> whether they like it or not, on their customer's devices.

You’d be surprised at the number of different ways eyeball providers have
to denature such an effort. I’m sorry to rain on your parade, but it’s not a new
idea. I was brought in as the operations guy once the software engineers
realized that they needed someone who could, you know, keep stuff running
and not just write code and test it in production. This model resulted in a
frank and uncomfortable conversation between me and the developer of
what was internally being called the “overlay network” code. We went through
a number of different scenarios where he had made incorrect assumptions
about how the internet worked, especially at the eyeball end and there
were a dozen or so that we simply couldn’t find ways to work around.

Networks are a bit different now and some content providers have people
far more clever than I working on things like this, so who knows… It might
succeed by 2026, but when we tried it in 2005, we couldn’t make it work
well enough to avoid 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Mon, 11 Oct 2021 at 21:05, Matthew Petach  wrote:

> I think it would be absolutely *stunning* for content providers
> to turn the model on its head; use a bittorrent like model for
> caching and serving content out of subscribers homes at
> recalcitrant ISPs, so that data doesn't come from outside,
> it comes out of the mesh within the eyeball network, with
> no clear place for the ISP to stick a $$$ bill to.
>

Ignoring for the moment that P2P is inherently difficult to stream with
(you're usually downloading chunks in parallel, and with devices like Smart
TVs etc you don't really have the storage to do so anyway) there's also the
problem that things like BitTorrent don't know network topology and
therefore only really increases the cross-sectional bandwidth required.

Not to mention that it has been tried before, and didn't work then either.

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Mon, Oct 11, 2021 at 10:09 AM Michael Thomas  wrote:

>
> On 10/11/21 12:49 AM, Matthew Petach wrote:
>
>
> Instead of a 4K stream, drop it to 480 or 240; the eyeball network
> should be happy at the reduced strain the resulting stream puts
> on their network.
>
> As a consumer paying for my 4k stream, I know who I'm calling when it
> drops to 480 and it ain't Netflix. The eyeballs are most definitely not
> happy.
>
> Mike
>

I apologize for that.  I was tired after two back-to-back days
of board meetings, and I missed putting a clear sarcasm
marker on that last line about "the eyeball networks
should be happy at the reduced strain...":(

There should have been a clear ;-P at the end of
the line to make it unmistakeable I was poking a
very sharp stick at the eyeball networks and
what it takes to actually make them happy.  ^_^;

Yes--the end consumers really shouldn't be the hostage
in this battle, being moved about the chess board by
either side, whether by their ISP trying to squeeze
more money out of the content side, or by the content
side trying to force more complaints into the service
desk of the ISP.

I mean, imagine this scenario for any other utility.

Pacific Gas and Electric calling up Hoover Dam to
say "hey, we're going to need to charge you some
additional money this month."

Hoover Dam: "...what?"

PGE: "well, you're sending a lot more electricity to
our customers this month, and we're going to have
to upgrade our power lines to handle it; and since
you're the one sending the electricity, you should
pay for part of the costs."

Hoover Dam: "...we're only sending enough electricity
to meet the demands YOUR customers are placing on
the grid.  If they want to run their air conditioners all
summer long, you need to charge them enough to
cover your costs for it."

Drat.  My analogy just ran out, because I realize the
dollars already flow the other way, and the hydroelectric
station would just laugh at PG and threaten to raise
the cost of the electricity simply for having to listen to their BS.   ^_^;

You can run the same scenario with your municipal water
company, and imagine how it would play out if the municipality
that put the pipes in to every home tried to charge the water
supplier more because homes were taking longer showers.

It's just such a fundamentally broken model, we laugh at it
in any other industry.  :(

Again, I'm sorry for being tired and missing the explicit
sarcasm indicator--not just for you, but for others who also
responded to that paragraph.   ^_^;

Thanks!

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka  wrote:

> However, in an era where content is making a push to get as close to the
> eyeballs as possible, kit getting cheaper and faster because of merchant
> silicon, and abundance of aggregated capacity at exchange points, can we
> leverage the shorter, faster links to change the model?
>
> Mark.
>

I think it would be absolutely *stunning* for content providers
to turn the model on its head; use a bittorrent like model for
caching and serving content out of subscribers homes at
recalcitrant ISPs, so that data doesn't come from outside,
it comes out of the mesh within the eyeball network, with
no clear place for the ISP to stick a $$$ bill to.

Imagine you've got a movie; you slice it into 1,000
encrypted chunks; you make part of your license
agreement for customers a requirement that they
will allow you to use up to 20GB of disk space on
their computer and to serve up data chunks into
the network in return for a slightly cheaper monthly
subscription cost to your service.  You put 1 slice
of that movie on each of 1,000 customers in a
network; then you replicate that across the next
thousand customers, and again for the next
thousand, until you've got enough replicas of
each shard to handle a particular household
going offline.  Your library is still safe from
piracy, no household has more than 1/1000th of a
movie locally, and they don't have the key to decrypt
it anyhow; but they've got 1/1000th of 4, different
movies, and when someone in that ISP wants to watch the
movie, the chunks are being fetched from other households
within the eyeball network.  The content provider would have
shard servers in region, able to serve up any missing shards that
can't be fetched locally within the ISP--but the idea would be that
as the number of subscribers within an ISP goes up, instead of the
ISP seeing a large, single-point-source increase in traffic, what they
see is an overall increase in east-west traffic among their users.

Because the "serving of shards to others" happens primarily while the
user is actively streaming content, you have a natural bell curve; during
peak streaming times, you have more nodes active to serve up shards,
handling the increased demand; at lower demand times, when fewer
people are active, and there's fewer home-nodes to serve shards, the
content network's shard servers can pick up the slack...but that'll
generally
only happen during lower traffic times, when the traffic won't be competing
and potentially causing pain for the ISP.

Really, it seems like a win-win scenario.

I'm confident we'll see a content network come out with a model like this
within the next 5 years, at which point the notion of blackmailing content
networks for additional $$$s will be a moot point, because the content will
be distributed and embedded within every major eyeball network already,
whether they like it or not, on their customer's devices.

Let's check back in 2026, and see if someone's become fantastically
successful doing this or not.  ;)

Thanks!

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Michael Thomas]

On 10/11/21 12:49 AM, Matthew Petach wrote:


Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.


As a consumer paying for my 4k stream, I know who I'm calling when it 
drops to 480 and it ain't Netflix. The eyeballs are most definitely not 
happy.


Mike

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> Going back to the fact that it's not the content providers "using" 
> a lot of bandwidth, it's the eyeball customer *requesting* a lot 
> of bandwidth, I think the best approach is for the content providers 
> to help manage traffic levels by lowering bit rates towards eyeball 
> networks that are feeling strained by their users.

This is the model that has pissed me off for decades… Somehow the
cellular carrier networks have been able to force phone application
and content providers to do things like limit the maximum size of file
that can be downloaded over the cellular network and force you to
download certain content over wifi.

Since I maintain a backup handset anyway and it is rarely utilized,
I let it accumulate rollover data until I want to do a large download.
Then I turn on its hotspot and the other phone uses that wifi to
download the large file I wasn’t allowed to download over the cellular
network due to exactly this stupid kind of limitation.

> Instead of a 4K stream, drop it to 480 or 240; the eyeball network 
> should be happy at the reduced strain the resulting stream puts 
> on their network.  

So your solution is to make the content provider punish the eyeball
user and deliver a poor user experience in order to let the crappy
eyeball network off the hook? I don’t think that’s a good solution.
It makes the content provider look bad to the end user and it
shifts the burden from the ISP that got paid to deliver content they
are failing to deliver onto the content provider that is trying to live
up to their agreement with their user.

IIRC, Netflix charges extra for a 4K level subscription these days,
so an end user that paid for 4K service and got 240p because their
ISP managed to force Netflix into a lower bitrate would likely be
pretty annoyed at both Netflix and the ISP if they understood the
situation.

> The content network can even point out they're being a good 
> Network Citizen by putting up a brief banner at the top of the 
> stream saying "reducing bit rate to relieve stress on your ISPs
> network".  That way, the happy customer knows that the 
> content provider is doing their part to help their ISP stay 
> profitable...I mean, doing their part to help the Internet 
> run better. 

Yeah, I’d be calling $CONTENT_PROVIDER and asking what I need
to do to get the full rate service I paid for. I’d also be calling $ISP
(or switching $ISP if possible) to one that didn’t have those issues
.
>  I'm pretty sure this is going to start happening more and more, 
> as ISPs realize that putting content caches into their IP space 
> to serve not only their own customers, but also customers of 
> selected peers can be a source of good leverage in the market. 

Agreed… Caching close to the edge makes complete sense,
especially with aggregated caching models through CDNs.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 00:32 , Mark Tinka  wrote:
> 
> 
> 
> On 10/11/21 02:58, Owen DeLong wrote:
> 
>> That’s irrelevant to what he is saying.
>> 
>> What he’s saying (and he’s 100% correct) is that any tax a corporation pays 
>> is collected from their customers one way or another.
>> 
>> A corporation has no other source of income with which to pay its taxes 
>> beyond those revenues collected from customers.
>> 
>> Of course I should probably expect this from someone who thinks IPv4 
>> shortages can be avoided by rationing IPv4 addresses.
> 
> There you go again, getting overly excited trying to divert the topic to your 
> keen area of interest - IPv4 in Africa.

My keen area of interest is IPv6 deployment globally, actually.

> I'll spell it out here so you are clear and have zero doubt:
> 
> I do not respect you - for what you represent on my continent, amongst other 
> things. So ignore me, because I am ignoring you. But if you don't ignore me, 
> that's your problem too, not mine.

Oh, you’ve made that quite clear and it’s mutual.

This word ignore, I do not think it means what you seem to think it means.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> I am almost sure Netflix have some degree of presence in South Korea. What 
> I'm not sure about is what else SK wants them to do beyond that.

They’ve made it pretty clear… They want Netflix to pay their 
protection^wbandwidth charges.

>> And for the record, not only have I never worked for an ISP, I was saying 
>> all the way back in the late '90s that the oversubscription business model 
>> (which almost always includes punishing users who actually use their 
>> bandwidth) is inherently unfair to the customers, and when the Internet 
>> becomes more pervasive in daily life will come back to bite them in the ass. 
>> I was laughed at for being hopelessly naive, not understanding how the 
>> bandwidth business works, etc.
> 
> Totally agreed. However, we are sort of forced to use this model because of 
> the underlying technology. Whenever a finite resource has to be shared 
> amongst several people, there has to be some way to manage that sharing.
> 
> Maybe if Internet services were circuit-switched, we wouldn't have this 
> problem. But then again, we wouldn't have an Internet like we do today either.

The oversubscription model is perfectly valid if rational numbers are chosen 
for oversubscription ratios and providers expect a reasonable number of 
customers to actually use what they paid for.

Many businesses outside of the internet depend on oversubscription to keep 
prices affordable while still making a profit.

Imagine if everyone actually used their gym memberships, for example.

Consider the standard practice of airline overbooking.

Hotels also often overbook.

Imagine if everyone who bought a season pass to an amusement park showed up 
every day they were open.

Now 50:1 oversubscription is probably insane and I agree that when you have a 
problem because your oversubscribed customers have difficulty when a few of them
use what you sold them, it’s not the customers’ fault and you need to reduce 
your oversubscription ratio to accommodate. That’s the business you’re in and 
if you
didn’t factor that into your pricing, you have a poor pricing structure.

For all I love to criticize Comcast for the many many things they do wrong, 
they have a reasonable model where you can buy your way out of bandwidth caps
for $30/month (at least in my case) and they don’t punish me when I use my full 
connection (or close to it).

YMMV.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 00:01 , Mark Tinka  wrote:
> 
> 
> 
> On 10/11/21 00:31, Geoff Huston wrote:
> 
>> In many environments, the words we use to describe this form of price 
>> setting are generally prefixed by the adjective “illegal” :-)
> 
> Indeed - colluding is generally frowned upon, in which case we are doomed to 
> the current model, and may the best man win.
> 
> Ultimately, many ISP's and telco's will die. Consolidation will occur, but 
> the "big operator" will no longer be as fat as they used to be. Focus on 
> hauling bits around with no frills will be a good model, especially if you 
> can keep the team lean. The chances of having large monopolies that do okay 
> but stifle the market, being chased by struggling ISP's that favour passion + 
> frills, is what is likely to happen, over the next decade or two.
> 
> Mark.

Ideally, a regulatory framework which prohibits vertical integration and thus 
prevents the natural monopoly of last mile physical
infrastructure from being leveraged into a monopoly on higher-layer services 
would significantly improve the current situation,
making room for a strong competing market with low barrier to entry for 
services while the last mile infrastructure was managed
by a regulated utility or the run by the local municipality.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 17:26, Niels Bakker wrote:




I don't think that's being entirely fair. Netflix in plenty places 
differentiates its subscriptions based partly on video resolution: 
https://help.netflix.com/en/node/24926/us


Some people will definitely care enough to sign up for a more 
expensive tier.


In much the same way those that care will opt for the D+ mode of the 
car, and likely more will be happy with just the D model.


Yes, many subscribers are clued up on the different Netflix plans, but 
I'm sure most of them choose the plans not for the video resolution, but 
for the number of active screens that can stream concurrently.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Niels Bakker]

* mark@tinka.africa (Mark Tinka) [Mon 11 Oct 2021, 17:18 CEST]:

To be fair, Jane + Thatho don't care about video resolution.


I don't think that's being entirely fair. Netflix in plenty places 
differentiates its subscriptions based partly on video resolution: 
https://help.netflix.com/en/node/24926/us


Some people will definitely care enough to sign up for a more 
expensive tier.



-- Niels.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Seun Ojedeji]

On Sun, Oct 10, 2021 at 8:11 PM Doug Barton  wrote:

> On 10/1/21 7:45 AM, Mark Tinka wrote:
>
>
> The reason that Netflix doesn't want to do it is the same reason that
> ISPs don't want to charge their customers what it really costs to
> provide them access.
>

SO: In my part of the world where end user internet services are largely
based on data caps, this should be good news to the ISP as the faster users
burn the data on streaming, the sooner they renew their subscription. What
is an ISP without content by the way? would one blame the likes of facebook
for trying to run their own pipe thereby further contributing to internet
defragmentation.

Regard
-- 



*Seun Ojedeji,Mobile: +2348035233535*

Bringing another down does not take you up - think about your action!

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 09:49, Matthew Petach wrote:


Going back to the fact that it's not the content providers "using"
a lot of bandwidth, it's the eyeball customer *requesting* a lot
of bandwidth, I think the best approach is for the content providers
to help manage traffic levels by lowering bit rates towards eyeball
networks that are feeling strained by their users.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.

The content network can even point out they're being a good
Network Citizen by putting up a brief banner at the top of the
stream saying "reducing bit rate to relieve stress on your ISPs
network".  That way, the happy customer knows that the
content provider is doing their part to help their ISP stay
profitable...I mean, doing their part to help the Internet
run better.


To be fair, Jane + Thatho don't care about video resolution. All they 
will see is that the picture isn't that great, and this creates an 
opportunity for a VoD provider who is "more favorable" toward the 
network operator, and gets granted full 4K resolution transport 
priviledges. If there are any surcharges levied by the network operator, 
the VoD provider compensates for those by attracting more business 
because, well, they can offer a more superior image quality.


I'm not sure about the term, but the "colluding" version for that would 
be - if my few IGF days do not fail me - "net neutrality".


I'm not sure we want to go down that path, either.



The market *is* determining that at the moment...but not in the
direction people expect.  Instead, it's creating a new market for
intermediaries; imagine you're an eyeball network that happens
to have peering with SKB, and largely inbound traffic flows.
Wouldn't it make sense for you to reach out to a player like
Netflix, and offer to host content cache boxes that happen to
only answer requests coming from SKB IP space, at a price
well below what SKB was going to charge the content provider?
As the eyeball network, you'd see your traffic ratios
balance out as the cache traffic filled your under-utilized outbound
port capacity, and you'd get a bit of additional revenue you otherwise
wouldn't get.  As the content provider, you're serving your customers
for a lower price than SKB wants to charge, and without giving into
SKB's extortion tactics.  It's a win-win-lose situation, in which the
content provider wins, the eyeball network that has a peering
relationship with SKB wins, and the only loser is SKB, which
doesn't get the additional revenue it was looking for, and actually
helps funnel money to a competitor that they otherwise wouldn't
have gotten.

I'm pretty sure this is going to start happening more and more,
as ISPs realize that putting content caches into their IP space
to serve not only their own customers, but also customers of
selected peers can be a source of good leverage in the market.


In reality, which small mom & pop will have peering with BigTelco :-)?

Suffice it to say, Netflix would also need to reconsider whether they 
can afford to give OCA's to mom & pop.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:10, Niels Bakker wrote:



Sounds like you think SK should be paying Netflix for bringing their 
content all the way from the US to the Korean peninsula. That's some 
expensive wet cable being used there.


Do we know whether Netflix don't already have OCA's and/or local peering 
in South Korea?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:57, Sabri Berisha wrote:


I have worked for ISPs. And I remember the late 90s. Bandwidth was $35/mbit
on average, at least for the outfit where I was. Consumers paid roughly $40
for their DSL connections, which at the time went up to 2Mbit depending
on the age of the copper and distance to the DSLAM. Consumer connections
were oversubscribed, on average, 1:35 to 1:50. B2B connections got a better
deal, 1:10 to 1:15.

It was simply not feasible to offer 1:1 bandwidth and still make a profit,
unless you're charging fees the average consumer cannot afford.

Especially considering that the average user doesn't even need or use that
much bandwidth. It's a recurring discussion. People demand more bandwidth
without considering whether or not they need it. End-users, business subs,
and host-owners at large enterprises where I worked. The last ones are the
funniest: entire racks using no more than 100mbit/s and hostowners are
demanding an upgrade from 10G to 25G bEcaUse LaTenCy.

The last consumer ISP I worked at had a very small subset of users that
really needed bandwidth: the "download dudes" who were 24/7 leeching news
servers, and the inevitable gamers that complained about the latency due
to the links being full as a result of said leechers. In that case, a
carefully implemented shaping of tcp/119 did the trick.


It's the conundrum of a shared resource. Like managing 1 lift (elevator, 
for the Americans :-)) that needs to move a building full of people 
between floors.


In the early days of the Internet, circuits were long, expensive and 
slow. Equipment cost a lot for what you could get out of it, and content 
was mostly centralized, making getting to it even more expensive, and 
hence, entrenching the current model.


However, in an era where content is making a push to get as close to the 
eyeballs as possible, kit getting cheaper and faster because of merchant 
silicon, and abundance of aggregated capacity at exchange points, can we 
leverage the shorter, faster links to change the model?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Sun, Oct 10, 2021 at 2:44 PM Doug Barton  wrote:

> [some snipping below]
>
> Also just to be clear, these are my own opinions, not necessarily shared
> by any current or former employers.
>
> On 10/10/21 12:31 PM, Mark Tinka wrote:
> > On 10/10/21 21:08, Doug Barton wrote
> >> Given that issue, I have some sympathy for eyeball networks wanting to
> >> charge content providers for the increased capacity that is needed to
> >> bring in their content. The cost would be passed on to the content
> >> provider's customers...
> >
> > But eyeballs are already paying you a monthly fee for 100Mbps of service
> > (for example). So they should pay a surcharge, over-and-above that, that
> > determines how they can use that 100Mbps? Seems overly odd, to me.
>
> Yes, I get that. But as you pointed out here and in other comments, the
> ISP market is based entirely on undercutting competitors (with a lot of
> gambling thrown in, as Matthew pointed out).
>
>  [...]

> > So what rat hole does this lead us down into? People who want to stream
> > Youtube should pay their ISP for that? People who want to spend
> > unmentionable hours on Linkedin should be their ISP for that? People who
> > want to gawk over Samsung's web site because they love it so much,
> > should pay their ISP for that?
>
> First, I'm not saying "should." I'm saying that given the market
> economics, having the content providers who use "a lot" of bandwidth do
> something to offset those costs to the ISPs might be the best/least bad
> option. Whether "something" is a local cache box, peering, money, or
>  is something I think that the market should determine.
>

Going back to the fact that it's not the content providers "using"
a lot of bandwidth, it's the eyeball customer *requesting* a lot
of bandwidth, I think the best approach is for the content providers
to help manage traffic levels by lowering bit rates towards eyeball
networks that are feeling strained by their users.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.

The content network can even point out they're being a good
Network Citizen by putting up a brief banner at the top of the
stream saying "reducing bit rate to relieve stress on your ISPs
network".  That way, the happy customer knows that the
content provider is doing their part to help their ISP stay
profitable...I mean, doing their part to help the Internet
run better.


> And to answer Matthew's question, I don't know what "a lot" is. I think
> the market should determine that as well.
>

The market *is* determining that at the moment...but not in the
direction people expect.  Instead, it's creating a new market for
intermediaries; imagine you're an eyeball network that happens
to have peering with SKB, and largely inbound traffic flows.
Wouldn't it make sense for you to reach out to a player like
Netflix, and offer to host content cache boxes that happen to
only answer requests coming from SKB IP space, at a price
well below what SKB was going to charge the content provider?
As the eyeball network, you'd see your traffic ratios
balance out as the cache traffic filled your under-utilized outbound
port capacity, and you'd get a bit of additional revenue you otherwise
wouldn't get.  As the content provider, you're serving your customers
for a lower price than SKB wants to charge, and without giving into
SKB's extortion tactics.  It's a win-win-lose situation, in which the
content provider wins, the eyeball network that has a peering
relationship with SKB wins, and the only loser is SKB, which
doesn't get the additional revenue it was looking for, and actually
helps funnel money to a competitor that they otherwise wouldn't
have gotten.

I'm pretty sure this is going to start happening more and more,
as ISPs realize that putting content caches into their IP space
to serve not only their own customers, but also customers of
selected peers can be a source of good leverage in the market.

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 02:58, Owen DeLong wrote:


That’s irrelevant to what he is saying.

What he’s saying (and he’s 100% correct) is that any tax a corporation pays is 
collected from their customers one way or another.

A corporation has no other source of income with which to pay its taxes beyond 
those revenues collected from customers.

Of course I should probably expect this from someone who thinks IPv4 shortages 
can be avoided by rationing IPv4 addresses.


There you go again, getting overly excited trying to divert the topic to 
your keen area of interest - IPv4 in Africa.


I'll spell it out here so you are clear and have zero doubt:

I do not respect you - for what you represent on my continent, amongst 
other things. So ignore me, because I am ignoring you. But if you don't 
ignore me, that's your problem too, not mine.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 03:05, Owen DeLong wrote:


Which is the kind of ignorant view of the situation that creates this problem 
in the first place.

It’s not for “no $$”, it’s for all the $$ they got from all those 100Mbps links 
that they are delivering those Tbps of traffic to.

If the aggregate $$ they are collecting is insufficient, then they have priced 
their service incorrectly and should either re-evaluate,
or go bankrupt and sell to someone that knows how to run a business.


Mate, keep your pants on... don't get yourself worked up into excitement 
at my folly.


And in case you haven't noticed, I am ignoring you.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:42, Doug Barton wrote:



I didn't say income tax. Corporate taxes are considered an expense by 
the corporation paying them. Like all other expenses, they are 
factored into the cost of goods/services sold.


Ah, yes, agreed. I thought you meant something else...




First, I'm not saying "should." I'm saying that given the market 
economics, having the content providers who use "a lot" of bandwidth 
do something to offset those costs to the ISPs might be the best/least 
bad option. Whether "something" is a local cache box, peering, money, 
or  is something I think that the market should determine.


But all the major content providers do this already. They come to 
exchange points. They provide caches. What more do we want them to do?


Content providers that don't do these things don't generally tend to be 
popular, in which case, we don't have to worry about them flooding 
backbone links.


I am almost sure Netflix have some degree of presence in South Korea. 
What I'm not sure about is what else SK wants them to do beyond that.





And to answer Matthew's question, I don't know what "a lot" is. I 
think the market should determine that as well.


Hehe, free markets determine the fundamental principles through price 
and competition, which is why we are in this mess to begin with. Unless 
you want "government" to make the determination :-).





And for the record, not only have I never worked for an ISP, I was 
saying all the way back in the late '90s that the oversubscription 
business model (which almost always includes punishing users who 
actually use their bandwidth) is inherently unfair to the customers, 
and when the Internet becomes more pervasive in daily life will come 
back to bite them in the ass. I was laughed at for being hopelessly 
naive, not understanding how the bandwidth business works, etc.


Totally agreed. However, we are sort of forced to use this model because 
of the underlying technology. Whenever a finite resource has to be 
shared amongst several people, there has to be some way to manage that 
sharing.


Maybe if Internet services were circuit-switched, we wouldn't have this 
problem. But then again, we wouldn't have an Internet like we do today 
either.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 01:43, Keith Medcalf wrote:


This is blatantly incorrect.  The bits were payed for by the requestor.


You totally missed my dig...

I was being sarcastic.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:31, Geoff Huston wrote:


In many environments, the words we use to describe this form of price setting 
are generally prefixed by the adjective “illegal” :-)


Indeed - colluding is generally frowned upon, in which case we are 
doomed to the current model, and may the best man win.


Ultimately, many ISP's and telco's will die. Consolidation will occur, 
but the "big operator" will no longer be as fat as they used to be. 
Focus on hauling bits around with no frills will be a good model, 
especially if you can keep the team lean. The chances of having large 
monopolies that do okay but stifle the market, being chased by 
struggling ISP's that favour passion + frills, is what is likely to 
happen, over the next decade or two.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 10, 2021, at 13:21 , Mark Tinka  wrote:
> 
> 
> 
> On 10/10/21 22:13, Michael Thomas wrote:
> 
>> Isn't that what Erlang numbers are all about? My suspicion is that after 
>> about 100Mbs most people wouldn't notice the difference in most cases. My 
>> ISP is about 25Mbs on a good day (DSL) and it serves our needs fine and have 
>> never run into bandwidth constraints. Maybe if we were streaming 4k all of 
>> the time it might be different, but frankly the difference for 4k isn't all 
>> that big. It's sort of like phone screen resolution: at some point it just 
>> doesn't matter and becomes marketing hype.
>> 
> 
> The ISP looking to charge BigContent for increased link saturation isn't 
> looking at the individual 100Mbps links they have sold to their downstream 
> customers.
> 
> They are looking at the aggregate Gbps or Tbps of traffic that BigContent is 
> seeking to deliver across their network, for "no $$".

Which is the kind of ignorant view of the situation that creates this problem 
in the first place.

It’s not for “no $$”, it’s for all the $$ they got from all those 100Mbps links 
that they are delivering those Tbps of traffic to.

If the aggregate $$ they are collecting is insufficient, then they have priced 
their service incorrectly and should either re-evaluate,
or go bankrupt and sell to someone that knows how to run a business.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 10, 2021, at 13:18 , Mark Tinka  wrote:
> 
> 
> 
> On 10/10/21 22:10, Geoff Huston wrote:
> 
>> I have to agree with Doug Barton's earlier observation is that the base 
>> problem is that the ISPs are using a flawed business model and they don't 
>> want to charge their customers what it really costs to provide them with 
>> high speed access, nor do they want to fund additional back-end capacity in 
>> their network without some form of offset revenue stream.
> 
> I think ISP's do want to charge their customers what it actually costs to 
> provide them with a service, but they can't because many ISP's business 
> models are based purely on undercutting their nearest competitor.

Then that’s a flawed business model and one of them will eventually get lucky 
in each market place and race prices once they are a monopoly.

> I might be naive and hopeful to think that operators will have a blood 
> handshake to set prices where customers can't wag the tail.

Such collusion is usually the basis of antitrust laws and ill-advised at best.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

>> (in the same way that corporations don't pay taxes, their customers do),...
> 
> 
> Many a company pays corporate tax, which is separate from the income tax they 
> pay for compensation to their staff.
> 
> Of course, YMMV depending on where you live.

That’s irrelevant to what he is saying.

What he’s saying (and he’s 100% correct) is that any tax a corporation pays is 
collected from their customers one way or another.

A corporation has no other source of income with which to pay its taxes beyond 
those revenues collected from customers.

Of course I should probably expect this from someone who thinks IPv4 shortages 
can be avoided by rationing IPv4 addresses.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 10, 2021, at 12:08 , Doug Barton  wrote:
> 
> On 10/1/21 7:45 AM, Mark Tinka wrote:
>> The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own 
>> global backbones is because of this nonsense that SK Broadband is trying to 
>> pull with Netflix. At some point, the content folk will get fed up, and go 
>> build it themselves. What an opportunity infrastructure cost itself!
> 
> Except that Facebook, Microsoft, and Amazon all caved to SK's demands:
> 
> "The popularity of the hit series "Squid Game" and other offerings have 
> underscored Netflix's status as the country's second-largest data traffic 
> generator after Google's YouTube, but the two are the only ones to not pay 
> network usage fees, which other content providers such as Amazon, Apple and 
> Facebook are paying, SK said."
> 
> Which has emboldened SK to go after the bigger fish.
> 
> One incentive I haven't seen anyone mention is that ISPs don't want to charge 
> customers what it really costs to provide them access. If you're the only one 
> in your market that is doing that, no one is going to sign up because your 
> pricing would be so far out of line with your competition.

Only if your competition is somehow getting funding from another source (e.g. 
extorting content providers).

As such, I’d guess that this situation won’t untangle itself any time soon 
because markets which lack transparency and/or have (mostly) ignorant customers 
tend to be dysfunctional in exactly these kinds of ways.

> Given that issue, I have some sympathy for eyeball networks wanting to charge 
> content providers for the increased capacity that is needed to bring in their 
> content. The cost would be passed on to the content provider's customers (in 
> the same way that corporations don't pay taxes, their customers do), so the 
> people on that ISP who are creating the increased demand would be 
> (indirectly) paying for the increased capacity. That's actually fairer for 
> the other customers who aren't Netflix subscribers.

An interesting argument, but I don’t entirely buy it.

I’m a high consumer of bandwidth. I end up paying an extra surcharge each month 
to get data without a cap. I think that’s perfectly fair.

OTOH, I think it’s not particularly fair if after I pay for that cap removal, 
my ISP turns around and extorts even more money from the companies I’m paying 
for content in order to effectively make my circuit even more expensive.

> The reason that Netflix doesn't want to do it is the same reason that ISPs 
> don't want to charge their customers what it really costs to provide them 
> access.


Sounds like the electrical deregulation plan that Enron wrote for California 
and then criticized while they gamed the system for fun and profit.

Owen

RE: S.Korea broadband firm sues Netflix after traffic surge

[Keith Medcalf]

On Sunday, 10 October, 2021 14:21, Mark Tinka wrote:

>They are looking at the aggregate Gbps or Tbps of traffic that
>BigContent is seeking to deliver across their network, for "no $$".

This is blatantly incorrect.  The bits were payed for by the requestor.

BigContent does not "send bits" to non-requestors.
The Internet is Point-to-Point, not a Broadcast medium.

If the seller (the network operator) cannot provide the service which they have 
sold, they should be imprisoned for the remainder of their natural lives at 
hard labour.  This sort of behaviour by the network operator is a Criminal 
Activity called FRAUD (based on Fraudulent Misrepresentation of Material Fact) 
and is, in fact, a Criminal Conspiracy.

--
You can tell a politician is lying because it's lips are moving.
The only good politician is a dead politician.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Glenn Kelley]

Netflix has programs for which many ISPs - even smaller are able to 
build a cache system.

This may help the ISP who filed suit here -

That being said - Our Consultancy has helped a number of smaller ISPs 
build using the Open Connect options - however for many they cannot 
justify the want from Netflix to have the minimum of 5Gbps of peak 
Netflix traffic let alone the 1.2Gbps of inbound traffic daily during 
the 12 hour update windows.


A move like this may help wake up Netflix to making these options a bit 
nicer for the smaller boys



Glenn

Re: S.Korea broadband firm sues Netflix after traffic surge

[Geoff Huston]

> On 11 Oct 2021, at 7:18 am, Mark Tinka  wrote:
> 
> 
> 
> On 10/10/21 22:10, Geoff Huston wrote:
> 
>> I have to agree with Doug Barton's earlier observation is that the base 
>> problem is that the ISPs are using a flawed business model and they don't 
>> want to charge their customers what it really costs to provide them with 
>> high speed access, nor do they want to fund additional back-end capacity in 
>> their network without some form of offset revenue stream.
> 
> I think ISP's do want to charge their customers what it actually costs to 
> provide them with a service, but they can't because many ISP's business 
> models are based purely on undercutting their nearest competitor.
> 
> I might be naive and hopeful to think that operators will have a blood 
> handshake to set prices where customers can't wag the tail.
> 

In many environments, the words we use to describe this form of price setting 
are generally prefixed by the adjective “illegal” :-)

Geoff

Re: S.Korea broadband firm sues Netflix after traffic surge

[Niels Bakker]

* do...@dougbarton.us (Doug Barton) [Sun 10 Oct 2021, 23:44 CEST]:
First, I'm not saying "should." I'm saying that given the market 
economics, having the content providers who use "a lot" of bandwidth 
do something to offset those costs to the ISPs might be the 
best/least bad option. Whether "something" is a local cache box, 
peering, money, or  is something I think that the market 
should determine.


Sounds like you think SK should be paying Netflix for bringing their 
content all the way from the US to the Korean peninsula. That's 
some expensive wet cable being used there.



-- Niels.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Sabri Berisha]

- On Oct 10, 2021, at 2:42 PM, Doug Barton do...@dougbarton.us wrote:

Hi,

> And for the record, not only have I never worked for an ISP, I was
> saying all the way back in the late '90s that the oversubscription
> business model (which almost always includes punishing users who
> actually use their bandwidth) is inherently unfair to the customers, and
> when the Internet becomes more pervasive in daily life will come back to
> bite them in the ass. I was laughed at for being hopelessly naive, not
> understanding how the bandwidth business works, etc.

I have worked for ISPs. And I remember the late 90s. Bandwidth was $35/mbit
on average, at least for the outfit where I was. Consumers paid roughly $40
for their DSL connections, which at the time went up to 2Mbit depending
on the age of the copper and distance to the DSLAM. Consumer connections
were oversubscribed, on average, 1:35 to 1:50. B2B connections got a better
deal, 1:10 to 1:15.

It was simply not feasible to offer 1:1 bandwidth and still make a profit,
unless you're charging fees the average consumer cannot afford. 

Especially considering that the average user doesn't even need or use that
much bandwidth. It's a recurring discussion. People demand more bandwidth
without considering whether or not they need it. End-users, business subs,
and host-owners at large enterprises where I worked. The last ones are the
funniest: entire racks using no more than 100mbit/s and hostowners are 
demanding an upgrade from 10G to 25G bEcaUse LaTenCy.

The last consumer ISP I worked at had a very small subset of users that 
really needed bandwidth: the "download dudes" who were 24/7 leeching news
servers, and the inevitable gamers that complained about the latency due
to the links being full as a result of said leechers. In that case, a
carefully implemented shaping of tcp/119 did the trick.

Thanks,

Sabri

Re: S.Korea broadband firm sues Netflix after traffic surge

[Doug Barton]

[some snipping below]

Also just to be clear, these are my own opinions, not necessarily shared 
by any current or former employers.


On 10/10/21 12:31 PM, Mark Tinka wrote:



On 10/10/21 21:08, Doug Barton wrote
Given that issue, I have some sympathy for eyeball networks wanting to 
charge content providers for the increased capacity that is needed to 
bring in their content. The cost would be passed on to the content 
provider's customers...


But eyeballs are already paying you a monthly fee for 100Mbps of service 
(for example). So they should pay a surcharge, over-and-above that, that 
determines how they can use that 100Mbps? Seems overly odd, to me.


Yes, I get that. But as you pointed out here and in other comments, the 
ISP market is based entirely on undercutting competitors (with a lot of 
gambling thrown in, as Matthew pointed out).


(in the same way that corporations don't pay taxes, their customers 
do),...



Many a company pays corporate tax, which is separate from the income tax 
they pay for compensation to their staff.


Of course, YMMV depending on where you live.


I didn't say income tax. Corporate taxes are considered an expense by 
the corporation paying them. Like all other expenses, they are factored 
into the cost of goods/services sold.


so the people on that ISP who are creating the increased demand would 
be (indirectly) paying for the increased capacity. That's actually 
fairer for the other customers who aren't Netflix subscribers.


The reason that Netflix doesn't want to do it is the same reason that 
ISPs don't want to charge their customers what it really costs to 
provide them access.


So what rat hole does this lead us down into? People who want to stream 
Youtube should pay their ISP for that? People who want to spend 
unmentionable hours on Linkedin should be their ISP for that? People who 
want to gawk over Samsung's web site because they love it so much, 
should pay their ISP for that?


First, I'm not saying "should." I'm saying that given the market 
economics, having the content providers who use "a lot" of bandwidth do 
something to offset those costs to the ISPs might be the best/least bad 
option. Whether "something" is a local cache box, peering, money, or 
 is something I think that the market should determine.


And to answer Matthew's question, I don't know what "a lot" is. I think 
the market should determine that as well.


And for the record, not only have I never worked for an ISP, I was 
saying all the way back in the late '90s that the oversubscription 
business model (which almost always includes punishing users who 
actually use their bandwidth) is inherently unfair to the customers, and 
when the Internet becomes more pervasive in daily life will come back to 
bite them in the ass. I was laughed at for being hopelessly naive, not 
understanding how the bandwidth business works, etc.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 22:13, Michael Thomas wrote:

Isn't that what Erlang numbers are all about? My suspicion is that 
after about 100Mbs most people wouldn't notice the difference in most 
cases. My ISP is about 25Mbs on a good day (DSL) and it serves our 
needs fine and have never run into bandwidth constraints. Maybe if we 
were streaming 4k all of the time it might be different, but frankly 
the difference for 4k isn't all that big. It's sort of like phone 
screen resolution: at some point it just doesn't matter and becomes 
marketing hype.




The ISP looking to charge BigContent for increased link saturation isn't 
looking at the individual 100Mbps links they have sold to their 
downstream customers.


They are looking at the aggregate Gbps or Tbps of traffic that 
BigContent is seeking to deliver across their network, for "no $$".


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 22:10, Geoff Huston wrote:


I have to agree with Doug Barton's earlier observation is that the base problem 
is that the ISPs are using a flawed business model and they don't want to 
charge their customers what it really costs to provide them with high speed 
access, nor do they want to fund additional back-end capacity in their network 
without some form of offset revenue stream.


I think ISP's do want to charge their customers what it actually costs 
to provide them with a service, but they can't because many ISP's 
business models are based purely on undercutting their nearest competitor.


I might be naive and hopeful to think that operators will have a blood 
handshake to set prices where customers can't wag the tail.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Michael Thomas]

On 10/10/21 12:57 PM, Mark Tinka wrote:



On 10/10/21 21:33, Matthew Petach wrote:


If you sell a service for less than it costs to provide, simply
based on the hopes that people won't actually *use* it, that's
called "gambling", and I have very little sympathy for businesses
that gamble and lose.


You arrived at the crux of the issue, quickly, which was the basis of 
my initial response last week - infrastructure is dying. And we simply 
aren't motivated enough to figure it out.


When you spend 25+ years sitting in a chair waiting for the phone to 
ring or the door to open, for someone to ask, "How much for 5Mbps?", 
your misfortune will never be your own fault.


Isn't that what Erlang numbers are all about? My suspicion is that after 
about 100Mbs most people wouldn't notice the difference in most cases. 
My ISP is about 25Mbs on a good day (DSL) and it serves our needs fine 
and have never run into bandwidth constraints. Maybe if we were 
streaming 4k all of the time it might be different, but frankly the 
difference for 4k isn't all that big. It's sort of like phone screen 
resolution: at some point it just doesn't matter and becomes marketing hype.


Mike

Re: S.Korea broadband firm sues Netflix after traffic surge

[Geoff Huston]

> On 11 Oct 2021, at 6:33 am, Matthew Petach  wrote:
> 
> […] Facebook, Microsoft, and Amazon all caved to SK's demands:
> 
> I will note that my $previous_employer was a top-10 web content provider 
> that did *not* pay SK Broadband.  Not all the content providers caved 
> to SKB.
> 


The situation in South Korea between content providers and broadband providers 
has a long history. Back in early 2012 Korea Telecom implemented a block on 
Samsung’s “smart TV” models because they had a streaming high def content 
service that KT claimed that was saturating their broadband network. According 
to KT, Samsung opted to take a "very negative response" to KT's actions. 
Samsung obtained a court injunction to lift KT's block on their TVs and an 
associated court order for KT and Samsung to enter into arbitration. At the 
same time Samsung filed a lawsuit against KT. In due course the temperature of 
the dispute abated and all the parties backed down. KT discontinued its block, 
and Samsung dropped its lawsuit. However, there was evidently some residual bad 
feeling here as Samsung expressed their desire for the national regulator to 
convey a "strict warning" to KT over its actions.

You have to wonder if the major difference some nine years later is that while 
Samsung is a Korean business, Netflix is a ‘foreign’ entity, and perhaps the 
broadband ISPs feel that the Korean legal actions in this round will have a 
different outcome and favour the local ISP enterprises over the foreign 
streamer.

I have to agree with Doug Barton's earlier observation is that the base problem 
is that the ISPs are using a flawed business model and they don't want to 
charge their customers what it really costs to provide them with high speed 
access, nor do they want to fund additional back-end capacity in their network 
without some form of offset revenue stream.

Geoff

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 21:33, Matthew Petach wrote:


If you sell a service for less than it costs to provide, simply
based on the hopes that people won't actually *use* it, that's
called "gambling", and I have very little sympathy for businesses
that gamble and lose.


You arrived at the crux of the issue, quickly, which was the basis of my 
initial response last week - infrastructure is dying. And we simply 
aren't motivated enough to figure it out.


When you spend 25+ years sitting in a chair waiting for the phone to 
ring or the door to open, for someone to ask, "How much for 5Mbps?", 
your misfortune will never be your own fault.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Sun, Oct 10, 2021 at 12:12 PM Doug Barton  wrote:

> On 10/1/21 7:45 AM, Mark Tinka wrote:
> > The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their
> > own global backbones is because of this nonsense that SK Broadband is
> > trying to pull with Netflix. At some point, the content folk will get
> > fed up, and go build it themselves. What an opportunity infrastructure
> > cost itself!
>
> Except that Facebook, Microsoft, and Amazon all caved to SK's demands:
>

I will note that my $previous_employer was a top-10 web content provider
that did *not* pay SK Broadband.  Not all the content providers caved
to SKB.



> One incentive I haven't seen anyone mention is that ISPs don't want to
> charge customers what it really costs to provide them access. If you're
> the only one in your market that is doing that, no one is going to sign
> up because your pricing would be so far out of line with your competition.
>

That's a problem with your (collective) business model, then.

If you sell something for less than it costs to make, it's called a
loss-leader; and while you can do it for a little while, you'll get
very little sympathy if people take advantage of it to drain your
coffers.

If you sell a service for less than it costs to provide, simply
based on the hopes that people won't actually *use* it, that's
called "gambling", and I have very little sympathy for businesses
that gamble and lose.


> Given that issue, I have some sympathy for eyeball networks wanting to
> charge content providers for the increased capacity that is needed to
> bring in their content. The cost would be passed on to the content
> provider's customers (in the same way that corporations don't pay taxes,
> their customers do), so the people on that ISP who are creating the
> increased demand would be (indirectly) paying for the increased
> capacity. That's actually fairer for the other customers who aren't
> Netflix subscribers.
>

That argument makes no sense whatsoever.

What if instead of a single content provider, the extra traffic
was generated by 10,000 small websites, each adding 1/10,000th
of the volume of a single content provider?

The cumulative impact on the eyeball network to handle the
increased traffic is the same whether it comes from one
content provider or from 10,000 separate smaller websites.

Why should it be OK to go after the one content provider,
but not go after the 10,000 smaller websites?

At one point does your argument break down, and can you
defend why that break point makes sense?  Why is it OK to
go after one, two, three, four content providers, but not to
go after every website that is contributing to the increased
traffic volume the eyeball network is handling?

Seriously.  Make your case.
At what point do you draw that line, and say "we can charge
content sites if there's less than 5 of them, but not if there's
more than 10,000 of them?"
How do you defend the choice of where you drew that
arbitrary line?

The reason that Netflix doesn't want to do it is the same reason that
> ISPs don't want to charge their customers what it really costs to
> provide them access.
>

ISPs who don't charge enough to cover their costs are gambling,
and hoping they get lucky.

When they don't get lucky, and they lose their bet, they shouldn't
get to make up for it by trying to strong-arm others to make up the
difference.

if you decide that "sender pays" is a fair model for the Internet to
follow, then it needs to be applied equally, not just cherry-picking
a few companies to extort, but leaving everyone else alone.

As it stands, what you're arguing for is completely arbitrary and
unfair.

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 21:08, Doug Barton wrote:



Except that Facebook, Microsoft, and Amazon all caved to SK's demands:

"The popularity of the hit series "Squid Game" and other offerings 
have underscored Netflix's status as the country's second-largest data 
traffic generator after Google's YouTube, but the two are the only 
ones to not pay network usage fees, which other content providers such 
as Amazon, Apple and Facebook are paying, SK said."


Which has emboldened SK to go after the bigger fish.


Prior to the popularity of "House Of Cards", Netflix would have bent 
over and taken it without any lube. Heck, they signed away plenty of 
rights around the world to several networks for "House Of Cards", purely 
because they didn't know how well their own in-house production would 
succeed. Fast-forward, it's 2021 now.


Other players in BigContent that haven't yet found their leverage, will do.




One incentive I haven't seen anyone mention is that ISPs don't want to 
charge customers what it really costs to provide them access. If 
you're the only one in your market that is doing that, no one is going 
to sign up because your pricing would be so far out of line with your 
competition.


Isn't this the curse of a service people consider to be a basic utility 
for life to occur?


Unlike water and power, nearly anyone can start an ISP, and further feed 
the race to the bottom.





Given that issue, I have some sympathy for eyeball networks wanting to 
charge content providers for the increased capacity that is needed to 
bring in their content. The cost would be passed on to the content 
provider's customers...


But eyeballs are already paying you a monthly fee for 100Mbps of service 
(for example). So they should pay a surcharge, over-and-above that, that 
determines how they can use that 100Mbps? Seems overly odd, to me.



(in the same way that corporations don't pay taxes, their customers 
do),...



Many a company pays corporate tax, which is separate from the income tax 
they pay for compensation to their staff.


Of course, YMMV depending on where you live.


so the people on that ISP who are creating the increased demand would 
be (indirectly) paying for the increased capacity. That's actually 
fairer for the other customers who aren't Netflix subscribers.


The reason that Netflix doesn't want to do it is the same reason that 
ISPs don't want to charge their customers what it really costs to 
provide them access.


So what rat hole does this lead us down into? People who want to stream 
Youtube should pay their ISP for that? People who want to spend 
unmentionable hours on Linkedin should be their ISP for that? People who 
want to gawk over Samsung's web site because they love it so much, 
should pay their ISP for that?


Hey, maybe you're right. Maybe that's the model that is needed. After 
all, when we go to a rave, the entry fee is just the entry fee. You 
still need to fork out more cash to actually buy drinks, food or engage 
in some kind of entertainment that may be taking place inside that 
walled garden you paid a cover charge to be a part of.


I don't know...

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Doug Barton]

On 10/1/21 7:45 AM, Mark Tinka wrote:
The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their 
own global backbones is because of this nonsense that SK Broadband is 
trying to pull with Netflix. At some point, the content folk will get 
fed up, and go build it themselves. What an opportunity infrastructure 
cost itself!


Except that Facebook, Microsoft, and Amazon all caved to SK's demands:

"The popularity of the hit series "Squid Game" and other offerings have 
underscored Netflix's status as the country's second-largest data 
traffic generator after Google's YouTube, but the two are the only ones 
to not pay network usage fees, which other content providers such as 
Amazon, Apple and Facebook are paying, SK said."


Which has emboldened SK to go after the bigger fish.

One incentive I haven't seen anyone mention is that ISPs don't want to 
charge customers what it really costs to provide them access. If you're 
the only one in your market that is doing that, no one is going to sign 
up because your pricing would be so far out of line with your competition.


Given that issue, I have some sympathy for eyeball networks wanting to 
charge content providers for the increased capacity that is needed to 
bring in their content. The cost would be passed on to the content 
provider's customers (in the same way that corporations don't pay taxes, 
their customers do), so the people on that ISP who are creating the 
increased demand would be (indirectly) paying for the increased 
capacity. That's actually fairer for the other customers who aren't 
Netflix subscribers.


The reason that Netflix doesn't want to do it is the same reason that 
ISPs don't want to charge their customers what it really costs to 
provide them access.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

The bottom line problem is that we have allowed vertical integration to allow
the natural monopoly that exists in last mile infrastructure in most locations 
to
be leveraged into an effective full-stack monopoly for those same players.

Lack of competition in the last-mile/eyeball space has allowed larger monopoly
eyeball providers to leverage double-payment for the same traffic, extorting
eyeballs that have no alternative to pay them to deliver the content while
simultaneously turning around and extorting the content providers themselves
to be allowed to reach “their customers”.

While monopoly $CABLECO/$DSLTELCO/$GPONPROVIDER can generally
get away making access problematic for a handful of content providers for
short periods of time, no content provider can really absorb the losses 
associated
with allowing that situation to continue, thus giving the content providers
leverage.

Unfortunately for the content providers, newer laws and lower costs to provide
higher bandwidth are making WISPs a viable competitor in both rural and
metropolitan settings. Consumers are catching on to this and telling the more
traditional ISPs that they now have a choice and the ISPs will have to up their
game to keep their business, so finally progress is being made, which is
removing that leverage from the previously monopoly providers on both
the consumer and the content provider side of that equation.

Obviously, ISPs don’t like this and the monopoly ones being not in the
communications business, but rather operating as a law firm with some
switching infrastructure will attempt to use legislation, PUC, and courts
as weapons to try and preserve the status quo.

I’m hopeful that the Korean courts will see SK’s shakedown for what it is
and toss it in summary judgment (or whatever the Korean equivalent is)
with costs and attorneys’ fees awarded to Netflix. I’m hoping that this will
also send a clear message to other ISPs contemplating such extortive
behavior.

OTOH, I admit I will watch in amusement if SK’s customers trying to play
Netflix videos are only able to watch in 480p after viewing a brief video
explaining that the video they are about to watch is degraded courtesy
of SK’s business practices (obviously with appropriate details).

Owen


> On Oct 1, 2021, at 11:24 , Joshua Pool via NANOG  wrote:
> 
> I think instances where the end ISP is peered directly with Netflix and 
> demands more money is not valid at all.  That should be normal cost of doing 
> business to increase capacity as the consumer demand grows.
> The topic of interest is instances where the ISP is not directly peered with 
> Netflix and uses upstream providers and those providers are trying to make 
> content providers absorb the cost of increasing peering capacity for services 
> that traverse their infrastructure.
> One could make the argument that Tier1's should never be the choke point as 
> they should be keeping up with the times and be proactively increasing 
> capacity.
> One could also note that it's 2021 and Cogent and Hurricane Electric are 
> still not peered.
> 
> 
> 
> 
> On Fri, Oct 1, 2021 at 10:47 AM Blake Hudson  > wrote:
> 
> 
> On 10/1/2021 11:23 AM, Sean Donelan wrote:
> >
> >
> > In the old days, postal services used to charge the recipient of a 
> > letter to deliver the letter. Then stamps were invented, and postal 
> > services charged the sender of the letter, and the recipent got free 
> > delivery.
> >
> > Now there is free-shipping, and pre-paid return envelopes for DVDs.
> >
> > Of course, the shipping isn't really "Free."  Its built into the cost 
> > of goods sold.
> >
> > There is no universal, fixed, unchangable way of allocating business 
> > costs.
> >
> 
> True. But when the sender has already paid the stamp to their courier to 
> deliver the bits on their leg of the journey, and the recipient has 
> already paid a stamp to deliver the bits on their leg of the journey, 
> what case does the recipient's courier have to demand additional payment 
> from the sender (lest the package get lost)? The stamp has already been 
> paid. TWICE! Withholding service until additional payments are made just 
> smells like extortion.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 18:31, Tom Hill wrote:


Many (most?) route servers provide little control over who your routes
are advertised toward. This can be fun where DDoS is concerned.

I've used some that did have deny-list controls for ASNs, fail to
consistently apply those rules. Again, that was a 'fun' surprise.


Nowadays, well-run exchange points would have solved this. The reason we 
don't use route servers anymore is because they sometimes break (either 
due to code, or from people), and the policies you thought would give a 
good 3AM sleep suddenly aren't working anymore.


It was cheaper for us to maintain bi-lateral sessions.

That said, route servers have their place, and I won't knock them down.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 18:05, Laura Smith wrote:


Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps 
you could answer me this riddle.

- PoP in one of your "half-decent data centres" ... tick.
- Connnection to one of your "exchange point" ... tick.
- $certain_large_cdn present on said "exchange point" ... tick.

And yet .

- $certain_large_cdn publishes routes on route server ? Nope.
- $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where 
the large networks see it beneath them to peer with (or publish routes for the benefit 
of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off 
their nose to spite their face ?


Yes, this is a rather painful one, and it has been on the rise in the 
last few years as the major content folk struggle to keep up with 
peering requirements, and everything that goes along with maintaining 
those relationships.


They also seem to have a number of complex operational requirements 
between their own backbones, their own PoP's, partner PoP's, transit 
links, e.t.c.


There is no easy answer for this one, apart from doing the leg-work to 
find out who the best person at the other end to speak to is. I'd 
recommend working with the data centre and exchange point operators to 
make meaningful introductions. You may not end up where you want, but 
you certainly increase your chances of doing so.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Laura Smith via NANOG]

Thanks for your insight Matt, much appreciated.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Fri, Oct 1, 2021 at 11:27 AM Joshua Pool via NANOG 
wrote:

> [...]
> One could also note that it's 2021 and Cogent and Hurricane Electric are
> still not peered.
>

Bugger.

You're right.

I forgot to add "stupid human ego issues" to my list of reasons
why direct peering requests get ignored or rejected.  :/

Thanks for the reminder.   ^_^;;

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Blake Hudson]

I wasn't aware of that, but I think that's perfect! And completely 
reasonable on Netflix (or any content provider's part).


I'm sure Verizon's wordsmiths would argue that the "crowding" happened 
upstream of the Verizon network, but if stated another way (like "the 
paths into Verizon's network are full") anyone can see that this is an 
issue that Verizon made and only Verizon could solve. Netflix isn't, and 
shouldn't be, responsible for runing Verizon's network. Only Verizon 
runs the Verizon network, and it's up to Verizon to deliver the service 
they advertise and sell to consumers: "America's most reliable network" 
(TM).


On 10/1/2021 1:20 PM, Jay Hennigan wrote:

On 10/1/21 07:19, Blake Hudson wrote:

It's about time Netflix played chicken with one of these ISPs and 
stopped offering service  (or offered limited service) to the ISPs 
that try to extort them and other content providers: Sorry, your 
service provider does not believe in net neutrality and has imposed 
limitations on your Netflix experience. For a better Netflix 
experience, consider exploring one of these other nearby internet 
providers: x, y, z.


Netflix actually did pretty much exactly that with Verizon back in 
2014, displaying a message that read "The Verizon network is crowded 
right now."


https://techcrunch.com/2014/06/05/netflix-error-blames-verizon-for-playback-issues-verizon-responds-with-legal-threats/ 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Fri, Oct 1, 2021 at 9:08 AM Laura Smith via NANOG 
wrote:

>
> > The bad news now, is, there are plenty of many, small, local
> > and regional ISP's who are willing to do whatever it takes to
> > work with the content providers. All that's required is some
> > network, a half-decent data centre and an exchange point. Gone
> > are the days where customers clamored to sign up with Big
> > Telco.
>
> Speaking as one of those smaller ISPs willing to do whatever it takes,
> perhaps you could answer me this riddle.
>
> - PoP in one of your "half-decent data centres" ... tick.
> - Connnection to one of your "exchange point" ... tick.
> - $certain_large_cdn present on said "exchange point" ... tick.
>
> And yet .
>
> - $certain_large_cdn publishes routes on route server ? Nope.
> - $certain_large_cdn willing to establish direct peering session ? Nope.
>
> I am well aware of the "big boys club" that operates at most exchanges
> where the large networks see it beneath them to peer with (or publish
> routes for the benefit of) the unwashed masses.
>
> But I struggle to comprehend why $certain_large_cdn would effectively cut
> off their nose to spite their face ?
>
>
Having worked on building out a global content delivery network,
I can hopefully shed some light here.

Let's imagine a scenario where you have a CDN node attached to
a large peering location.

At that peering location, you have Nx100G direct peering links to
large eyeball networks, and Nx10G direct peering links to midsized
eyeball networks.

You have a historical connection into the peering switch as well,
which you've kept around and upgraded as time went by.  Life
is generally good.  You've got your own backbone between
your locations, and you capacity engineer your infrastructure
so that if your peering sessions go down at that site, users
are redirected to the next closest CDN node.

Then one day, Bad Juju(tm) happens.  The router that your
Nx100G links with Large Eyeball Network A terminate on
goes down, doesn't matter if it's on their end or your end,
those direct sessions go down.  But instead of your traffic
failing over to site B instead, it continues flowing--but now
it's going through the route server sessions, across the
now grossly-undersized connection into the public peering
switch, and *nobody* is happy about that.  You tried putting
import and export lists into the route server, but they don't
seem to be taking effect, and the phone is ringing off the
hook, so you simply stop exporting routes into the route
server, and suddenly traffic is now failing over to site B
for all the eyeball requests from Large Eyeball Network A,
and you get to explain to your boss how those sessions
with the route servers caused the failover plan to fail badly.

Decision is made that route server peering sessions are
too far removed from direct control to be safe, and the
only way to effectively prevent this scenario from happening
again is to stop announcing routes to the route servers.

You're a smaller network; you're caught up in this, because you'd
been getting routes from the route server, and now you're not.
You request direct peering; unfortunately, the ports on the existing
line cards on the CDN edge routers are full, and it's going to be
months before the next capacity upgrade is planned; and those
line cards are all 100G cards, with no 10G ports planned.  It's
decided that letting that traffic get handled by the transit ports is
cheaper than trying to get some additional 10G peering port capacity
at the site.  And so, your request for direct peering is denied; there's
just no place to plug you in on the router on the other side, unless
you can drum up enough traffic to justify a 100G peering link.

Establishing direct peering across the peering fabric becomes
the only possible point of commonality, then; but that requires
adding explicit neighbor configuration into the router, unlike
the route-server mediated prefix exchange; and now you're
still running into limitations on the edge routers--though in this
case it's not physical port limitations, it's neighbor adjacency
limits on the routers.  The more neighbors you configure on an
edge router, the longer the configs are, and depending on how
the neighbors are established, the more processing power and
memory it takes on the router.  At a certain point, there's just not
enough to go around, and the CDN makes the decision to not add
any new BGP neighbors at that site.  A few years down the line,
when hardware is added/upgraded, they may revisit that decision,
or it may persist, even after the upgrade, due simply to corporate
inertia--nobody goes back to revisit the decision, and see if it still
makes sense or not.

Sometimes it's just a case of limited humans; without good automation
tools in place, adding BGP adjacencies takes a ticket being created
in a request system, and then another more highly skilled human
going into the router to configure the neighbor.  If you've got a limited

Re: S.Korea broadband firm sues Netflix after traffic surge

[Joshua Pool via NANOG]

I think instances where the end ISP is peered directly with Netflix and
demands more money is not valid at all.  That should be normal cost of
doing business to increase capacity as the consumer demand grows.
The topic of interest is instances where the ISP is not directly peered
with Netflix and uses upstream providers and those providers are trying to
make content providers absorb the cost of increasing peering capacity for
services that traverse their infrastructure.
One could make the argument that Tier1's should never be the choke point as
they should be keeping up with the times and be proactively increasing
capacity.
One could also note that it's 2021 and Cogent and Hurricane Electric are
still not peered.




On Fri, Oct 1, 2021 at 10:47 AM Blake Hudson  wrote:

>
>
> On 10/1/2021 11:23 AM, Sean Donelan wrote:
> >
> >
> > In the old days, postal services used to charge the recipient of a
> > letter to deliver the letter. Then stamps were invented, and postal
> > services charged the sender of the letter, and the recipent got free
> > delivery.
> >
> > Now there is free-shipping, and pre-paid return envelopes for DVDs.
> >
> > Of course, the shipping isn't really "Free."  Its built into the cost
> > of goods sold.
> >
> > There is no universal, fixed, unchangable way of allocating business
> > costs.
> >
>
> True. But when the sender has already paid the stamp to their courier to
> deliver the bits on their leg of the journey, and the recipient has
> already paid a stamp to deliver the bits on their leg of the journey,
> what case does the recipient's courier have to demand additional payment
> from the sender (lest the package get lost)? The stamp has already been
> paid. TWICE! Withholding service until additional payments are made just
> smells like extortion.
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Jay Hennigan]

On 10/1/21 07:19, Blake Hudson wrote:

It's about time Netflix played 
chicken with one of these ISPs and stopped offering service  (or offered 
limited service) to the ISPs that try to extort them and other content 
providers: Sorry, your service provider does not believe in net 
neutrality and has imposed limitations on your Netflix experience. For a 
better Netflix experience, consider exploring one of these other nearby 
internet providers: x, y, z.


Netflix actually did pretty much exactly that with Verizon back in 2014, 
displaying a message that read "The Verizon network is crowded right now."


https://techcrunch.com/2014/06/05/netflix-error-blames-verizon-for-playback-issues-verizon-responds-with-legal-threats/

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: S.Korea broadband firm sues Netflix after traffic surge

[Blake Hudson]

On 10/1/2021 11:23 AM, Sean Donelan wrote:



In the old days, postal services used to charge the recipient of a 
letter to deliver the letter. Then stamps were invented, and postal 
services charged the sender of the letter, and the recipent got free 
delivery.


Now there is free-shipping, and pre-paid return envelopes for DVDs.

Of course, the shipping isn't really "Free."  Its built into the cost 
of goods sold.


There is no universal, fixed, unchangable way of allocating business 
costs.




True. But when the sender has already paid the stamp to their courier to 
deliver the bits on their leg of the journey, and the recipient has 
already paid a stamp to deliver the bits on their leg of the journey, 
what case does the recipient's courier have to demand additional payment 
from the sender (lest the package get lost)? The stamp has already been 
paid. TWICE! Withholding service until additional payments are made just 
smells like extortion.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Tom Hill]

On 01/10/2021 17:05, Laura Smith via NANOG wrote:
> - $certain_large_cdn publishes routes on route server ? Nope.

Many (most?) route servers provide little control over who your routes
are advertised toward. This can be fun where DDoS is concerned.

I've used some that did have deny-list controls for ASNs, fail to
consistently apply those rules. Again, that was a 'fun' surprise.

> - $certain_large_cdn willing to establish direct peering session ? Nope.

There is a non-zero cost to peering. Many CDNs are happy to send cache
boxes/setup peering sessions for small peers, but their definition of
"small" will no doubt vary between CDNs, based on their perspective &
business costs. Some networks may fall well below each individual
network's thresholds, and indeed some networks may have different
thresholds between countries.

It's never as simple as "why don't you peer???"

Regards,

-- 
Tom

Re: S.Korea broadband firm sues Netflix after traffic surge

[jim deleskie]

Having done peering for many $big_boys_club and $small_isps, it always
comes down to politics, $$ and time.  The balance may change but end of day
its those variables and its a painful game some days.  From all sides :(


-jim

On Fri, Oct 1, 2021 at 1:07 PM Laura Smith via NANOG 
wrote:

>
> > The bad news now, is, there are plenty of many, small, local
> > and regional ISP's who are willing to do whatever it takes to
> > work with the content providers. All that's required is some
> > network, a half-decent data centre and an exchange point. Gone
> > are the days where customers clamored to sign up with Big
> > Telco.
>
> Speaking as one of those smaller ISPs willing to do whatever it takes,
> perhaps you could answer me this riddle.
>
> - PoP in one of your "half-decent data centres" ... tick.
> - Connnection to one of your "exchange point" ... tick.
> - $certain_large_cdn present on said "exchange point" ... tick.
>
> And yet .
>
> - $certain_large_cdn publishes routes on route server ? Nope.
> - $certain_large_cdn willing to establish direct peering session ? Nope.
>
> I am well aware of the "big boys club" that operates at most exchanges
> where the large networks see it beneath them to peer with (or publish
> routes for the benefit of) the unwashed masses.
>
> But I struggle to comprehend why $certain_large_cdn would effectively cut
> off their nose to spite their face ?
>

Re: S.Korea broadband firm sues Netflix after traffic surge

[Sean Donelan]

In the old days, postal services used to charge the recipient of a letter 
to deliver the letter. Then stamps were invented, and postal services 
charged the sender of the letter, and the recipent got free delivery.


Now there is free-shipping, and pre-paid return envelopes for DVDs.

Of course, the shipping isn't really "Free."  Its built into the cost of 
goods sold.


There is no universal, fixed, unchangable way of allocating business 
costs.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Laura Smith via NANOG]

> The bad news now, is, there are plenty of many, small, local
> and regional ISP's who are willing to do whatever it takes to
> work with the content providers. All that's required is some
> network, a half-decent data centre and an exchange point. Gone
> are the days where customers clamored to sign up with Big
> Telco.

Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps 
you could answer me this riddle.

- PoP in one of your "half-decent data centres" ... tick.
- Connnection to one of your "exchange point" ... tick.
- $certain_large_cdn present on said "exchange point" ... tick.

And yet .

- $certain_large_cdn publishes routes on route server ? Nope.
- $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where 
the large networks see it beneath them to peer with (or publish routes for the 
benefit of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off 
their nose to spite their face ?

Re: S.Korea broadband firm sues Netflix after traffic surge

[d...@darwincosta.com]

> On 1 Oct 2021, at 16:46, Mark Tinka  wrote:
> 
> 
> 
>> On 10/1/21 16:19, Blake Hudson wrote:
>> 
>> 
>> I'll never understand over how ISPs see content providers as the enemy (or a 
>> rival). The content is why ISPs have customers. Don't get upset when your 
>> customer uses the service that you sold them (in a way that is precisely in 
>> accordance with the expected usage)!
> 
> It's because infrastructure (that's us, the network operators), still don't 
> get it.
> 
> We are no longer front & centre in the eyes of our users. They see us as an 
> impediment... providers they must buy costly megabytes of mobile data from, 
> providers they must call to fix broken fibre, providers they must shout at 
> when a single CGN IPv4 address they sit behind breaks their Netflix, and so 
> on and so on.
> 
> Users only care about the service they use their mobile phone, tablet, 
> console or laptop for. They don't care how many customers their ISP has, 
> whether the ISP is a small mom & pop or some global behemoth, or whether the 
> ISP's CEO is was on the cover of TIME magazine last week.
> 
> As my American friend used to say, "They just want their MTV".
> 
> In the late 90's and early 2000's, when content folk wanted to work with us, 
> infrastructure folk, to grow their businesses, we just saw easy, free money 
> to tax toward our shiny new Lamborghinis and beach side holiday villas. Well, 
> guess whom we are now begging for seats on their submarine cable build 
> projects, community funding programs, and caches to be installed in our 
> not-so-huge data centres, all for free?
> 
> The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own 
> global backbones is because of this nonsense that SK Broadband is trying to 
> pull with Netflix. At some point, the content folk will get fed up, and go 
> build it themselves. What an opportunity infrastructure cost itself!
> 
> Akamai have also quietly been building their own backbone. Wonder why.
> 
> No doubt Netflix have someone either thinking about the same, or putting a 
> plan into motion.
> 
> The bad news now, is, there are plenty of many, small, local and regional 
> ISP's who are willing to do whatever it takes to work with the content 
> providers. All that's required is some network, a half-decent data centre and 
> an exchange point. Gone are the days where customers clamored to sign up with 
> Big Telco.
> 
> If anyone wonders why "infrastructure is dead", well, this is why.
> 
> 21 years later, and we still don't get it! No wonder the mobile companies are 
> watching their slow death, from the rosy days of billions from basic SMS, to 
> the perils of 5G investments for diddly return.
> 
> Wake me up when all this is over. I'll be in my wine stupor until then.
I couldn’t agree more. 
> 
> Mark.

Cheers. 
Darwin-. 

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 16:19, Blake Hudson wrote:



I'll never understand over how ISPs see content providers as the enemy 
(or a rival). The content is why ISPs have customers. Don't get upset 
when your customer uses the service that you sold them (in a way that 
is precisely in accordance with the expected usage)!


It's because infrastructure (that's us, the network operators), still 
don't get it.


We are no longer front & centre in the eyes of our users. They see us as 
an impediment... providers they must buy costly megabytes of mobile data 
from, providers they must call to fix broken fibre, providers they must 
shout at when a single CGN IPv4 address they sit behind breaks their 
Netflix, and so on and so on.


Users only care about the service they use their mobile phone, tablet, 
console or laptop for. They don't care how many customers their ISP has, 
whether the ISP is a small mom & pop or some global behemoth, or whether 
the ISP's CEO is was on the cover of TIME magazine last week.


As my American friend used to say, "They just want their MTV".

In the late 90's and early 2000's, when content folk wanted to work with 
us, infrastructure folk, to grow their businesses, we just saw easy, 
free money to tax toward our shiny new Lamborghinis and beach side 
holiday villas. Well, guess whom we are now begging for seats on their 
submarine cable build projects, community funding programs, and caches 
to be installed in our not-so-huge data centres, all for free?


The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their 
own global backbones is because of this nonsense that SK Broadband is 
trying to pull with Netflix. At some point, the content folk will get 
fed up, and go build it themselves. What an opportunity infrastructure 
cost itself!


Akamai have also quietly been building their own backbone. Wonder why.

No doubt Netflix have someone either thinking about the same, or putting 
a plan into motion.


The bad news now, is, there are plenty of many, small, local and 
regional ISP's who are willing to do whatever it takes to work with the 
content providers. All that's required is some network, a half-decent 
data centre and an exchange point. Gone are the days where customers 
clamored to sign up with Big Telco.


If anyone wonders why "infrastructure is dead", well, this is why.

21 years later, and we still don't get it! No wonder the mobile 
companies are watching their slow death, from the rosy days of billions 
from basic SMS, to the perils of 5G investments for diddly return.


Wake me up when all this is over. I'll be in my wine stupor until then.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Blake Hudson]

On 10/1/2021 8:48 AM, Sean Donelan wrote:
South Korean Internet service provider SK Broadband has sued Netflix 
to pay for costs from increased network traffic and maintenance work 
because of a surge of viewers to the U.S. firm's content, an SK 
spokesperson said on Friday.

[...]
Last year, Netflix had brought its own lawsuit on whether it had any 
obligation to pay SK for network usage, arguing Netflix's duty ends 
with creating content and leaving it accessible. It said SK's expenses 
were incurred while fulfilling its contractual obligations to Internet 
users, and delivery in the Internet world is "free of charge as a 
principle", according to court documents.

[...]

https://www.reuters.com/business/media-telecom/skorea-broadband-firm-sues-netflix-after-traffic-surge-squid-game-2021-10-01/ 





I'll never understand over how ISPs see content providers as the enemy 
(or a rival). The content is why ISPs have customers. Don't get upset 
when your customer uses the service that you sold them (in a way that is 
precisely in accordance with the expected usage)!


Netflix, as an example, has even been willing to bear most of the cost 
with peering or bringing servers to ISPs to reduce the ISP's costs and 
improve the ISP customer's experience. It's about time Netflix played 
chicken with one of these ISPs and stopped offering service  (or offered 
limited service) to the ISPs that try to extort them and other content 
providers: Sorry, your service provider does not believe in net 
neutrality and has imposed limitations on your Netflix experience. For a 
better Netflix experience, consider exploring one of these other nearby 
internet providers: x, y, z.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Suresh Ramasubramanian]

Yet another peering dispute ending in litigation?

From: NANOG  on behalf of Sean 
Donelan 
Date: Friday, 1 October 2021 at 7:21 PM
To: nanog@nanog.org 
Subject: S.Korea broadband firm sues Netflix after traffic surge
South Korean Internet service provider SK Broadband has sued Netflix to
pay for costs from increased network traffic and maintenance work because
of a surge of viewers to the U.S. firm's content, an SK spokesperson said
on Friday.
[...]
Last year, Netflix had brought its own lawsuit on whether it had any
obligation to pay SK for network usage, arguing Netflix's duty ends with
creating content and leaving it accessible. It said SK's expenses were
incurred while fulfilling its contractual obligations to Internet users,
and delivery in the Internet world is "free of charge as a principle",
according to court documents.
[...]

https://www.reuters.com/business/media-telecom/skorea-broadband-firm-sues-netflix-after-traffic-surge-squid-game-2021-10-01/