Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 13:57 , Matthew Walster  wrote:
> 
> 
> 
> On Mon, 11 Oct 2021 at 21:05, Matthew Petach  > wrote:
> I think it would be absolutely *stunning* for content providers 
> to turn the model on its head; use a bittorrent like model for 
> caching and serving content out of subscribers homes at 
> recalcitrant ISPs, so that data doesn't come from outside, 
> it comes out of the mesh within the eyeball network, with 
> no clear place for the ISP to stick a $$$ bill to.
> 
> Ignoring for the moment that P2P is inherently difficult to stream with 
> (you're usually downloading chunks in parallel, and with devices like Smart 
> TVs etc you don't really have the storage to do so anyway) there's also the 
> problem that things like BitTorrent don't know network topology and therefore 
> only really increases the cross-sectional bandwidth required.

A 4K 2 hour movie is about 40GB. Most modern smart TVs around 32GB of RAM and 
can probably devote about 20GB of that to buffering a stream, so yeah, that 
should actually be doable.

While torrent-like distribution isn’t particularly good for the eyeball 
provider, it can be good for getting content to eyeballs under some 
circumstances regardless of how bad it is for said network.

Unfortunately, it’s not good at knowing how bad it’s being for the network and 
it’s also not good at detecting the circumstances when it’s good for the end 
user or not.

> Not to mention that it has been tried before, and didn't work then either.

Yep.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 13:05 , Matthew Petach  wrote:
> 
> 
> 
> On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka  wrote:
> However, in an era where content is making a push to get as close to the 
> eyeballs as possible, kit getting cheaper and faster because of merchant 
> silicon, and abundance of aggregated capacity at exchange points, can we 
> leverage the shorter, faster links to change the model?
> 
> Mark.
> 
> I think it would be absolutely *stunning* for content providers 
> to turn the model on its head; use a bittorrent like model for 
> caching and serving content out of subscribers homes at 
> recalcitrant ISPs, so that data doesn't come from outside, 
> it comes out of the mesh within the eyeball network, with 
> no clear place for the ISP to stick a $$$ bill to.

I worked for a company that tried this model several years back.
It did not go well. Users were not happy. Eyeball ISPs were very
not happy (and started actively mitigating users who participated
through a variety of nefarious means), and content providers got
a bit hot under the collar about it (the experiment was a content
aggregator for lack of a better term).

> Imagine you've got a movie; you slice it into 1,000 
> encrypted chunks; you make part of your license 
> agreement for customers a requirement that they 
> will allow you to use up to 20GB of disk space on 
> their computer and to serve up data chunks into 
> the network in return for a slightly cheaper monthly 
> subscription cost to your service.  You put 1 slice 
> of that movie on each of 1,000 customers in a 
> network; then you replicate that across the next 
> thousand customers, and again for the next 
> thousand, until you've got enough replicas of 
> each shard to handle a particular household 
> going offline.  Your library is still safe from 
> piracy, no household has more than 1/1000th of a 
> movie locally, and they don't have the key to decrypt 
> it anyhow; but they've got 1/1000th of 4, different 
> movies, and when someone in that ISP wants to watch the 
> movie, the chunks are being fetched from other households 
> within the eyeball network.  The content provider would have 
> shard servers in region, able to serve up any missing shards that 
> can't be fetched locally within the ISP--but the idea would be that 
> as the number of subscribers within an ISP goes up, instead of the 
> ISP seeing a large, single-point-source increase in traffic, what they 
> see is an overall increase in east-west traffic among their users.

That’s a pretty close description of exactly what we did.

> Because the "serving of shards to others" happens primarily while the 
> user is actively streaming content, you have a natural bell curve; during 
> peak streaming times, you have more nodes active to serve up shards, 
> handling the increased demand; at lower demand times, when fewer 
> people are active, and there's fewer home-nodes to serve shards, the 
> content network's shard servers can pick up the slack...but that'll generally 
> only happen during lower traffic times, when the traffic won't be competing 
> and potentially causing pain for the ISP.

It’s a beautiful theory. We expected this to be the case, too. Turns out, not
so much. There’s an awful lot of long tail out there that is poorly accounted
for in this. Yes, it does help with some of the most popular content, but there
are pitfalls there, too.

> Really, it seems like a win-win scenario.

We thought so too, until we learned otherwise. Today, with more symmetrical
connections and larger uplinks, it might be more feasible. Back then (Around
2005), it was a good idea on paper.

> I'm confident we'll see a content network come out with a model like this 
> within the next 5 years, at which point the notion of blackmailing content 
> networks for additional $$$s will be a moot point, because the content will 
> be distributed and embedded within every major eyeball network already,
> whether they like it or not, on their customer's devices.

You’d be surprised at the number of different ways eyeball providers have
to denature such an effort. I’m sorry to rain on your parade, but it’s not a new
idea. I was brought in as the operations guy once the software engineers
realized that they needed someone who could, you know, keep stuff running
and not just write code and test it in production. This model resulted in a
frank and uncomfortable conversation between me and the developer of
what was internally being called the “overlay network” code. We went through
a number of different scenarios where he had made incorrect assumptions
about how the internet worked, especially at the eyeball end and there
were a dozen or so that we simply couldn’t find ways to work around.

Networks are a bit different now and some content providers have people
far more clever than I working on things like this, so who knows… It might
succeed by 2026, but when we tried it in 2005, we couldn’t make it work
well enough to avoid 

ARIN BoT ballot petition underway..

[Ron da Silva]

Just bringing to attention of the broader audience here, and since aria-discuss 
no longer exists there’s also been some discussion on ppml...  Dan Alexandar 
and I are both exercising the petition process to be added to the ballot for 
the ARIN Board of Trustees election.  There are two seats up and only 3 
candidates on the ballot produced by the nominating committee.  If you are the 
DMR (or know the DMR for your org), please support by simply logging into your 
ARIN account and following the “vote now” link.

https://www.arin.net/announcements/20211004_petitions/?fbclid=IwAR1yPdai7ZF5Pi2zU0yk3-xVGCynwIgsIxGPjo93VWErRdP3EPSXHRGFBI4
 


Thanks in advance!
-ron

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Walster]

On Mon, 11 Oct 2021 at 21:05, Matthew Petach  wrote:

> I think it would be absolutely *stunning* for content providers
> to turn the model on its head; use a bittorrent like model for
> caching and serving content out of subscribers homes at
> recalcitrant ISPs, so that data doesn't come from outside,
> it comes out of the mesh within the eyeball network, with
> no clear place for the ISP to stick a $$$ bill to.
>

Ignoring for the moment that P2P is inherently difficult to stream with
(you're usually downloading chunks in parallel, and with devices like Smart
TVs etc you don't really have the storage to do so anyway) there's also the
problem that things like BitTorrent don't know network topology and
therefore only really increases the cross-sectional bandwidth required.

Not to mention that it has been tried before, and didn't work then either.

M

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Mon, Oct 11, 2021 at 10:09 AM Michael Thomas  wrote:

>
> On 10/11/21 12:49 AM, Matthew Petach wrote:
>
>
> Instead of a 4K stream, drop it to 480 or 240; the eyeball network
> should be happy at the reduced strain the resulting stream puts
> on their network.
>
> As a consumer paying for my 4k stream, I know who I'm calling when it
> drops to 480 and it ain't Netflix. The eyeballs are most definitely not
> happy.
>
> Mike
>

I apologize for that.  I was tired after two back-to-back days
of board meetings, and I missed putting a clear sarcasm
marker on that last line about "the eyeball networks
should be happy at the reduced strain...":(

There should have been a clear ;-P at the end of
the line to make it unmistakeable I was poking a
very sharp stick at the eyeball networks and
what it takes to actually make them happy.  ^_^;

Yes--the end consumers really shouldn't be the hostage
in this battle, being moved about the chess board by
either side, whether by their ISP trying to squeeze
more money out of the content side, or by the content
side trying to force more complaints into the service
desk of the ISP.

I mean, imagine this scenario for any other utility.

Pacific Gas and Electric calling up Hoover Dam to
say "hey, we're going to need to charge you some
additional money this month."

Hoover Dam: "...what?"

PGE: "well, you're sending a lot more electricity to
our customers this month, and we're going to have
to upgrade our power lines to handle it; and since
you're the one sending the electricity, you should
pay for part of the costs."

Hoover Dam: "...we're only sending enough electricity
to meet the demands YOUR customers are placing on
the grid.  If they want to run their air conditioners all
summer long, you need to charge them enough to
cover your costs for it."

Drat.  My analogy just ran out, because I realize the
dollars already flow the other way, and the hydroelectric
station would just laugh at PG and threaten to raise
the cost of the electricity simply for having to listen to their BS.   ^_^;

You can run the same scenario with your municipal water
company, and imagine how it would play out if the municipality
that put the pipes in to every home tried to charge the water
supplier more because homes were taking longer showers.

It's just such a fundamentally broken model, we laugh at it
in any other industry.  :(

Again, I'm sorry for being tired and missing the explicit
sarcasm indicator--not just for you, but for others who also
responded to that paragraph.   ^_^;

Thanks!

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka  wrote:

> However, in an era where content is making a push to get as close to the
> eyeballs as possible, kit getting cheaper and faster because of merchant
> silicon, and abundance of aggregated capacity at exchange points, can we
> leverage the shorter, faster links to change the model?
>
> Mark.
>

I think it would be absolutely *stunning* for content providers
to turn the model on its head; use a bittorrent like model for
caching and serving content out of subscribers homes at
recalcitrant ISPs, so that data doesn't come from outside,
it comes out of the mesh within the eyeball network, with
no clear place for the ISP to stick a $$$ bill to.

Imagine you've got a movie; you slice it into 1,000
encrypted chunks; you make part of your license
agreement for customers a requirement that they
will allow you to use up to 20GB of disk space on
their computer and to serve up data chunks into
the network in return for a slightly cheaper monthly
subscription cost to your service.  You put 1 slice
of that movie on each of 1,000 customers in a
network; then you replicate that across the next
thousand customers, and again for the next
thousand, until you've got enough replicas of
each shard to handle a particular household
going offline.  Your library is still safe from
piracy, no household has more than 1/1000th of a
movie locally, and they don't have the key to decrypt
it anyhow; but they've got 1/1000th of 4, different
movies, and when someone in that ISP wants to watch the
movie, the chunks are being fetched from other households
within the eyeball network.  The content provider would have
shard servers in region, able to serve up any missing shards that
can't be fetched locally within the ISP--but the idea would be that
as the number of subscribers within an ISP goes up, instead of the
ISP seeing a large, single-point-source increase in traffic, what they
see is an overall increase in east-west traffic among their users.

Because the "serving of shards to others" happens primarily while the
user is actively streaming content, you have a natural bell curve; during
peak streaming times, you have more nodes active to serve up shards,
handling the increased demand; at lower demand times, when fewer
people are active, and there's fewer home-nodes to serve shards, the
content network's shard servers can pick up the slack...but that'll
generally
only happen during lower traffic times, when the traffic won't be competing
and potentially causing pain for the ISP.

Really, it seems like a win-win scenario.

I'm confident we'll see a content network come out with a model like this
within the next 5 years, at which point the notion of blackmailing content
networks for additional $$$s will be a moot point, because the content will
be distributed and embedded within every major eyeball network already,
whether they like it or not, on their customer's devices.

Let's check back in 2026, and see if someone's become fantastically
successful doing this or not.  ;)

Thanks!

Matt

Re: DNS pulling BGP routes?

[Matthew Petach]

On Mon, Oct 11, 2021 at 8:07 AM Christopher Morrow 
wrote:

> On Sat, Oct 9, 2021 at 11:16 AM Masataka Ohta <
> mo...@necom830.hpcl.titech.ac.jp> wrote:
>
>> Bill Woodcock wrote:
>>
>
[...]

>
> it seems that the problem FB ran into was really that there wasn't either:
>"secondary path to communicate: "You are the last one standing, do not
> die"  (to an edge node)
>  or:
>   "maintain a very long/less-preferred path to a core location(s) to
> maintain service in case the CDN disappears"
>
> There are almost certainly more complexities which FB is not discussion in
> their design/deployment which
> affected their services last week, but it doesn't look like they were very
> far off on their deployment, if they
> need to maintain back-end connectivity to serve customers from the CDN
> locales.
>
> -chris
>

Having worked on trying to solve health-checking situations
in large production complexes in the past, I can definitely
say that is is an exponentially difficult problem for a single
site to determine whether it is "safe" for it to fail out, or if
doing so will result in an entire service going offline, short
of having a central controller which tracks every edge site's
health, and can determine "no, we're below $magic_threshold
number of sites, you can't fail yourself out no matter how
unhealthy you think you are".   Which of course you can't
really have, without undoing one of the key reasons for
distributing your serving sites to geographically distant
places in different buildings on different providers--namely
to eliminate single points of failure in your serving infrastructure.

Doing the equivalent of "no router bgp" on your core backbone
is going to make things suck, no matter how you slice it, and
I don't think any amount of tweaking the anycast setup or
DNS values would have made a whit of difference to the
underlying outage.

I think the only question we can armchair quarterback
at this point is whether there were prudent steps that
could go into a design to shorten the recovery interval.

So far, we seem to have collected a few key points:

1) make sure your disaster recovery plan doesn't depend
on your production DNS servers being usable; have
key nodes in /etc/hosts files that are periodically updated
via $automation_tool, but ONLY for non-production,
out-of-band recovery nodes; don't static any of your
production-facing entries.

2) Have a working out-of-band that exists entirely independent
of your production network.  Dial, frame relay, SMDS, LTE
modems, starlink dishes on the roof; pick your poison, but
budget it in for every production site.  Test it monthly to ensure
connectivity to all sites works.  Audit regularly to ensure no
dependencies on the production infrastructure have crept in.

3) Ensure you have a good "oh sh**" physical access plan for
key personnel.  Some of you at a recent virtual happy hour
heard me talk about the time I isolated the credit card payment
center for a $dayjob, which also cut off access for the card readers
to get into it to restore the network.   Use of a fire axe was granted
to on-site personnel during that.  Take the time to think through
how physical access is controlled for every key site in your network,
think about failure scenarios, and have a "in case of emergency,
break glass to get the key" plan in place to shorten recovery times.

4) Have a dependency map/graph of your production network.
 a) if everything dies, and you have to restart, what has to come up first?
 b) what dependencies are there that have to be done in the right order
 c) what services are independent that can be brought up in parallel to
speed
   up recovery?
d) does every team supporting services on the critical, dependent pathway
  have 24x7 on-call coverage, and do they know where in the recovery graph
  they're needed?  It doesn't help to have teams that can't start back up
until
  step 9 crowding around asking "are you ready for us yet?" when you still
can't
  raise the team needed for step 1 on the dependency graph.  ^_^;

5) do you know how close the nearest personnel are to each POP/CDN node,
   in case you have to do emergency "drive over with a laptop, hop on the
console,
   and issue the following commands" rousting in the middle of the night?
If someone
   lives.3 miles from the CDN node, it's good to know that, so you don't
call the person
   who is on-call but is 2 hours away without first checking if the person
3 miles away
   can do it faster.

I'm sure others have even better experiences than I, who can contribute
and add to the list.  If nothing else, perhaps collectively we can help
other companies prepare a bit better, so that when the next big "ooops"
happens, the recovery time can be a little bit shorter.   :)

Thanks!

Matt

Re: DNS pulling BGP routes?

[Matthew Petach]

On Sat, Oct 9, 2021 at 1:40 AM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Christopher Morrow wrote:
> >> means their DNS servers were serving the zone, even after they
> >> recognize their zone data were too old, that is, expired.
>
> > that's not what this means. I think Mr. Petach previously described
> > this,
>
> He wrote:
>
> > So, the idea is that if the edge CDN node loses connectivity to
> > the core datacenters, the DNS servers should stop answering
> > queries for A records with the local CDN node's address, and
> > let a different site respond back to the client's DNS request.
>
> which may be performed by standard DNS with short expire period,
> after which name servers will return SERVFAIL and other name
> servers in other edge node with different IP addresses are tried.
>

(Apologies for the delayed response--I had back-to-back board
meetings the past two days which had me completely tied up.)

That is one way in which it *could* be done--but is by no means
the ONLY way in which it can be done.

With an anycast setup using the same IP addresses in every
location, returning SERVFAIL doesn't have the same effect,
however, because failing over from anycast address 1 to
anycast address 2 is likely to be routed to the same pop
location, where the same result will occur.

You don't really want to hunt among different *IP addresses*,
you want to hunt to a different *location*.

This is why withdrawing the BGP announcement from that
location works more effectively, because it allows the clients
to continue querying the same IP address, but get routed to
the next most proximal location.

If you simply return SERVFAIL and have the client pick a
different IP address from the list of NS entries, it falls into
one of two situations:
a) the new IP address is also anycasted, and is therefore
 likely to pick the same pop that is unhealthy, with similar
 results, or
b) the new IP address is *not* anycasted, but is served from
a single geographical location, which means answers given
back by that DNS server are unlikely to be geolocated with
any accuracy, and therefore the content served is also unlikely
to be geographically relevant or correct.


>
> It may be that facebook uses all the four name server IP addresses
> in each edge node. But, it effectively kills essential redundancy
> of DNS to have two or more name servers (at separate locations)
> and the natural consequence is, as you can see, mass disaster.
>

Even if the four anycasted nameserver IP addresses weren't
completely overlapping (let's assume as a hypothetical that
a.ns is served out of EU pops, b.ns is served out of NA pops,
c.ns is served out of SA pops, and d.ns is served out of APAC
pops), if all sites run the same healthcheck code, then if the
underlying healthcheck fails, *every site* will decide it is
unhealthy, and stop answering requests; so, all the EU sites
fail health check and stop serving a.ns; all the North America
sites fail health check, and stop serving b.ns...and so forth.

You followed the best practices, you had different NS entries
that were on different subnets, that were geographically
dispersed around the globe, that were redundant for each
other.  But because they all used the same fundamental
health check, they all *independently* decided they were
unhealthy and needed to stop giving out DNS answers,
and instead let one of the other healthier sites take over.


>
> > but: 1) dns server in pop serves some content (ttls aren't
> > important right now)
>
> You MUST distinguish TTL and EXPIRE. They are different.
>

TTL and EXPIRE are irrelevant here.
The only thing changing those values would do is change
how long it took for caching resolvers to reflect the loss of
connectivity at the DNS layer.  Once the underlying layer 3
connectivity had broken, DNS answers became meaningless.
No matter what records were returned, or cached, you couldn't
reach the servers.

Yes, yes, as an academic exercise you can point out that
there's a difference in how and when those DNS records
stop being used, and you're right about that--but in terms
of this particular failure, this particular post-mortem we're
beating to a horse-shaped pulp, it's entirely meaningless.   ^_^;


>
>  > there's not a lot of magic here... and it's not about the zone data
>  > really at all.
>
> Statement of Petach: "the edge CDN node loses connectivity to
> the core datacenters, the DNS servers should stop answering"
> means, with DNS terminology, zone data is expired, which has
> nothing to do with TTL.
>

As you're using my words, I'm going to have to point out that
"the DNS servers should stop answering" does not require that
any change happens *at the DNS layer* -- in this case, the
change can happen at the routing layer, ensuring that even
if some caching resolver out there is completely defiant of
your expire time, you *will not answer* because the query
packets can never reach you in the first place.


>  

Re: S.Korea broadband firm sues Netflix after traffic surge

[Michael Thomas]

On 10/11/21 12:49 AM, Matthew Petach wrote:


Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.


As a consumer paying for my 4k stream, I know who I'm calling when it 
drops to 480 and it ain't Netflix. The eyeballs are most definitely not 
happy.


Mike

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> Going back to the fact that it's not the content providers "using" 
> a lot of bandwidth, it's the eyeball customer *requesting* a lot 
> of bandwidth, I think the best approach is for the content providers 
> to help manage traffic levels by lowering bit rates towards eyeball 
> networks that are feeling strained by their users.

This is the model that has pissed me off for decades… Somehow the
cellular carrier networks have been able to force phone application
and content providers to do things like limit the maximum size of file
that can be downloaded over the cellular network and force you to
download certain content over wifi.

Since I maintain a backup handset anyway and it is rarely utilized,
I let it accumulate rollover data until I want to do a large download.
Then I turn on its hotspot and the other phone uses that wifi to
download the large file I wasn’t allowed to download over the cellular
network due to exactly this stupid kind of limitation.

> Instead of a 4K stream, drop it to 480 or 240; the eyeball network 
> should be happy at the reduced strain the resulting stream puts 
> on their network.  

So your solution is to make the content provider punish the eyeball
user and deliver a poor user experience in order to let the crappy
eyeball network off the hook? I don’t think that’s a good solution.
It makes the content provider look bad to the end user and it
shifts the burden from the ISP that got paid to deliver content they
are failing to deliver onto the content provider that is trying to live
up to their agreement with their user.

IIRC, Netflix charges extra for a 4K level subscription these days,
so an end user that paid for 4K service and got 240p because their
ISP managed to force Netflix into a lower bitrate would likely be
pretty annoyed at both Netflix and the ISP if they understood the
situation.

> The content network can even point out they're being a good 
> Network Citizen by putting up a brief banner at the top of the 
> stream saying "reducing bit rate to relieve stress on your ISPs
> network".  That way, the happy customer knows that the 
> content provider is doing their part to help their ISP stay 
> profitable...I mean, doing their part to help the Internet 
> run better. 

Yeah, I’d be calling $CONTENT_PROVIDER and asking what I need
to do to get the full rate service I paid for. I’d also be calling $ISP
(or switching $ISP if possible) to one that didn’t have those issues
.
>  I'm pretty sure this is going to start happening more and more, 
> as ISPs realize that putting content caches into their IP space 
> to serve not only their own customers, but also customers of 
> selected peers can be a source of good leverage in the market. 

Agreed… Caching close to the edge makes complete sense,
especially with aggregated caching models through CDNs.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 00:32 , Mark Tinka  wrote:
> 
> 
> 
> On 10/11/21 02:58, Owen DeLong wrote:
> 
>> That’s irrelevant to what he is saying.
>> 
>> What he’s saying (and he’s 100% correct) is that any tax a corporation pays 
>> is collected from their customers one way or another.
>> 
>> A corporation has no other source of income with which to pay its taxes 
>> beyond those revenues collected from customers.
>> 
>> Of course I should probably expect this from someone who thinks IPv4 
>> shortages can be avoided by rationing IPv4 addresses.
> 
> There you go again, getting overly excited trying to divert the topic to your 
> keen area of interest - IPv4 in Africa.

My keen area of interest is IPv6 deployment globally, actually.

> I'll spell it out here so you are clear and have zero doubt:
> 
> I do not respect you - for what you represent on my continent, amongst other 
> things. So ignore me, because I am ignoring you. But if you don't ignore me, 
> that's your problem too, not mine.

Oh, you’ve made that quite clear and it’s mutual.

This word ignore, I do not think it means what you seem to think it means.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> I am almost sure Netflix have some degree of presence in South Korea. What 
> I'm not sure about is what else SK wants them to do beyond that.

They’ve made it pretty clear… They want Netflix to pay their 
protection^wbandwidth charges.

>> And for the record, not only have I never worked for an ISP, I was saying 
>> all the way back in the late '90s that the oversubscription business model 
>> (which almost always includes punishing users who actually use their 
>> bandwidth) is inherently unfair to the customers, and when the Internet 
>> becomes more pervasive in daily life will come back to bite them in the ass. 
>> I was laughed at for being hopelessly naive, not understanding how the 
>> bandwidth business works, etc.
> 
> Totally agreed. However, we are sort of forced to use this model because of 
> the underlying technology. Whenever a finite resource has to be shared 
> amongst several people, there has to be some way to manage that sharing.
> 
> Maybe if Internet services were circuit-switched, we wouldn't have this 
> problem. But then again, we wouldn't have an Internet like we do today either.

The oversubscription model is perfectly valid if rational numbers are chosen 
for oversubscription ratios and providers expect a reasonable number of 
customers to actually use what they paid for.

Many businesses outside of the internet depend on oversubscription to keep 
prices affordable while still making a profit.

Imagine if everyone actually used their gym memberships, for example.

Consider the standard practice of airline overbooking.

Hotels also often overbook.

Imagine if everyone who bought a season pass to an amusement park showed up 
every day they were open.

Now 50:1 oversubscription is probably insane and I agree that when you have a 
problem because your oversubscribed customers have difficulty when a few of them
use what you sold them, it’s not the customers’ fault and you need to reduce 
your oversubscription ratio to accommodate. That’s the business you’re in and 
if you
didn’t factor that into your pricing, you have a poor pricing structure.

For all I love to criticize Comcast for the many many things they do wrong, 
they have a reasonable model where you can buy your way out of bandwidth caps
for $30/month (at least in my case) and they don’t punish me when I use my full 
connection (or close to it).

YMMV.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Owen DeLong via NANOG]

> On Oct 11, 2021, at 00:01 , Mark Tinka  wrote:
> 
> 
> 
> On 10/11/21 00:31, Geoff Huston wrote:
> 
>> In many environments, the words we use to describe this form of price 
>> setting are generally prefixed by the adjective “illegal” :-)
> 
> Indeed - colluding is generally frowned upon, in which case we are doomed to 
> the current model, and may the best man win.
> 
> Ultimately, many ISP's and telco's will die. Consolidation will occur, but 
> the "big operator" will no longer be as fat as they used to be. Focus on 
> hauling bits around with no frills will be a good model, especially if you 
> can keep the team lean. The chances of having large monopolies that do okay 
> but stifle the market, being chased by struggling ISP's that favour passion + 
> frills, is what is likely to happen, over the next decade or two.
> 
> Mark.

Ideally, a regulatory framework which prohibits vertical integration and thus 
prevents the natural monopoly of last mile physical
infrastructure from being leveraged into a monopoly on higher-layer services 
would significantly improve the current situation,
making room for a strong competing market with low barrier to entry for 
services while the last mile infrastructure was managed
by a regulated utility or the run by the local municipality.

Owen

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 17:26, Niels Bakker wrote:




I don't think that's being entirely fair. Netflix in plenty places 
differentiates its subscriptions based partly on video resolution: 
https://help.netflix.com/en/node/24926/us


Some people will definitely care enough to sign up for a more 
expensive tier.


In much the same way those that care will opt for the D+ mode of the 
car, and likely more will be happy with just the D model.


Yes, many subscribers are clued up on the different Netflix plans, but 
I'm sure most of them choose the plans not for the video resolution, but 
for the number of active screens that can stream concurrently.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Niels Bakker]

* mark@tinka.africa (Mark Tinka) [Mon 11 Oct 2021, 17:18 CEST]:

To be fair, Jane + Thatho don't care about video resolution.


I don't think that's being entirely fair. Netflix in plenty places 
differentiates its subscriptions based partly on video resolution: 
https://help.netflix.com/en/node/24926/us


Some people will definitely care enough to sign up for a more 
expensive tier.



-- Niels.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Seun Ojedeji]

On Sun, Oct 10, 2021 at 8:11 PM Doug Barton  wrote:

> On 10/1/21 7:45 AM, Mark Tinka wrote:
>
>
> The reason that Netflix doesn't want to do it is the same reason that
> ISPs don't want to charge their customers what it really costs to
> provide them access.
>

SO: In my part of the world where end user internet services are largely
based on data caps, this should be good news to the ISP as the faster users
burn the data on streaming, the sooner they renew their subscription. What
is an ISP without content by the way? would one blame the likes of facebook
for trying to run their own pipe thereby further contributing to internet
defragmentation.

Regard
-- 



*Seun Ojedeji,Mobile: +2348035233535*

Bringing another down does not take you up - think about your action!

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 09:49, Matthew Petach wrote:


Going back to the fact that it's not the content providers "using"
a lot of bandwidth, it's the eyeball customer *requesting* a lot
of bandwidth, I think the best approach is for the content providers
to help manage traffic levels by lowering bit rates towards eyeball
networks that are feeling strained by their users.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.

The content network can even point out they're being a good
Network Citizen by putting up a brief banner at the top of the
stream saying "reducing bit rate to relieve stress on your ISPs
network".  That way, the happy customer knows that the
content provider is doing their part to help their ISP stay
profitable...I mean, doing their part to help the Internet
run better.


To be fair, Jane + Thatho don't care about video resolution. All they 
will see is that the picture isn't that great, and this creates an 
opportunity for a VoD provider who is "more favorable" toward the 
network operator, and gets granted full 4K resolution transport 
priviledges. If there are any surcharges levied by the network operator, 
the VoD provider compensates for those by attracting more business 
because, well, they can offer a more superior image quality.


I'm not sure about the term, but the "colluding" version for that would 
be - if my few IGF days do not fail me - "net neutrality".


I'm not sure we want to go down that path, either.



The market *is* determining that at the moment...but not in the
direction people expect.  Instead, it's creating a new market for
intermediaries; imagine you're an eyeball network that happens
to have peering with SKB, and largely inbound traffic flows.
Wouldn't it make sense for you to reach out to a player like
Netflix, and offer to host content cache boxes that happen to
only answer requests coming from SKB IP space, at a price
well below what SKB was going to charge the content provider?
As the eyeball network, you'd see your traffic ratios
balance out as the cache traffic filled your under-utilized outbound
port capacity, and you'd get a bit of additional revenue you otherwise
wouldn't get.  As the content provider, you're serving your customers
for a lower price than SKB wants to charge, and without giving into
SKB's extortion tactics.  It's a win-win-lose situation, in which the
content provider wins, the eyeball network that has a peering
relationship with SKB wins, and the only loser is SKB, which
doesn't get the additional revenue it was looking for, and actually
helps funnel money to a competitor that they otherwise wouldn't
have gotten.

I'm pretty sure this is going to start happening more and more,
as ISPs realize that putting content caches into their IP space
to serve not only their own customers, but also customers of
selected peers can be a source of good leverage in the market.


In reality, which small mom & pop will have peering with BigTelco :-)?

Suffice it to say, Netflix would also need to reconsider whether they 
can afford to give OCA's to mom & pop.


Mark.

Re: DNS pulling BGP routes?

[Christopher Morrow]

On Sat, Oct 9, 2021 at 11:16 AM Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Bill Woodcock wrote:
>
> >> It may be that facebook uses all the four name server IP addresses
> >> in each edge node. But, it effectively kills essential redundancy
> >> of DNS to have two or more name servers (at separate locations)
> >> and the natural consequence is, as you can see, mass disaster.
> >
> > Yep.  I think we even had a NANOG talk on exactly that specific topic a
> long time ago.
> >
> >
> https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf
>
> Yes, having separate sets of anycast addresses by two or more pops
> should be fine.
>
>
To be fair, it looks like FB has 4 /32's (and 4 /128's) for their DNS
authoritatives.
All from different /24's or /48's, so they should have decent routing
diversity.
They could choose to announce half/half from alternate pops, or other games
such as this.
I don't know that that would have solved any of the problems last week nor
any problems in the future.
I think Bill's slide 30 is pretty much what FB has/had deployed:
  1) I would think the a/b cloud is really 'as similar a set of paths from
like deployments as possible
  2) redundant pairs of servers in the same transit/network
  3) hidden masters (almost certainly these are in the depths of the FB
datacenter network)
  (though also this part isn't important for the conversation)
  4) control/sync traffic on a different topology than the customer serving
one


> However, if CDN provider has their own transit backbone, which is,
> seemingly, not assumed by your slides, and retail ISPs are tightly
>

I think it is, actually, in slide 30 ?
   "We need a network topology to carry control and synchronization traffic
between the nodes"

connected to only one pop of the CDN provider, the CDN provider
>

it's also not clear that FB is connecting their CDN to single points in any
provider...
I'd guess there are some cases of that, but for larger networks I would
imagine there are multiple CDN
deployments per network. I can't imagine that it's safe to deploy 1 CDN
node for all of 7018 or 3320...
for instance.


> may be motivated to let users access only one pop killing essential
> redundancy of DNS, which should be overengineering, which is my
> concern of the paragraph quoted by you.
>
>
it seems that the problem FB ran into was really that there wasn't either:
   "secondary path to communicate: "You are the last one standing, do not
die"  (to an edge node)
 or:
  "maintain a very long/less-preferred path to a core location(s) to
maintain service in case the CDN disappears"

There are almost certainly more complexities which FB is not discussion in
their design/deployment which
affected their services last week, but it doesn't look like they were very
far off on their deployment, if they
need to maintain back-end connectivity to serve customers from the CDN
locales.

-chris

Re: DNS pulling BGP routes?

[Masataka Ohta]

Owen DeLong wrote:


But, it has nothing specifically to do with anycast. As there
are other name servers with different IP addresses, there is
no reason to withdraw routes. So?


WRONG.

First, assuming that there are non-anycast name servers assumes
facts not in evidence.


There is no such assumption.


Second, if you are a participant in an anycast name server network,
there are good reasons to withdraw your announcement of that
prefix


You completely misunderstand my points. See other posts of mine.

Masataka Ohta

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:10, Niels Bakker wrote:



Sounds like you think SK should be paying Netflix for bringing their 
content all the way from the US to the Korean peninsula. That's some 
expensive wet cable being used there.


Do we know whether Netflix don't already have OCA's and/or local peering 
in South Korea?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:57, Sabri Berisha wrote:


I have worked for ISPs. And I remember the late 90s. Bandwidth was $35/mbit
on average, at least for the outfit where I was. Consumers paid roughly $40
for their DSL connections, which at the time went up to 2Mbit depending
on the age of the copper and distance to the DSLAM. Consumer connections
were oversubscribed, on average, 1:35 to 1:50. B2B connections got a better
deal, 1:10 to 1:15.

It was simply not feasible to offer 1:1 bandwidth and still make a profit,
unless you're charging fees the average consumer cannot afford.

Especially considering that the average user doesn't even need or use that
much bandwidth. It's a recurring discussion. People demand more bandwidth
without considering whether or not they need it. End-users, business subs,
and host-owners at large enterprises where I worked. The last ones are the
funniest: entire racks using no more than 100mbit/s and hostowners are
demanding an upgrade from 10G to 25G bEcaUse LaTenCy.

The last consumer ISP I worked at had a very small subset of users that
really needed bandwidth: the "download dudes" who were 24/7 leeching news
servers, and the inevitable gamers that complained about the latency due
to the links being full as a result of said leechers. In that case, a
carefully implemented shaping of tcp/119 did the trick.


It's the conundrum of a shared resource. Like managing 1 lift (elevator, 
for the Americans :-)) that needs to move a building full of people 
between floors.


In the early days of the Internet, circuits were long, expensive and 
slow. Equipment cost a lot for what you could get out of it, and content 
was mostly centralized, making getting to it even more expensive, and 
hence, entrenching the current model.


However, in an era where content is making a push to get as close to the 
eyeballs as possible, kit getting cheaper and faster because of merchant 
silicon, and abundance of aggregated capacity at exchange points, can we 
leverage the shorter, faster links to change the model?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Matthew Petach]

On Sun, Oct 10, 2021 at 2:44 PM Doug Barton  wrote:

> [some snipping below]
>
> Also just to be clear, these are my own opinions, not necessarily shared
> by any current or former employers.
>
> On 10/10/21 12:31 PM, Mark Tinka wrote:
> > On 10/10/21 21:08, Doug Barton wrote
> >> Given that issue, I have some sympathy for eyeball networks wanting to
> >> charge content providers for the increased capacity that is needed to
> >> bring in their content. The cost would be passed on to the content
> >> provider's customers...
> >
> > But eyeballs are already paying you a monthly fee for 100Mbps of service
> > (for example). So they should pay a surcharge, over-and-above that, that
> > determines how they can use that 100Mbps? Seems overly odd, to me.
>
> Yes, I get that. But as you pointed out here and in other comments, the
> ISP market is based entirely on undercutting competitors (with a lot of
> gambling thrown in, as Matthew pointed out).
>
>  [...]

> > So what rat hole does this lead us down into? People who want to stream
> > Youtube should pay their ISP for that? People who want to spend
> > unmentionable hours on Linkedin should be their ISP for that? People who
> > want to gawk over Samsung's web site because they love it so much,
> > should pay their ISP for that?
>
> First, I'm not saying "should." I'm saying that given the market
> economics, having the content providers who use "a lot" of bandwidth do
> something to offset those costs to the ISPs might be the best/least bad
> option. Whether "something" is a local cache box, peering, money, or
>  is something I think that the market should determine.
>

Going back to the fact that it's not the content providers "using"
a lot of bandwidth, it's the eyeball customer *requesting* a lot
of bandwidth, I think the best approach is for the content providers
to help manage traffic levels by lowering bit rates towards eyeball
networks that are feeling strained by their users.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.

The content network can even point out they're being a good
Network Citizen by putting up a brief banner at the top of the
stream saying "reducing bit rate to relieve stress on your ISPs
network".  That way, the happy customer knows that the
content provider is doing their part to help their ISP stay
profitable...I mean, doing their part to help the Internet
run better.


> And to answer Matthew's question, I don't know what "a lot" is. I think
> the market should determine that as well.
>

The market *is* determining that at the moment...but not in the
direction people expect.  Instead, it's creating a new market for
intermediaries; imagine you're an eyeball network that happens
to have peering with SKB, and largely inbound traffic flows.
Wouldn't it make sense for you to reach out to a player like
Netflix, and offer to host content cache boxes that happen to
only answer requests coming from SKB IP space, at a price
well below what SKB was going to charge the content provider?
As the eyeball network, you'd see your traffic ratios
balance out as the cache traffic filled your under-utilized outbound
port capacity, and you'd get a bit of additional revenue you otherwise
wouldn't get.  As the content provider, you're serving your customers
for a lower price than SKB wants to charge, and without giving into
SKB's extortion tactics.  It's a win-win-lose situation, in which the
content provider wins, the eyeball network that has a peering
relationship with SKB wins, and the only loser is SKB, which
doesn't get the additional revenue it was looking for, and actually
helps funnel money to a competitor that they otherwise wouldn't
have gotten.

I'm pretty sure this is going to start happening more and more,
as ISPs realize that putting content caches into their IP space
to serve not only their own customers, but also customers of
selected peers can be a source of good leverage in the market.

Matt

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 02:58, Owen DeLong wrote:


That’s irrelevant to what he is saying.

What he’s saying (and he’s 100% correct) is that any tax a corporation pays is 
collected from their customers one way or another.

A corporation has no other source of income with which to pay its taxes beyond 
those revenues collected from customers.

Of course I should probably expect this from someone who thinks IPv4 shortages 
can be avoided by rationing IPv4 addresses.


There you go again, getting overly excited trying to divert the topic to 
your keen area of interest - IPv4 in Africa.


I'll spell it out here so you are clear and have zero doubt:

I do not respect you - for what you represent on my continent, amongst 
other things. So ignore me, because I am ignoring you. But if you don't 
ignore me, that's your problem too, not mine.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 03:05, Owen DeLong wrote:


Which is the kind of ignorant view of the situation that creates this problem 
in the first place.

It’s not for “no $$”, it’s for all the $$ they got from all those 100Mbps links 
that they are delivering those Tbps of traffic to.

If the aggregate $$ they are collecting is insufficient, then they have priced 
their service incorrectly and should either re-evaluate,
or go bankrupt and sell to someone that knows how to run a business.


Mate, keep your pants on... don't get yourself worked up into excitement 
at my folly.


And in case you haven't noticed, I am ignoring you.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:42, Doug Barton wrote:



I didn't say income tax. Corporate taxes are considered an expense by 
the corporation paying them. Like all other expenses, they are 
factored into the cost of goods/services sold.


Ah, yes, agreed. I thought you meant something else...




First, I'm not saying "should." I'm saying that given the market 
economics, having the content providers who use "a lot" of bandwidth 
do something to offset those costs to the ISPs might be the best/least 
bad option. Whether "something" is a local cache box, peering, money, 
or  is something I think that the market should determine.


But all the major content providers do this already. They come to 
exchange points. They provide caches. What more do we want them to do?


Content providers that don't do these things don't generally tend to be 
popular, in which case, we don't have to worry about them flooding 
backbone links.


I am almost sure Netflix have some degree of presence in South Korea. 
What I'm not sure about is what else SK wants them to do beyond that.





And to answer Matthew's question, I don't know what "a lot" is. I 
think the market should determine that as well.


Hehe, free markets determine the fundamental principles through price 
and competition, which is why we are in this mess to begin with. Unless 
you want "government" to make the determination :-).





And for the record, not only have I never worked for an ISP, I was 
saying all the way back in the late '90s that the oversubscription 
business model (which almost always includes punishing users who 
actually use their bandwidth) is inherently unfair to the customers, 
and when the Internet becomes more pervasive in daily life will come 
back to bite them in the ass. I was laughed at for being hopelessly 
naive, not understanding how the bandwidth business works, etc.


Totally agreed. However, we are sort of forced to use this model because 
of the underlying technology. Whenever a finite resource has to be 
shared amongst several people, there has to be some way to manage that 
sharing.


Maybe if Internet services were circuit-switched, we wouldn't have this 
problem. But then again, we wouldn't have an Internet like we do today 
either.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 01:43, Keith Medcalf wrote:


This is blatantly incorrect.  The bits were payed for by the requestor.


You totally missed my dig...

I was being sarcastic.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:31, Geoff Huston wrote:


In many environments, the words we use to describe this form of price setting 
are generally prefixed by the adjective “illegal” :-)


Indeed - colluding is generally frowned upon, in which case we are 
doomed to the current model, and may the best man win.


Ultimately, many ISP's and telco's will die. Consolidation will occur, 
but the "big operator" will no longer be as fat as they used to be. 
Focus on hauling bits around with no frills will be a good model, 
especially if you can keep the team lean. The chances of having large 
monopolies that do okay but stifle the market, being chased by 
struggling ISP's that favour passion + frills, is what is likely to 
happen, over the next decade or two.


Mark.