Re: wow, lots of akamai

[Charles Polisher]

On 4/5/21 10:23 PM, Robert Brockway wrote:

On Thu, 1 Apr 2021, Jean St-Laurent via NANOG wrote:

What happened is that it would create a kind of internal DDoS and 
they would all timed out and give a weird error message. Something 
very useful like Error Code 0x8098808 Please call our support line at 
this phone number.


If only there was a way to address the Thundering Herd problem before 
the cloud. :)


This simple change to add 3 lines of code to add a random artificial 
boot penalty of few seconds, completely solve the problem.


Bingo.  Now, the trick is to catch this before it causes an self-DDoS.

This is a problem that has been recognised for decades and this is 
unfortunately a good example of how operational experience is still 
not being distributed properly.  Too many managers think that 
operational work is obvious and just a result of common sense.  It isn't.


Same problem as disk drives powering up simultaneously
in datacenters. SCSI drives have (had?) a random delay
mechanism to distribute the initial power surge over a few
seconds.

RE: wow, lots of akamai

[Robert Brockway]

On Thu, 1 Apr 2021, Jean St-Laurent via NANOG wrote:

What happened is that it would create a kind of internal DDoS and they would 
all timed out and give a weird error message. Something very useful like 
Error Code 0x8098808 Please call our support line at this phone number.


If only there was a way to address the Thundering Herd problem before the 
cloud. :)


This simple change to add 3 lines of code to add a random artificial boot 
penalty of few seconds, completely solve the problem.


Bingo.  Now, the trick is to catch this before it causes an self-DDoS.

This is a problem that has been recognised for decades and this is 
unfortunately a good example of how operational experience is still not being 
distributed properly.  Too many managers think that operational work is obvious 
and just a result of common sense.  It isn't.


Cheers,

Rob

RE: wow, lots of akamai

[aaron1]

Yes, I was reaching out to my NANOG folks to find out as you stated... "Hey I 
was curious what happened and I thought to ask here on NANOG?"

I appreciate the membership with you all and value your position and visibility 
in regional, continental and global operations.  Thanks for your insights, and 
I hope I contribute occasionally as well.

-Aaron

Re: wow, lots of akamai

[Mike Hammett]

As an IX, we've noticed that a lot of networks don't avail themselves of all 
opportunities to connect to sources of content. That lack of diversity can 
cause issues when there are failures, congestion, etc. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jared Mauch"  
To: "Dave Brockman - DVS"  
Cc: nanog@nanog.org 
Sent: Friday, April 2, 2021 7:08:09 AM 
Subject: Re: wow, lots of akamai 

On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote: 
> On 4/1/2021 3:21 PM, Niels Bakker wrote: 
> > * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 
> > CEST]: 
> >> An artificial roll out penalty somehow? Probably not at the ISP level, 
> >> but more at the game level. Well, ISP could also have some mechanisms 
> >> to reduce the impact or even Akamai could force a progressive roll out. 
> > 
> > It's an online game. You can't play the game with outdated assets. You'd 
> > not see walls where other players would, for example. 
> > 
> > What you're suggesting is the ability of ISPs to market Internet access 
> > at a certain speed but not have to deliver it based on conditions they 
> > create. 
> > 
> > 
> > -- Niels. 
> 
> 
> There was at least one online gaming platform that used to stage their 
> game updates, where clients would download the update over the previous 
> week or so, and then on go-live day, the game updated itself. There 
> were a few that had to download on go-live day, but the clients I ran, 
> and friends ran, had downloaded prior. I don't know if it is still that 
> way, but it seems like reasonable solution to me. 

Actually in the past year many of the gaming studios have 
started to do this for the release process. It will start during an 
off-peak hour and then flow through the day. The end-user induced 
demand and stress it places on networks doesn't fit into the historical 
95/5 30-day peak planning model. I know some carriers are struggling to 
adjust. There's a few of us here on-list, please reach out so we can 
help. 

- Jared 

-- 
Jared Mauch | pgp key available via finger from ja...@puck.nether.net 
clue++; | http://puck.nether.net/~jared/ My statements are only mine. 

Re: wow, lots of akamai

[Jared Mauch]

On Thu, Apr 01, 2021 at 08:09:24PM +, Luke Guillory wrote:
> IX’s don’t really help the source doesn’t use them.
> 
> Akamai traffic.
> 17G via Local Cache
> 17G via Transit
> 8G via IXs.
> 
> Plenty of room on IXs for more on our side.

Often we can see the ports at the IX flatline in these cases.
I've been working to improve some of them, for exampe at the DetroitIX
we have 300G of capacity and you can see the impact at the IX here:

https://www.detroitix.com/#stats

If you look at the monthly graph you can see other game download events.

It can be incredibly hard to right-size this stuff, please do reach out
to myself or Niels if things didn't work right for your network.  Also
you should reach out to your carriers who may have had congestion to
ensure they are upgrading their networks as well.  I've been told by
some teams they won't upgraded for a day a month of thsi big traffic,
but when one of their customers has issues they immediately escalate to
us to ask us to move (and we try hard to do that).

It's not all throw bandwidth at the problem, but with 100g optics being
so cheap these days, it may be the lower bar

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: wow, lots of akamai

[Jared Mauch]

On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote:
> On 4/1/2021 3:21 PM, Niels Bakker wrote:
> > * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> > CEST]:
> >> An artificial roll out penalty somehow? Probably not at the ISP level,
> >> but more at the game level. Well, ISP could also have some mechanisms
> >> to reduce the impact or even Akamai could force a progressive roll out.
> > 
> > It's an online game. You can't play the game with outdated assets. You'd
> > not see walls where other players would, for example.
> > 
> > What you're suggesting is the ability of ISPs to market Internet access
> > at a certain speed but not have to deliver it based on conditions they
> > create.
> > 
> > 
> > -- Niels.
> 
> 
> There was at least one online gaming platform that used to stage their
> game updates, where clients would download the update over the previous
> week or so, and then on go-live day, the game updated itself.  There
> were a few that had to download on go-live day, but the clients I ran,
> and friends ran, had downloaded prior.  I don't know if it is still that
> way, but it seems like reasonable solution to me.

Actually in the past year many of the gaming studios have
started to do this for the release process.  It will start during an
off-peak hour and then flow through the day.  The end-user induced
demand and stress it places on networks doesn't fit into the historical
95/5 30-day peak planning model.  I know some carriers are struggling to
adjust.  There's a few of us here on-list, please reach out so we can
help.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: wow, lots of akamai

[Dave Brockman - DVS]

On 4/1/2021 3:21 PM, Niels Bakker wrote:
> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> CEST]:
>> An artificial roll out penalty somehow? Probably not at the ISP level,
>> but more at the game level. Well, ISP could also have some mechanisms
>> to reduce the impact or even Akamai could force a progressive roll out.
> 
> It's an online game. You can't play the game with outdated assets. You'd
> not see walls where other players would, for example.
> 
> What you're suggesting is the ability of ISPs to market Internet access
> at a certain speed but not have to deliver it based on conditions they
> create.
> 
> 
> -- Niels.


There was at least one online gaming platform that used to stage their
game updates, where clients would download the update over the previous
week or so, and then on go-live day, the game updated itself.  There
were a few that had to download on go-live day, but the clients I ran,
and friends ran, had downloaded prior.  I don't know if it is still that
way, but it seems like reasonable solution to me.

Cheers,

-dtb

Re: wow, lots of akamai

[d...@darwincosta.com]

> On 2 Apr 2021, at 11:47, Mark Tinka  wrote:
> 
>  
> 
>> On 4/2/21 01:41, Tony Wicks wrote:
>> 
>> Local backhaul is plentiful and relatively cheap where as subsea wavelengths 
>> are extremely expensive and require months of planning.
> 
> Funny, it's the exact opposite for us.
Yup, it is... 

> 
> Mark.
Darwin-. 

Re: wow, lots of akamai

[Mark Tinka]

On 4/2/21 01:41, Tony Wicks wrote:

Local backhaul is plentiful and relatively cheap where as subsea 
wavelengths are extremely expensive and require months of planning.




Funny, it's the exact opposite for us.

Mark.

Re: wow, lots of akamai

[Mark Tinka]

On 4/2/21 00:56, Patrick W. Gilmore wrote:



And after all that, I still do not see what we are arguing about? You 
want the game companies to change their business model, but you do not 
want to change yours. Please do not say something like “but if they 
just ….” Unless you want the game companies to say “but if the ISPs 
just ….” Either way, stop trying to say someone else - the game 
provider, the CDN, the user, whoever - should change their model or 
spend their money to keep your business above water.


I know - don't sign up customers who smell like they have gaming 
consoles, gaming PC's, or gaming hand-held devices :-).


#ProblemSolved

Mark.

Re: wow, lots of akamai

[Mark Tinka]

On 4/1/21 21:01, Jean St-Laurent via NANOG wrote:


Are big games roll out really impacting NANOG? or it's more a: Hey I was 
curious what happened and I thought to ask here on NANOG?


The latter, I'd say.

Mark.

RE: wow, lots of akamai

[aaron1]

U, throw bandwidth at it.  ...which reminds me... I actually want a t-shirt 
that says   "Bandwidth solves a lot"

-aaron

-Original Message-
From: Jean St-Laurent  
Sent: Thursday, April 1, 2021 2:01 PM
To: aar...@gvtc.com; 'Jared Mauch' ; 'Töma Gavrichenkov' 

Cc: 'NANOG' 
Subject: RE: wow, lots of akamai

I remembered working for a big ISP in Europe offering cable tv + internet with 
+20M subscribers

Every time there was a huge power outage in major cities, all tv`s would go off 
at the same time. I don`t have stats on power grid stability in Europe Vs N/A.

The problem, was when the power was coming back in big cities, all the tv 
subscribers would come back online at the exact same second or minute.
More or less the same 2 or 3 minutes.

What happened is that it would create a kind of internal DDoS and they would 
all timed out and give a weird error message. Something very useful like Error 
Code 0x8098808 Please call our support line at this phone number.

The server sysadmins would go on a panic because all systems were overloaded. 
They often needed to do overtime because DB crashed, key servers there crashed, 
DB here crashed, whatever... there was always something crashing.  This was 
before the cloud when you could just push a slider and have tons of VMs or 
containers to absorb the load in real time. (in my dream)

This would every time create frustration from the clients, the help desk, the 
support teams and also the upper management. Every time the teams were really 
tired after that. It was draining juice.

Anyway, after some years of talking internally (red tape), we finally managed 
to install a random artificial penalty in the setup boxes when they boot after 
a power outage. Nothing like 20 minutes, but just enough to spread the load 
over a longer period of time. For the end user, it went transparent for them 
because, if the setup box would boot in 206 seconds instead of the super 
aggressive 34 seconds, well it booted and they could watch tv. 
Vs 

my system is totally frozen and it`s been like that for 20 minutes with weird 
messages because all your systems are down and the error msg said to call the 
help desk.

This simple change to add 3 lines of code to add a random artificial boot 
penalty of few seconds, completely solve the problem. This way, when a city 
would black out, we wouldn't be self DDoS, because the systems would slowly 
rampup. The setup boxes would all reboot but, wait randomly before asking for 
the DRM package to unlock the cable TV service and validate whether billing is 
right.

I`m no Call of Duty expert nor Akamai, but it's been many times that I observe 
the same question here:

What's happening?
Call of Duty!
Okay.

Would a kind of throttle help here? 

An artificial roll out penalty somehow? Probably not at the ISP level, but more 
at the game level. Well, ISP could also have some mechanisms to reduce the 
impact or even Akamai could force a progressive roll out. 

I`m not sure that the proposed solutions could work, but it seems to impact 
NANOG frequently and/or at least generate a call overnight/weekend. It seems to 
also happens just before long holidays when operations are sometimes on reduce 
personnel.

Are big games roll out really impacting NANOG? or it's more a: Hey I was 
curious what happened and I thought to ask here on NANOG?

#JustCurious

Jean

-Original Message-
From: NANOG  On Behalf Of 
aar...@gvtc.com
Sent: April 1, 2021 12:12 PM
To: 'Jared Mauch' ; 'Töma Gavrichenkov' 

Cc: 'NANOG' 
Subject: RE: wow, lots of akamai

Gaming update... I had a feeling.  Thanks for the feedback folks.

Thanks Jared, it's running well, before, during and after.  We have a lot of 
capacity there.

-Aaron

RE: wow, lots of akamai

[Tony Wicks]

No absolutely not, having the traffic coming from local CDN’s and the shorter 
but higher traffic is very much preferred. My comment was just to point out 
that yes there is a significant difference on ISP traffic between delivery via 
CDN/PNI/Peering than transit as in our case transit is a long way away. Local 
backhaul is plentiful and relatively cheap where as subsea wavelengths are 
extremely expensive and require months of planning. I’m not assuming transport 
just going on real world traffic affects.

 

From: NANOG  On Behalf Of Patrick W. 
Gilmore
Sent: Friday, 2 April 2021 11:32 am
To: North American Operators' Group 
Subject: Re: wow, lots of akamai

 

Just so I am clear, you are saying “I would rather have it come over my 
undersea cables than from inside the datacenter”?

 

And you are assuming TCP transport.

 

Re: wow, lots of akamai

[Niels Bakker]

* patr...@ianai.net (Patrick W. Gilmore) [Fri 02 Apr 2021, 01:01 CEST]:
I know first hand that Akamai has explained to large customers the 
possible problems with multi-GB updates to millions of users 
simultaneously. If the game company does not care, then I do not see 
what you expect the CDN to do about it.


Thanks, Patrick. In fact, Akamai has been a partner in this dialogue: 
https://blogs.akamai.com/2020/03/working-together-to-manage-global-internet-traffic-increases.html

One of the positive results is described here: 
https://www.callofduty.com/blog/2021/03/Season-Two-Reloaded-Warzone-File-Size-Reduction
"Enhancements to the overall content management system has been made 
possible through data optimization and streamlining content packs 
needed for individual game modes. This will come after a larger than 
usual, one-time update for Season Two Reloaded, which will include 
these optimizations and is necessary in order to reduce the overall 
footprint; future patch sizes for Modern Warfare and Warzone are 
expected to be smaller than the one set to release on March 30 at 11PM 
PST."


So the ISPs as well as the CDNs and the players are being listened to 
by at least this publisher.



-- Niels.

Re: wow, lots of akamai

[Patrick W. Gilmore]

I am a bit worried about phrases like "If Akamai was doing these updates more 
frequently”. Akamai does not decide these things. You may as well say “if the 
fiber carriers sent the bits over several hours instead of all at once.” And 
please do not say you were just using shorthand. You have blamed the CDNs and 
Akamai by name several times in this thread.

I know first hand that Akamai has explained to large customers the possible 
problems with multi-GB updates to millions of users simultaneously. If the game 
company does not care, then I do not see what you expect the CDN to do about it.

Most CDNs do their best to deliver traffic optimally. It is in their own best 
interest. They want to avoid dropped packets even more than you do. If you do 
not like the way a CDN will deliver the traffic, talk to them. Perhaps there is 
a compromise, perhaps not. But most of them will at least kick ideas around to 
see what can be done.


And after all that, I still do not see what we are arguing about? You want the 
game companies to change their business model, but you do not want to change 
yours. Please do not say something like “but if they just ….” Unless you want 
the game companies to say “but if the ISPs just ….” Either way, stop trying to 
say someone else - the game provider, the CDN, the user, whoever - should 
change their model or spend their money to keep your business above water.

-- 
TTFN,
patrick

P.S. It is not 1995. “The Internet” is a bit more mature, and users expect a 
bit more.


> On Apr 1, 2021, at 6:27 PM, Matt Erculiani  wrote:
> 
> Patrick,
> 
> > Matt: Are you arguing the CDNs are at fault because the game companies tell 
> > everyone to download simultaneously, and
> > the ISPs sold the users connectivity to do that download?
> 
> While a gross oversimplification, yes, that's basically what I'm saying; I 
> know it may not be a popular opinion, but I stand by it. There aren't any 
> villains here though, just lots of good suggestions in this thread to make 
> the internet work better for everyone, without spending large swaths of money 
> to cover the demand of an infrequent, large, update for a single game.
> 
> CNDs do, however, have a responsibility to be good netizens and get this data 
> out in a manner that doesn't cause disruption. They know the technical 
> challenges of distributing that much data to the masses, the game company 
> does not, that's why they outsourced it to a CDN. If the CDN knows what the 
> gaming company is asking for is pushing the limits of our current 
> infrastructure, they have a responsibility to relay those limitations that 
> are outside of their control to their customer, as any responsible vendor 
> should. Instead, there may be an element of "oh yeah sure, we can do that" or 
> "the customer is always right" going on here and modern limitations are being 
> disregarded.
> 
> The idea behind the internet is not that every user can always have their 
> entire capacity available for a single destination regardless of what 
> everyone else is doing (and especially if they're all going to the same place 
> too), the user has purchased that capacity into their provider's network as a 
> whole, gaining access to all of their connections to all of the various 
> endpoints on the internet at a backbone and peering capacity that is 
> economically viable given normal peak demand with some cushion built-in for 
> redundancy. If that's the desire to have full capacity available to Akamai 
> available at all times, then everyone needs dedicated P2P circuits direct to 
> Akamai, but that's not practical.
> 
> If you own an ISP and you're not oversubscribing, you're not making money, 
> period. To use your analogy, if you've ever been to a gym in January, you've 
> seen a similar phenomenon first-hand. There aren't enough machines for 
> everyone, and the gym isn't going to add them because this is a once-a-year 
> thing and it goes away after a few weeks when many people get tired of 
> fulfilling their new year's resolutions. Why should the gym limit its sales 
> to exactly the capacity it has available (or add a lot more machines), when 
> it knows that for the overwhelming majority of the year, there will always be 
> dozens of empty machines across the floor?
> 
> If Akamai was doing these updates more frequently (weekly for example) then 
> sure, it is on the ISP to augment, because this has become the "new normal". 
> But these updates for a single game that happen once per quarter are hardly 
> able to be considered normal. Sure, some day 50GB updates will be the norm, 
> but that's not today, and when it is, somebody else will be pushing out 250GB 
> updates quarterly. This problem isn't going away soon, and it can't be fixed 
> permanently by just adding more capacity, it's a complex technical challenge 
> that CDN's ought to give some more thought, and game publishers should start 
> considering too.
> 
> -Matt
> 
> On Thu, Apr 1, 2021 at 3:30 PM 

Re: wow, lots of akamai

[Mike Hammett]

Maybe? 6 months? 12 months? Okay, maybe I'll buy it. 

36 hours? No. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Niels Bakker"  
To: nanog@nanog.org 
Sent: Thursday, April 1, 2021 5:40:41 PM 
Subject: Re: wow, lots of akamai 

* na...@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 23:17 CEST]: 
>However, the game publisher queues those requests. I'm meaning 
>request generically, not a GET request or anything like that. The 
>game publisher that contracts to the CDNs decides when to fulfill 
>those requests, in the big picture. The game publisher is the one 
>that then tells 100 million devices "Content Available". The rate 
>that they do that is at their discretion. 

Keep in mind that the publisher doesn't just create a bunch of new 
assets to give away for free from the goodness of their hearts. Every 
update comes with new hats, weapon finishes, heroes, trading cards 
etc. that players are expected to buy for real money. Any delay could 
kill the hype and impact revenue. 

There is a lot of "But muh discard counters!" in this thread that is 
perfectly valid and understandable and that I absolutely relate to, 
but it's not the only perspective. 


-- Niels. 

Re: wow, lots of akamai

[Niels Bakker]

* na...@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 23:17 CEST]:
However, the game publisher queues those requests. I'm meaning 
request generically, not a GET request or anything like that. The 
game publisher that contracts to the CDNs decides when to fulfill 
those requests, in the big picture. The game publisher is the one 
that then tells 100 million devices "Content Available". The rate 
that they do that is at their discretion.


Keep in mind that the publisher doesn't just create a bunch of new 
assets to give away for free from the goodness of their hearts. Every 
update comes with new hats, weapon finishes, heroes, trading cards 
etc. that players are expected to buy for real money. Any delay could 
kill the hype and impact revenue.


There is a lot of "But muh discard counters!" in this thread that is 
perfectly valid and understandable and that I absolutely relate to, 
but it's not the only perspective.



-- Niels.

Re: wow, lots of akamai

[Patrick W. Gilmore]

Just so I am clear, you are saying “I would rather have it come over my 
undersea cables than from inside the datacenter”?

And you are assuming TCP transport.

-- 
TTFN,
patrick

> On Apr 1, 2021, at 6:23 PM, Tony Wicks  wrote:
> 
> This is not actually (as in yes it does matter) the case, if a file comes 
> from a CDN it is often a close and low latency source that will run up to 
> very high speeds. For example in our case we connect to local peering 
> exchanges (or PNI’s/local caches) at 100G or Nx10G with latency to the end 
> user in the 1-30ms range resulting in very large peaks of local backhaul 
> traffic. If a file is delivers from source or from remote CDN’s/exchanges 
> these are located in other countries with between 25ms (New Zealand to 
> Australia) and 130-200ms (New Zealand to LA/SJC or Singapore) latency, this 
> results in a much slower and normally barely noticeable traffic blip. Yes as 
> an ISP we need to carry the traffic in both cases but the first case can 
> result in a 20-30% local backhaul increase for a couple of hours and in the 
> second case its just BAU traffic for a day or two. Local CDN is obviously the 
> better option for cost and the consumer, but you certainly do notice the 
> traffic in local backhaul.
>  
> From: NANOG  <mailto:nanog-bounces+tony=wicks.co...@nanog.org>> On Behalf Of Tom Beecher
> Sent: Friday, 2 April 2021 10:05 am
> To: Matt Erculiani mailto:merculi...@gmail.com>>
> Cc: North American Operators' Group mailto:nanog@nanog.org>>
> Subject: Re: wow, lots of akamai
>  
>  
> If thousands of users are downloading 50G files at the same time, it really 
> doesn't matter if they are pulling from a CDN or the origin directly. The 
> volume of traffic still has to be handled. Yes, it's a burden on the ISP, but 
> it's a burden created by the usage created by their subscribers. 

Re: wow, lots of akamai

[Matt Erculiani]

Patrick,

> Matt: Are you arguing the CDNs are at fault because the game companies
tell everyone to download simultaneously, and
> the ISPs sold the users connectivity to do that download?

While a gross oversimplification, yes, that's basically what I'm saying; I
know it may not be a popular opinion, but I stand by it. There aren't any
villains here though, just lots of good suggestions in this thread to make
the internet work better for everyone, without spending large swaths of
money to cover the demand of an infrequent, large, update for a single game.

CNDs do, however, have a responsibility to be good netizens and get this
data out in a manner that doesn't cause disruption. They know the technical
challenges of distributing that much data to the masses, the game company
does not, that's why they outsourced it to a CDN. If the CDN knows what the
gaming company is asking for is pushing the limits of our current
infrastructure, they have a responsibility to relay those limitations that
are outside of their control to their customer, as any responsible vendor
should. Instead, there may be an element of "oh yeah sure, we can do that"
or "the customer is always right" going on here and modern limitations are
being disregarded.

The idea behind the internet is not that every user can always have their
entire capacity available for a single destination *regardless of what
everyone else is doing *(and *especially *if they're all going to the same
place too), the user has purchased that capacity into their provider's
network as a whole, gaining access to all of their connections to all of
the various endpoints on the internet at a backbone and peering capacity
that is economically viable given normal peak demand with some cushion
built-in for redundancy. If that's the desire to have full capacity
available to Akamai available at all times, then everyone needs dedicated
P2P circuits direct to Akamai, but that's not practical.

If you own an ISP and you're not oversubscribing, you're not making
money, period. To use your analogy, if you've ever been to a gym in
January, you've seen a similar phenomenon first-hand. There aren't enough
machines for everyone, and the gym isn't going to add them because this is
a once-a-year thing and it goes away after a few weeks when many people get
tired of fulfilling their new year's resolutions. Why should the gym limit
its sales to exactly the capacity it has available (or add a lot more
machines), when it knows that for the overwhelming majority of the year,
there will always be dozens of empty machines across the floor?

If Akamai was doing these updates more frequently (weekly for example) then
sure, it is on the ISP to augment, because this has become the "new
normal". But these updates for a single game that happen once per quarter
are hardly able to be considered normal. Sure, some day 50GB updates will
be the norm, but that's not today, and when it is, somebody else will be
pushing out 250GB updates quarterly. This problem isn't going away soon,
and it can't be fixed permanently by just adding more capacity, it's a
complex technical challenge that CDN's ought to give some more thought, and
game publishers should start considering too.

-Matt

On Thu, Apr 1, 2021 at 3:30 PM Patrick W. Gilmore  wrote:

> I am sorry, maybe I misunderstand.
>
> Matt: Are you arguing the CDNs are at fault because the game companies
> tell everyone to download simultaneously, and the ISPs sold the users
> connectivity to do that download?
>
> If so, are you really arguing “I sold my users XXX Mbps, but if they try
> to use it, I want *YOU* to tell them no”? Because that is what it sounds
> like to me.
>
> Imagine a gym sold 10,000 memberships with 10 machines because they
> figured everyone would sit on their ass. They would be right most of the
> time - and rake in that sweet, sweet monthly cash for zero effort after the
> initial sale. But if Oprah or Cher or Biden or some other person famous
> enough to go by one name tweets “get your ass to the gym!!", does the gym
> really think getting mad at Oprah is the solution? Or do they expect Oprah
> to pay for the extra machines they have to buy now?
>
> Selling a service you know will not work if everyone uses it
> simultaneously can be profitable, but there is risk. Do not blame third
> parties when you lose that bet.
>
> --
> TTFN,
> patrick
>
> On Apr 1, 2021, at 5:04 PM, Tom Beecher  wrote:
>
> No disrespect taken, or intended back in your direction, but again, I
> disagree.
>
> If thousands of users are downloading 50G files at the same time, it
> really doesn't matter if they are pulling from a CDN or the origin
> directly. The volume of traffic still has to be handled. Yes, it's a burden
> on the ISP, but it's a burden created by the usage created by their
> subscribers.
>
>
> On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani 
> wrote:
>
>> Tom,
>>
>> All due respect, but there is a massive difference between one user
>> downloading 50G 

RE: wow, lots of akamai

[Tony Wicks]

This is not actually (as in yes it does matter) the case, if a file comes from 
a CDN it is often a close and low latency source that will run up to very high 
speeds. For example in our case we connect to local peering exchanges (or 
PNI’s/local caches) at 100G or Nx10G with latency to the end user in the 1-30ms 
range resulting in very large peaks of local backhaul traffic. If a file is 
delivers from source or from remote CDN’s/exchanges these are located in other 
countries with between 25ms (New Zealand to Australia) and 130-200ms (New 
Zealand to LA/SJC or Singapore) latency, this results in a much slower and 
normally barely noticeable traffic blip. Yes as an ISP we need to carry the 
traffic in both cases but the first case can result in a 20-30% local backhaul 
increase for a couple of hours and in the second case its just BAU traffic for 
a day or two. Local CDN is obviously the better option for cost and the 
consumer, but you certainly do notice the traffic in local backhaul.

 

From: NANOG  On Behalf Of Tom Beecher
Sent: Friday, 2 April 2021 10:05 am
To: Matt Erculiani 
Cc: North American Operators' Group 
Subject: Re: wow, lots of akamai

 

 

If thousands of users are downloading 50G files at the same time, it really 
doesn't matter if they are pulling from a CDN or the origin directly. The 
volume of traffic still has to be handled. Yes, it's a burden on the ISP, but 
it's a burden created by the usage created by their subscribers. 

 

 

Re: wow, lots of akamai

[Patrick W. Gilmore]

I am sorry, maybe I misunderstand.

Matt: Are you arguing the CDNs are at fault because the game companies tell 
everyone to download simultaneously, and the ISPs sold the users connectivity 
to do that download?

If so, are you really arguing “I sold my users XXX Mbps, but if they try to use 
it, I want *YOU* to tell them no”? Because that is what it sounds like to me.

Imagine a gym sold 10,000 memberships with 10 machines because they figured 
everyone would sit on their ass. They would be right most of the time - and 
rake in that sweet, sweet monthly cash for zero effort after the initial sale. 
But if Oprah or Cher or Biden or some other person famous enough to go by one 
name tweets “get your ass to the gym!!", does the gym really think getting mad 
at Oprah is the solution? Or do they expect Oprah to pay for the extra machines 
they have to buy now?

Selling a service you know will not work if everyone uses it simultaneously can 
be profitable, but there is risk. Do not blame third parties when you lose that 
bet.

-- 
TTFN,
patrick

> On Apr 1, 2021, at 5:04 PM, Tom Beecher  wrote:
> 
> No disrespect taken, or intended back in your direction, but again, I 
> disagree. 
> 
> If thousands of users are downloading 50G files at the same time, it really 
> doesn't matter if they are pulling from a CDN or the origin directly. The 
> volume of traffic still has to be handled. Yes, it's a burden on the ISP, but 
> it's a burden created by the usage created by their subscribers. 
> 
> 
> On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani  > wrote:
> Tom,
> 
> All due respect, but there is a massive difference between one user 
> downloading 50G and thousands of users each downloading 50G when they all go 
> to play their videogame of choice at around the same time.
> 
> -Matt
> 
> 
> 
> On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher  wrote:
> A user sends a few megabytes of request and receives 50 gigs of reply. They 
> aren't DDoSing the network, but they're amplifying a single 50 gig copy they 
> receive from the mothership and turning it into likely tens of terabytes of 
> traffic.
> Yes, that's a CDN's job, but that volume of legitimate traffic and the very 
> tiny window with which it is transmitted is likely to be a burden for even 
> the largest residential ISPs.
> 
> I'm sitting at home, and I could send a 50k request for a 50G file right now 
> from a source not fronted by a CDN. What do? My ISP is still has to deliver 
> it to me. The fact that the 50G file does or does not come from a CDN is 
> irrelevant. The CDN just happens to be a point source that a lot of users 
> happen to connect to. 
> 
> CDNs want to have the best performance to users because that's what brings 
> them business. A poorly performing CDN will lose customers to a better 
> performing one. The trend for years has been instead of ISPs investing in 
> infrastructure to effectively handle the traffic that their users request, 
> they turf that to CDNs. In many cases, a CDN will put a cache box in or 
> extend a circuit at a loss to them, because they know if the performance 
> metrics get bad, business will be taken elsewhere, even if the CAUSE of the 
> poor performance is actually at the edge of, or inside , the ISPs network. 
> 
> ISPs in the US can get away with this because their users are captive and 
> rarely have an alternative choice of provider.  
> 
> 
> On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani  > wrote:
> Patrick,
> 
> > First, to be blunt, if you really think Akamai nodes are “sitting idle for 
> > weeks” before CoD comes out with a new game,
> > you are clearly confused.
> 
> "Idle" in the sense that when you look at a graph of traffic before and after 
> a large push such as this makes the rest of the week's traffic look like a 
> horizontal line at the bottom, admittedly poor word choice, yes, but far from 
> "confused" as to what CDNs do under relatively normal circumstances. 
> Otherwise very valid points you've raised.
> 
> Tom,
> 
> > Akamai, and other CDNs, do not **generate** traffic ; they serve the 
> > requests generated by users.  
> 
> A user sends a few megabytes of request and receives 50 gigs of reply. They 
> aren't DDoSing the network, but they're amplifying a single 50 gig copy they 
> receive from the mothership and turning it into likely tens of terabytes of 
> traffic.
> Yes, that's a CDN's job, but that volume of legitimate traffic and the very 
> tiny window with which it is transmitted is likely to be a burden for even 
> the largest residential ISPs.
> 
> -Matt
> 
> On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore  > wrote:
> Matt:
> 
> I am going to disagree with your characterization of how Akamai - and many 
> other CDNs - manage things. First, to be blunt, if you really think Akamai 
> nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
> are clearly confused.
> 
> More 

Re: wow, lots of akamai

[Mike Hammett]

In terms of dollar flows, yes, the subscriber makes all requests. They make the 
requests of the ISP and of the game developer\publisher\whatever. 


However, the game publisher queues those requests. I'm meaning request 
generically, not a GET request or anything like that. The game publisher that 
contracts to the CDNs decides when to fulfill those requests, in the big 
picture. The game publisher is the one that then tells 100 million devices 
"Content Available". The rate that they do that is at their discretion. 


Me deciding to download 50 gigs of GIS imagery because I requested it at that 
moment isn't the same situation as 100 million people downloading COD because 
the publisher released it. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Matt Erculiani"  
Cc: "North American Operators' Group"  
Sent: Thursday, April 1, 2021 4:04:34 PM 
Subject: Re: wow, lots of akamai 



No disrespect taken, or intended back in your direction, but again, I disagree. 


If thousands of users are downloading 50G files at the same time, it really 
doesn't matter if they are pulling from a CDN or the origin directly. The 
volume of traffic still has to be handled. Yes, it's a burden on the ISP, but 
it's a burden created by the usage created by their subscribers. 




On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani < merculi...@gmail.com > wrote: 



Tom, 


All due respect, but there is a massive difference between one user downloading 
50G and thousands of users each downloading 50G when they all go to play their 
videogame of choice at around the same time. 


-Matt 







On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher  wrote: 






A user sends a few megabytes of request and receives 50 gigs of reply. They 
aren't DDoSing the network, but they're amplifying a single 50 gig copy they 
receive from the mothership and turning it into likely tens of terabytes of 
traffic. 
Yes, that's a CDN's job, but that volume of legitimate traffic and the very 
tiny window with which it is transmitted is likely to be a burden for even the 
largest residential ISPs. 



I'm sitting at home, and I could send a 50k request for a 50G file right now 
from a source not fronted by a CDN. What do? My ISP is still has to deliver it 
to me. The fact that the 50G file does or does not come from a CDN is 
irrelevant. The CDN just happens to be a point source that a lot of users 
happen to connect to. 


CDNs want to have the best performance to users because that's what brings them 
business. A poorly performing CDN will lose customers to a better performing 
one. The trend for years has been instead of ISPs investing in infrastructure 
to effectively handle the traffic that their users request, they turf that to 
CDNs. In many cases, a CDN will put a cache box in or extend a circuit at a 
loss to them, because they know if the performance metrics get bad, business 
will be taken elsewhere, even if the CAUSE of the poor performance is actually 
at the edge of, or inside , the ISPs network. 


ISPs in the US can get away with this because their users are captive and 
rarely have an alternative choice of provider. 




On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani < merculi...@gmail.com > wrote: 




Patrick, 


> First, to be blunt, if you really think Akamai nodes are “sitting idle for 
> weeks” before CoD comes out with a new game, 
> you are clearly confused. 

"Idle" in the sense that when you look at a graph of traffic before and after a 
large push such as this makes the rest of the week's traffic look like a 
horizontal line at the bottom, admittedly poor word choice, yes, but far from 
"confused" as to what CDNs do under relatively normal circumstances. Otherwise 
very valid points you've raised. 



Tom, 


> Akamai, and other CDNs, do not **generate** traffic ; they serve the requests 
> generated by users. 


A user sends a few megabytes of request and receives 50 gigs of reply. They 
aren't DDoSing the network, but they're amplifying a single 50 gig copy they 
receive from the mothership and turning it into likely tens of terabytes of 
traffic. 
Yes, that's a CDN's job, but that volume of legitimate traffic and the very 
tiny window with which it is transmitted is likely to be a burden for even the 
largest residential ISPs. 


-Matt 


On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore < patr...@ianai.net > wrote: 




Matt: 


I am going to disagree with your characterization of how Akamai - and many 
other CDNs - manage things. First, to be blunt, if you really think Akamai 
nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
are clearly confused. 


More importantly, I know for a fact Akamai has spent ungodly amounts of money & 
resources putting content precisely where the 

Re: wow, lots of akamai

[Tom Beecher]

No disrespect taken, or intended back in your direction, but again, I
disagree.

If thousands of users are downloading 50G files at the same time, it really
doesn't matter if they are pulling from a CDN or the origin directly. The
volume of traffic still has to be handled. Yes, it's a burden on the ISP,
but it's a burden created by the usage created by their subscribers.


On Thu, Apr 1, 2021 at 4:57 PM Matt Erculiani  wrote:

> Tom,
>
> All due respect, but there is a massive difference between one user
> downloading 50G and thousands of users each downloading 50G when they all
> go to play their videogame of choice at around the same time.
>
> -Matt
>
>
>
> On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher  wrote:
>
>> A user sends a few megabytes of request and receives 50 gigs of reply.
>>> They aren't DDoSing the network, but they're amplifying a single 50 gig
>>> copy they receive from the mothership and turning it into likely tens of
>>> terabytes of traffic.
>>> Yes, that's a CDN's job, but that volume of legitimate traffic and the
>>> very tiny window with which it is transmitted is likely to be a burden for
>>> even the largest residential ISPs.
>>>
>>
>> I'm sitting at home, and I could send a 50k request for a 50G file right
>> now from a source not fronted by a CDN. What do? My ISP is still has to
>> deliver it to me. The fact that the 50G file does or does not come from a
>> CDN is irrelevant. The CDN just happens to be a point source that a lot of
>> users happen to connect to.
>>
>> CDNs want to have the best performance to users because that's what
>> brings them business. A poorly performing CDN will lose customers to a
>> better performing one. The trend for years has been instead of ISPs
>> investing in infrastructure to effectively handle the traffic that their
>> users request, they turf that to CDNs. In many cases, a CDN will put a
>> cache box in or extend a circuit at a loss to them, because they know if
>> the performance metrics get bad, business will be taken elsewhere, even if
>> the CAUSE of the poor performance is actually at the edge of, or inside ,
>> the ISPs network.
>>
>> ISPs in the US can get away with this because their users are captive and
>> rarely have an alternative choice of provider.
>>
>>
>> On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani 
>> wrote:
>>
>>> Patrick,
>>>
>>> > First, to be blunt, if you really think Akamai nodes are “sitting idle
>>> for weeks” before CoD comes out with a new game,
>>> > you are clearly confused.
>>>
>>> "Idle" in the sense that when you look at a graph of traffic before and
>>> after a large push such as this makes the rest of the week's traffic look
>>> like a horizontal line at the bottom, admittedly poor word choice, yes, but
>>> far from "confused" as to what CDNs do under relatively normal
>>> circumstances. Otherwise very valid points you've raised.
>>>
>>> Tom,
>>>
>>> > Akamai, and other CDNs, do not **generate** traffic ; they serve the
>>> requests generated by users.
>>>
>>> A user sends a few megabytes of request and receives 50 gigs of reply.
>>> They aren't DDoSing the network, but they're amplifying a single 50 gig
>>> copy they receive from the mothership and turning it into likely tens of
>>> terabytes of traffic.
>>> Yes, that's a CDN's job, but that volume of legitimate traffic and the
>>> very tiny window with which it is transmitted is likely to be a burden for
>>> even the largest residential ISPs.
>>>
>>> -Matt
>>>
>>> On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore 
>>> wrote:
>>>
 Matt:

 I am going to disagree with your characterization of how Akamai - and
 many other CDNs - manage things. First, to be blunt, if you really think
 Akamai nodes are “sitting idle for weeks” before CoD comes out with a new
 game, you are clearly confused.

 More importantly, I know for a fact Akamai has spent ungodly amounts of
 money & resources putting content precisely where the ISPs ask them to put
 it, deliver it over the pipes the ISPs ask them to deliver it, at precisely
 the capacity the ISPs tell them.

 On the other hand, I agree with your characterization of residential
 broadband. It is ridiculous to expect a neighborhood with 1,000 homes each
 with 1 Gbps links to have a terabit of uplink capacity. But it also should
 have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I
 have seen are closer to the latter than the former.

 Finally, this could quickly devolve into finger pointing. You say the
 CDNs bear some responsibility? They may well respond that the large
 broadband providers ask for cash to interconnect - but still require the
 CDNs to do all the work. The CDNs did not create the content, or tell the
 users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them
 to provide me with access to the Internet. Not just to the content that
 pays that provider.

 Personally, I 

Re: wow, lots of akamai

[Matt Erculiani]

Tom,

All due respect, but there is a massive difference between one user
downloading 50G and thousands of users each downloading 50G when they all
go to play their videogame of choice at around the same time.

-Matt



On Thu, Apr 1, 2021 at 2:46 PM Tom Beecher  wrote:

> A user sends a few megabytes of request and receives 50 gigs of reply.
>> They aren't DDoSing the network, but they're amplifying a single 50 gig
>> copy they receive from the mothership and turning it into likely tens of
>> terabytes of traffic.
>> Yes, that's a CDN's job, but that volume of legitimate traffic and the
>> very tiny window with which it is transmitted is likely to be a burden for
>> even the largest residential ISPs.
>>
>
> I'm sitting at home, and I could send a 50k request for a 50G file right
> now from a source not fronted by a CDN. What do? My ISP is still has to
> deliver it to me. The fact that the 50G file does or does not come from a
> CDN is irrelevant. The CDN just happens to be a point source that a lot of
> users happen to connect to.
>
> CDNs want to have the best performance to users because that's what brings
> them business. A poorly performing CDN will lose customers to a better
> performing one. The trend for years has been instead of ISPs investing in
> infrastructure to effectively handle the traffic that their users request,
> they turf that to CDNs. In many cases, a CDN will put a cache box in or
> extend a circuit at a loss to them, because they know if the performance
> metrics get bad, business will be taken elsewhere, even if the CAUSE of the
> poor performance is actually at the edge of, or inside , the ISPs network.
>
> ISPs in the US can get away with this because their users are captive and
> rarely have an alternative choice of provider.
>
>
> On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani 
> wrote:
>
>> Patrick,
>>
>> > First, to be blunt, if you really think Akamai nodes are “sitting idle
>> for weeks” before CoD comes out with a new game,
>> > you are clearly confused.
>>
>> "Idle" in the sense that when you look at a graph of traffic before and
>> after a large push such as this makes the rest of the week's traffic look
>> like a horizontal line at the bottom, admittedly poor word choice, yes, but
>> far from "confused" as to what CDNs do under relatively normal
>> circumstances. Otherwise very valid points you've raised.
>>
>> Tom,
>>
>> > Akamai, and other CDNs, do not **generate** traffic ; they serve the
>> requests generated by users.
>>
>> A user sends a few megabytes of request and receives 50 gigs of reply.
>> They aren't DDoSing the network, but they're amplifying a single 50 gig
>> copy they receive from the mothership and turning it into likely tens of
>> terabytes of traffic.
>> Yes, that's a CDN's job, but that volume of legitimate traffic and the
>> very tiny window with which it is transmitted is likely to be a burden for
>> even the largest residential ISPs.
>>
>> -Matt
>>
>> On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore 
>> wrote:
>>
>>> Matt:
>>>
>>> I am going to disagree with your characterization of how Akamai - and
>>> many other CDNs - manage things. First, to be blunt, if you really think
>>> Akamai nodes are “sitting idle for weeks” before CoD comes out with a new
>>> game, you are clearly confused.
>>>
>>> More importantly, I know for a fact Akamai has spent ungodly amounts of
>>> money & resources putting content precisely where the ISPs ask them to put
>>> it, deliver it over the pipes the ISPs ask them to deliver it, at precisely
>>> the capacity the ISPs tell them.
>>>
>>> On the other hand, I agree with your characterization of residential
>>> broadband. It is ridiculous to expect a neighborhood with 1,000 homes each
>>> with 1 Gbps links to have a terabit of uplink capacity. But it also should
>>> have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I
>>> have seen are closer to the latter than the former.
>>>
>>> Finally, this could quickly devolve into finger pointing. You say the
>>> CDNs bear some responsibility? They may well respond that the large
>>> broadband providers ask for cash to interconnect - but still require the
>>> CDNs to do all the work. The CDNs did not create the content, or tell the
>>> users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them
>>> to provide me with access to the Internet. Not just to the content that
>>> pays that provider.
>>>
>>> Personally, I have zero problems with the ISPs saying “give me a cache
>>> to put here with this sized uplink” or “please deliver to these users over
>>> this xconn / IX / whatever”. I have a huge problem with the ISPs blaming
>>> the ISPs for delivering what the ISP’s users request.
>>>
>>> Of course, this could all be solved if there were more competition in
>>> broadband in the US (and many other countries). But that is a totally
>>> different 10,000 post thread (that we have had many dozens of times).
>>>
>>> --
>>> TTFN,
>>> patrick
>>>
>>> On 

Re: wow, lots of akamai

[Mike Hammett]

Yes, but if they're an avid gamer, the console or PC is on all of the time 
anyway. 


If they're not an avid gamer, they probably don't care when it downloads. 




Also, most people I know leave their PCs on 24/7, save for whatever power 
saving modes they have. I know many (most?) consoles auto shut off after an 
amount of time and in some rare (underutilized) instances, the console will 
wake up for housekeeping such as this. 




I used to want everything instantly. Now as long as whatever game (that I don't 
really play anymore) or TV show or whatever downloads in the next week, I'm 
happy. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Niels Bakker"  
To: nanog@nanog.org 
Sent: Thursday, April 1, 2021 3:49:17 PM 
Subject: Re: wow, lots of akamai 

* na...@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]: 
>I'm not sure what kind of time lines are expected or engineered for 
>now, but it *seems* like its a 12 - 36 hour sprint to push the 
>content out. If so, push it out to 36 - 72 hours? Adjust accordingly 
>for however much off I am on the first time frame. 

Doesn't that mostly depend on when people turn on their gaming 
consoles to play? It's not on the publisher to dictate how often 
people want to play their game. 


-- Niels. 

Re: wow, lots of akamai

[Tom Beecher]

Lots of publishers will allow for new stuff to be pre-downloaded before a
specified release time. There was a time that it was probably helpful in
spreading the load out over time, but today it doesn't help much because
either everyone starts the preload at the same time, or people don't have
enough space on their console hard drives, so they have to delete the old
thing and download the entire new thing on release day. which also doesn't
help matters any.

On Thu, Apr 1, 2021 at 4:50 PM Niels Bakker  wrote:

> * na...@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]:
> >I'm not sure what kind of time lines are expected or engineered for
> >now, but it *seems* like its a 12 - 36 hour sprint to push the
> >content out. If so, push it out to 36 - 72 hours? Adjust accordingly
> >for however much off I am on the first time frame.
>
> Doesn't that mostly depend on when people turn on their gaming
> consoles to play? It's not on the publisher to dictate how often
> people want to play their game.
>
>
> -- Niels.
>

Re: wow, lots of akamai

[Niels Bakker]

* na...@ics-il.net (Mike Hammett) [Thu 01 Apr 2021, 21:51 CEST]:
I'm not sure what kind of time lines are expected or engineered for 
now, but it *seems* like its a 12 - 36 hour sprint to push the 
content out. If so, push it out to 36 - 72 hours? Adjust accordingly 
for however much off I am on the first time frame.


Doesn't that mostly depend on when people turn on their gaming 
consoles to play? It's not on the publisher to dictate how often 
people want to play their game.



-- Niels.

Re: wow, lots of akamai

[Tom Beecher]

>
> A user sends a few megabytes of request and receives 50 gigs of reply.
> They aren't DDoSing the network, but they're amplifying a single 50 gig
> copy they receive from the mothership and turning it into likely tens of
> terabytes of traffic.
> Yes, that's a CDN's job, but that volume of legitimate traffic and the
> very tiny window with which it is transmitted is likely to be a burden for
> even the largest residential ISPs.
>

I'm sitting at home, and I could send a 50k request for a 50G file right
now from a source not fronted by a CDN. What do? My ISP is still has to
deliver it to me. The fact that the 50G file does or does not come from a
CDN is irrelevant. The CDN just happens to be a point source that a lot of
users happen to connect to.

CDNs want to have the best performance to users because that's what brings
them business. A poorly performing CDN will lose customers to a better
performing one. The trend for years has been instead of ISPs investing in
infrastructure to effectively handle the traffic that their users request,
they turf that to CDNs. In many cases, a CDN will put a cache box in or
extend a circuit at a loss to them, because they know if the performance
metrics get bad, business will be taken elsewhere, even if the CAUSE of the
poor performance is actually at the edge of, or inside , the ISPs network.

ISPs in the US can get away with this because their users are captive and
rarely have an alternative choice of provider.


On Thu, Apr 1, 2021 at 4:33 PM Matt Erculiani  wrote:

> Patrick,
>
> > First, to be blunt, if you really think Akamai nodes are “sitting idle
> for weeks” before CoD comes out with a new game,
> > you are clearly confused.
>
> "Idle" in the sense that when you look at a graph of traffic before and
> after a large push such as this makes the rest of the week's traffic look
> like a horizontal line at the bottom, admittedly poor word choice, yes, but
> far from "confused" as to what CDNs do under relatively normal
> circumstances. Otherwise very valid points you've raised.
>
> Tom,
>
> > Akamai, and other CDNs, do not **generate** traffic ; they serve the
> requests generated by users.
>
> A user sends a few megabytes of request and receives 50 gigs of reply.
> They aren't DDoSing the network, but they're amplifying a single 50 gig
> copy they receive from the mothership and turning it into likely tens of
> terabytes of traffic.
> Yes, that's a CDN's job, but that volume of legitimate traffic and the
> very tiny window with which it is transmitted is likely to be a burden for
> even the largest residential ISPs.
>
> -Matt
>
> On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore 
> wrote:
>
>> Matt:
>>
>> I am going to disagree with your characterization of how Akamai - and
>> many other CDNs - manage things. First, to be blunt, if you really think
>> Akamai nodes are “sitting idle for weeks” before CoD comes out with a new
>> game, you are clearly confused.
>>
>> More importantly, I know for a fact Akamai has spent ungodly amounts of
>> money & resources putting content precisely where the ISPs ask them to put
>> it, deliver it over the pipes the ISPs ask them to deliver it, at precisely
>> the capacity the ISPs tell them.
>>
>> On the other hand, I agree with your characterization of residential
>> broadband. It is ridiculous to expect a neighborhood with 1,000 homes each
>> with 1 Gbps links to have a terabit of uplink capacity. But it also should
>> have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I
>> have seen are closer to the latter than the former.
>>
>> Finally, this could quickly devolve into finger pointing. You say the
>> CDNs bear some responsibility? They may well respond that the large
>> broadband providers ask for cash to interconnect - but still require the
>> CDNs to do all the work. The CDNs did not create the content, or tell the
>> users which content to pull. When I pay $NATIONAL_PROVIDER, I expect them
>> to provide me with access to the Internet. Not just to the content that
>> pays that provider.
>>
>> Personally, I have zero problems with the ISPs saying “give me a cache to
>> put here with this sized uplink” or “please deliver to these users over
>> this xconn / IX / whatever”. I have a huge problem with the ISPs blaming
>> the ISPs for delivering what the ISP’s users request.
>>
>> Of course, this could all be solved if there were more competition in
>> broadband in the US (and many other countries). But that is a totally
>> different 10,000 post thread (that we have had many dozens of times).
>>
>> --
>> TTFN,
>> patrick
>>
>> On Apr 1, 2021, at 3:53 PM, Matt Erculiani  wrote:
>>
>> Niels,
>>
>> I think to clarify Jean's point, when you buy a 300mbps circuit, you're
>> paying for 300mbps of *internet *access.
>>
>> That does not mean that a network should (and in this case small-medium
>> ones simply can't) build all of their capacity to service a large number of
>> customer circuits at line rate at the same 

Re: wow, lots of akamai

[Matt Erculiani]

Patrick,

> First, to be blunt, if you really think Akamai nodes are “sitting idle
for weeks” before CoD comes out with a new game,
> you are clearly confused.

"Idle" in the sense that when you look at a graph of traffic before and
after a large push such as this makes the rest of the week's traffic look
like a horizontal line at the bottom, admittedly poor word choice, yes, but
far from "confused" as to what CDNs do under relatively normal
circumstances. Otherwise very valid points you've raised.

Tom,

> Akamai, and other CDNs, do not **generate** traffic ; they serve the
requests generated by users.

A user sends a few megabytes of request and receives 50 gigs of reply. They
aren't DDoSing the network, but they're amplifying a single 50 gig copy
they receive from the mothership and turning it into likely tens of
terabytes of traffic.
Yes, that's a CDN's job, but that volume of legitimate traffic and the very
tiny window with which it is transmitted is likely to be a burden for even
the largest residential ISPs.

-Matt

On Thu, Apr 1, 2021 at 2:09 PM Patrick W. Gilmore  wrote:

> Matt:
>
> I am going to disagree with your characterization of how Akamai - and many
> other CDNs - manage things. First, to be blunt, if you really think Akamai
> nodes are “sitting idle for weeks” before CoD comes out with a new game,
> you are clearly confused.
>
> More importantly, I know for a fact Akamai has spent ungodly amounts of
> money & resources putting content precisely where the ISPs ask them to put
> it, deliver it over the pipes the ISPs ask them to deliver it, at precisely
> the capacity the ISPs tell them.
>
> On the other hand, I agree with your characterization of residential
> broadband. It is ridiculous to expect a neighborhood with 1,000 homes each
> with 1 Gbps links to have a terabit of uplink capacity. But it also should
> have a lot more than 10 Gbps, IMHO. Unfortunately, most neighborhoods I
> have seen are closer to the latter than the former.
>
> Finally, this could quickly devolve into finger pointing. You say the CDNs
> bear some responsibility? They may well respond that the large broadband
> providers ask for cash to interconnect - but still require the CDNs to do
> all the work. The CDNs did not create the content, or tell the users which
> content to pull. When I pay $NATIONAL_PROVIDER, I expect them to provide me
> with access to the Internet. Not just to the content that pays that
> provider.
>
> Personally, I have zero problems with the ISPs saying “give me a cache to
> put here with this sized uplink” or “please deliver to these users over
> this xconn / IX / whatever”. I have a huge problem with the ISPs blaming
> the ISPs for delivering what the ISP’s users request.
>
> Of course, this could all be solved if there were more competition in
> broadband in the US (and many other countries). But that is a totally
> different 10,000 post thread (that we have had many dozens of times).
>
> --
> TTFN,
> patrick
>
> On Apr 1, 2021, at 3:53 PM, Matt Erculiani  wrote:
>
> Niels,
>
> I think to clarify Jean's point, when you buy a 300mbps circuit, you're
> paying for 300mbps of *internet *access.
>
> That does not mean that a network should (and in this case small-medium
> ones simply can't) build all of their capacity to service a large number of
> customer circuits at line rate at the same time for an extended
> period, ESPECIALLY to the exact same endpoint. It's just not economically
> reasonable to expect that. Remember we're talking about residential service
> here, not enterprise circuits.
>
> Therefore, how do you prevent this spike of [insert large number here]
> gigabits traversing the network at the same time from causing issues? Build
> more network? That sounds easy, but there are plenty of legitimate reasons
> why ISPs can't or don't want to do that, particularly for an event that
> only occurs once per quarter or so.
>
> Does Akamai bear some burden here to make these rollouts less troublesome
> for the ISPs they traverse through the last mile(s)? IMO yes, yes they do.
> When you're doing something new and unprecedented, as Akamai frequently
> brags about on Twitter, like having rapid, bursty growth of traffic, you
> need to consider that just because you can generate it, doesn't mean it can
> be delivered.  They've gotta be more sophisticated than a bunch of servers
> with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for
> them here to just blindly fill every link they have after sitting idle for
> weeks/months at a time and expect everything to come out alright and nobody
> to complain about it.
>
> On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker 
> wrote:
>
>> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
>> CEST]:
>> >An artificial roll out penalty somehow? Probably not at the ISP
>> >level, but more at the game level. Well, ISP could also have some
>> >mechanisms to reduce the impact or even Akamai could force a
>> >progressive roll 

Re: wow, lots of akamai

[Töma Gavrichenkov]

Peace,

On Thu, Apr 1, 2021, 11:16 PM Tom Beecher  wrote:

> Akamai, and other CDNs, do not **generate** traffic ; they serve the
> requests generated by users.
>

L3/4-wise, this is true.  Application-wise, this is quite the other way
around.

--
Töma

>

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

They add a cookie.

 

This generate traffic

 

From: NANOG  On Behalf Of Tom Beecher
Sent: April 1, 2021 4:12 PM
To: Matt Erculiani 
Cc: nanog@nanog.org list 
Subject: Re: wow, lots of akamai

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered. 

 

Akamai, and other CDNs, do not **generate** traffic ; they serve the requests 
generated by users.  

 

On Thu, Apr 1, 2021 at 3:54 PM Matt Erculiani mailto:merculi...@gmail.com> > wrote:

Niels,

 

I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying 
for 300mbps of internet access. 

 

That does not mean that a network should (and in this case small-medium ones 
simply can't) build all of their capacity to service a large number of customer 
circuits at line rate at the same time for an extended period, ESPECIALLY to 
the exact same endpoint. It's just not economically reasonable to expect that. 
Remember we're talking about residential service here, not enterprise circuits.

 

Therefore, how do you prevent this spike of [insert large number here] gigabits 
traversing the network at the same time from causing issues? Build more 
network? That sounds easy, but there are plenty of legitimate reasons why ISPs 
can't or don't want to do that, particularly for an event that only occurs once 
per quarter or so.

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered.  
They've gotta be more sophisticated than a bunch of servers with SSD arrays, 
ramdisks, and 100 gig interfaces, so there's no excuse for them here to just 
blindly fill every link they have after sitting idle for weeks/months at a time 
and expect everything to come out alright and nobody to complain about it.

 

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.




 

-- 

Matt Erculiani

ERCUL-ARIN

RE: wow, lots of akamai

[Luke Guillory]

Patrick,

“Personally, I have zero problems with the ISPs saying “give me a cache to put 
here with this sized uplink” or “please deliver to these users over this xconn 
/ IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for 
delivering what the ISP’s users request.”

We had to beg to get more local CDN resources which still doesn’t even deliver 
half of their traffic.


“More importantly, I know for a fact Akamai has spent ungodly amounts of money 
& resources putting content precisely where the ISPs ask them to put it, 
deliver it over the pipes the ISPs ask them to deliver it, at precisely the 
capacity the ISPs tell them.”

From our experience this hasn’t been the case, can’t get PNI, can’t get them to 
add resources on their end to send more IX traffic. Don’t take that as me 
thinking they’re not doing things on their end to try and make things better, 
I’m not blind that it takes a massive number of resources. We just haven’t see 
things the way you have is all.






From: NANOG  On Behalf Of 
Patrick W. Gilmore
Sent: Thursday, April 1, 2021 3:09 PM
To: North American Operators' Group 
Subject: Re: wow, lots of akamai

*External Email: Use Caution*
Matt:

I am going to disagree with your characterization of how Akamai - and many 
other CDNs - manage things. First, to be blunt, if you really think Akamai 
nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
are clearly confused.

More importantly, I know for a fact Akamai has spent ungodly amounts of money & 
resources putting content precisely where the ISPs ask them to put it, deliver 
it over the pipes the ISPs ask them to deliver it, at precisely the capacity 
the ISPs tell them.

On the other hand, I agree with your characterization of residential broadband. 
It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps 
links to have a terabit of uplink capacity. But it also should have a lot more 
than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to 
the latter than the former.

Finally, this could quickly devolve into finger pointing. You say the CDNs bear 
some responsibility? They may well respond that the large broadband providers 
ask for cash to interconnect - but still require the CDNs to do all the work. 
The CDNs did not create the content, or tell the users which content to pull. 
When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the 
Internet. Not just to the content that pays that provider.

Personally, I have zero problems with the ISPs saying “give me a cache to put 
here with this sized uplink” or “please deliver to these users over this xconn 
/ IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for 
delivering what the ISP’s users request.

Of course, this could all be solved if there were more competition in broadband 
in the US (and many other countries). But that is a totally different 10,000 
post thread (that we have had many dozens of times).

--
TTFN,
patrick


On Apr 1, 2021, at 3:53 PM, Matt Erculiani 
mailto:merculi...@gmail.com>> wrote:

Niels,

I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying 
for 300mbps of internet access.

That does not mean that a network should (and in this case small-medium ones 
simply can't) build all of their capacity to service a large number of customer 
circuits at line rate at the same time for an extended period, ESPECIALLY to 
the exact same endpoint. It's just not economically reasonable to expect that. 
Remember we're talking about residential service here, not enterprise circuits.

Therefore, how do you prevent this spike of [insert large number here] gigabits 
traversing the network at the same time from causing issues? Build more 
network? That sounds easy, but there are plenty of legitimate reasons why ISPs 
can't or don't want to do that, particularly for an event that only occurs once 
per quarter or so.

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered.  
They've gotta be more sophisticated than a bunch of servers with SSD arrays, 
ramdisks, and 100 gig interfaces, so there's no excuse for them here to just 
blindly fill every link they have after sitting idle for weeks/months at a time 
and expect everything to come out alright and nobody to complain about it.

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker 
mailto:na...@bakker.net>> wrote:
* nanog@nanog.org<mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP
>level, but more at the game level. Well, ISP 

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

I find all of these comments interesting and useful even if we don’t all agree 
on what would be the best solution.

 

Maybe it’s just a misunderstanding from the perspective of each parties. Maybe 
there is no solution, but it’s still interesting.

 

So far, everybody seems to have a bit of the solution.

 

In the end, if I search on this mailing list:

 

Wow, spike, huge, akamai, yesterday or a combination of these words… the answer 
is always CoD. 


Jean

 

From: NANOG  On Behalf Of Patrick W. 
Gilmore
Sent: April 1, 2021 4:09 PM
To: North American Operators' Group 
Subject: Re: wow, lots of akamai

 

Matt:

 

I am going to disagree with your characterization of how Akamai - and many 
other CDNs - manage things. First, to be blunt, if you really think Akamai 
nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
are clearly confused.

 

More importantly, I know for a fact Akamai has spent ungodly amounts of money & 
resources putting content precisely where the ISPs ask them to put it, deliver 
it over the pipes the ISPs ask them to deliver it, at precisely the capacity 
the ISPs tell them.

 

On the other hand, I agree with your characterization of residential broadband. 
It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps 
links to have a terabit of uplink capacity. But it also should have a lot more 
than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to 
the latter than the former.

 

Finally, this could quickly devolve into finger pointing. You say the CDNs bear 
some responsibility? They may well respond that the large broadband providers 
ask for cash to interconnect - but still require the CDNs to do all the work. 
The CDNs did not create the content, or tell the users which content to pull. 
When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the 
Internet. Not just to the content that pays that provider.

 

Personally, I have zero problems with the ISPs saying “give me a cache to put 
here with this sized uplink” or “please deliver to these users over this xconn 
/ IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for 
delivering what the ISP’s users request.

 

Of course, this could all be solved if there were more competition in broadband 
in the US (and many other countries). But that is a totally different 10,000 
post thread (that we have had many dozens of times).

 

-- 
TTFN,
patrick 





On Apr 1, 2021, at 3:53 PM, Matt Erculiani mailto:merculi...@gmail.com> > wrote:

 

Niels,

 

I think to clarify Jean's point, when you buy a 300mbps circuit, you're paying 
for 300mbps of internet access. 

 

That does not mean that a network should (and in this case small-medium ones 
simply can't) build all of their capacity to service a large number of customer 
circuits at line rate at the same time for an extended period, ESPECIALLY to 
the exact same endpoint. It's just not economically reasonable to expect that. 
Remember we're talking about residential service here, not enterprise circuits.

 

Therefore, how do you prevent this spike of [insert large number here] gigabits 
traversing the network at the same time from causing issues? Build more 
network? That sounds easy, but there are plenty of legitimate reasons why ISPs 
can't or don't want to do that, particularly for an event that only occurs once 
per quarter or so.

 

Does Akamai bear some burden here to make these rollouts less troublesome for 
the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
you're doing something new and unprecedented, as Akamai frequently brags about 
on Twitter, like having rapid, bursty growth of traffic, you need to consider 
that just because you can generate it, doesn't mean it can be delivered.  
They've gotta be more sophisticated than a bunch of servers with SSD arrays, 
ramdisks, and 100 gig interfaces, so there's no excuse for them here to just 
blindly fill every link they have after sitting idle for weeks/months at a time 
and expect everything to come out alright and nobody to complain about it.

 

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.




 

-- 

Matt Erculiani

ERCUL-ARIN

 

Re: wow, lots of akamai

[Tom Beecher]

>
> Does Akamai bear some burden here to make these rollouts less troublesome
> for the ISPs they traverse through the last mile(s)? IMO yes, yes they do.
> When you're doing something new and unprecedented, as Akamai frequently
> brags about on Twitter, like having rapid, bursty growth of traffic, you
> need to consider that just because you can generate it, doesn't mean it can
> be delivered.
>

Akamai, and other CDNs, do not **generate** traffic ; they serve the
requests generated by users.

On Thu, Apr 1, 2021 at 3:54 PM Matt Erculiani  wrote:

> Niels,
>
> I think to clarify Jean's point, when you buy a 300mbps circuit, you're
> paying for 300mbps of *internet *access.
>
> That does not mean that a network should (and in this case small-medium
> ones simply can't) build all of their capacity to service a large number of
> customer circuits at line rate at the same time for an extended
> period, ESPECIALLY to the exact same endpoint. It's just not economically
> reasonable to expect that. Remember we're talking about residential service
> here, not enterprise circuits.
>
> Therefore, how do you prevent this spike of [insert large number here]
> gigabits traversing the network at the same time from causing issues? Build
> more network? That sounds easy, but there are plenty of legitimate reasons
> why ISPs can't or don't want to do that, particularly for an event that
> only occurs once per quarter or so.
>
> Does Akamai bear some burden here to make these rollouts less troublesome
> for the ISPs they traverse through the last mile(s)? IMO yes, yes they do.
> When you're doing something new and unprecedented, as Akamai frequently
> brags about on Twitter, like having rapid, bursty growth of traffic, you
> need to consider that just because you can generate it, doesn't mean it can
> be delivered.  They've gotta be more sophisticated than a bunch of servers
> with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for
> them here to just blindly fill every link they have after sitting idle for
> weeks/months at a time and expect everything to come out alright and nobody
> to complain about it.
>
> On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker 
> wrote:
>
>> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
>> CEST]:
>> >An artificial roll out penalty somehow? Probably not at the ISP
>> >level, but more at the game level. Well, ISP could also have some
>> >mechanisms to reduce the impact or even Akamai could force a
>> >progressive roll out.
>>
>> It's an online game. You can't play the game with outdated assets.
>> You'd not see walls where other players would, for example.
>>
>> What you're suggesting is the ability of ISPs to market Internet access
>> at a certain speed but not have to deliver it based on conditions they
>> create.
>>
>>
>> -- Niels.
>>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>

RE: wow, lots of akamai

[Luke Guillory]

IX’s don’t really help the source doesn’t use them.

Akamai traffic.
17G via Local Cache
17G via Transit
8G via IXs.

Plenty of room on IXs for more on our side.



From: NANOG  On Behalf Of 
Mike Hammett
Sent: Thursday, April 1, 2021 2:31 PM
To: Niels Bakker 
Cc: nanog@nanog.org
Subject: Re: wow, lots of akamai

*External Email: Use Caution*
There likely is some amount of time between the product being "done" and the 
activation date. That time could be used (and may very well be for some 
platforms) to distribute the content ahead of when people need it. If too many 
points of congestion arise, the above mentioned time would need to be longer.


Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join 
IXes and push them bits at maximum speed!  ;-)


As an eyeball ISP, sometimes the congestion is in the home, creating a poor 
experience, yet no one above them is to blame.




-
Mike Hammett
Intelligent Computing Solutions
https://link.edgepilot.com/s/ac964af3/CIKwQEO-ZkiYpW6Z8sKObQ?u=http://www.ics-il.com/

Midwest-IX
https://link.edgepilot.com/s/aac6d8b8/o8NzA_6ZJESKpzRLQOS0Pw?u=http://www.midwest-ix.com/


From: "Niels Bakker" mailto:niels=na...@bakker.net>>
To: nanog@nanog.org<mailto:nanog@nanog.org>
Sent: Thursday, April 1, 2021 2:21:24 PM
Subject: Re: wow, lots of akamai

* nanog@nanog.org<mailto:nanog@nanog.org> (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP
>level, but more at the game level. Well, ISP could also have some
>mechanisms to reduce the impact or even Akamai could force a
>progressive roll out.

It's an online game. You can't play the game with outdated assets.
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access
at a certain speed but not have to deliver it based on conditions they
create.


-- Niels.



Links contained in this email have been replaced. If you click on a link in the 
email above, the link will be analyzed for known threats. If a known threat is 
found, you will not be able to proceed to the destination. If suspicious 
content is detected, you will see a warning.

Re: wow, lots of akamai

[Patrick W. Gilmore]

Matt:

I am going to disagree with your characterization of how Akamai - and many 
other CDNs - manage things. First, to be blunt, if you really think Akamai 
nodes are “sitting idle for weeks” before CoD comes out with a new game, you 
are clearly confused.

More importantly, I know for a fact Akamai has spent ungodly amounts of money & 
resources putting content precisely where the ISPs ask them to put it, deliver 
it over the pipes the ISPs ask them to deliver it, at precisely the capacity 
the ISPs tell them.

On the other hand, I agree with your characterization of residential broadband. 
It is ridiculous to expect a neighborhood with 1,000 homes each with 1 Gbps 
links to have a terabit of uplink capacity. But it also should have a lot more 
than 10 Gbps, IMHO. Unfortunately, most neighborhoods I have seen are closer to 
the latter than the former.

Finally, this could quickly devolve into finger pointing. You say the CDNs bear 
some responsibility? They may well respond that the large broadband providers 
ask for cash to interconnect - but still require the CDNs to do all the work. 
The CDNs did not create the content, or tell the users which content to pull. 
When I pay $NATIONAL_PROVIDER, I expect them to provide me with access to the 
Internet. Not just to the content that pays that provider.

Personally, I have zero problems with the ISPs saying “give me a cache to put 
here with this sized uplink” or “please deliver to these users over this xconn 
/ IX / whatever”. I have a huge problem with the ISPs blaming the ISPs for 
delivering what the ISP’s users request.

Of course, this could all be solved if there were more competition in broadband 
in the US (and many other countries). But that is a totally different 10,000 
post thread (that we have had many dozens of times).

-- 
TTFN,
patrick

> On Apr 1, 2021, at 3:53 PM, Matt Erculiani  wrote:
> 
> Niels,
> 
> I think to clarify Jean's point, when you buy a 300mbps circuit, you're 
> paying for 300mbps of internet access. 
> 
> That does not mean that a network should (and in this case small-medium ones 
> simply can't) build all of their capacity to service a large number of 
> customer circuits at line rate at the same time for an extended period, 
> ESPECIALLY to the exact same endpoint. It's just not economically reasonable 
> to expect that. Remember we're talking about residential service here, not 
> enterprise circuits.
> 
> Therefore, how do you prevent this spike of [insert large number here] 
> gigabits traversing the network at the same time from causing issues? Build 
> more network? That sounds easy, but there are plenty of legitimate reasons 
> why ISPs can't or don't want to do that, particularly for an event that only 
> occurs once per quarter or so.
> 
> Does Akamai bear some burden here to make these rollouts less troublesome for 
> the ISPs they traverse through the last mile(s)? IMO yes, yes they do. When 
> you're doing something new and unprecedented, as Akamai frequently brags 
> about on Twitter, like having rapid, bursty growth of traffic, you need to 
> consider that just because you can generate it, doesn't mean it can be 
> delivered.  They've gotta be more sophisticated than a bunch of servers with 
> SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for them 
> here to just blindly fill every link they have after sitting idle for 
> weeks/months at a time and expect everything to come out alright and nobody 
> to complain about it.
> 
> On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker  > wrote:
> * nanog@nanog.org  (Jean St-Laurent via NANOG) [Thu 
> 01 Apr 2021, 21:03 CEST]:
> >An artificial roll out penalty somehow? Probably not at the ISP 
> >level, but more at the game level. Well, ISP could also have some 
> >mechanisms to reduce the impact or even Akamai could force a 
> >progressive roll out.
> 
> It's an online game. You can't play the game with outdated assets. 
> You'd not see walls where other players would, for example.
> 
> What you're suggesting is the ability of ISPs to market Internet access 
> at a certain speed but not have to deliver it based on conditions they 
> create.
> 
> 
> -- Niels.
> 
> 
> -- 
> Matt Erculiani
> ERCUL-ARIN

Re: wow, lots of akamai

[Matt Erculiani]

Niels,

I think to clarify Jean's point, when you buy a 300mbps circuit, you're
paying for 300mbps of *internet *access.

That does not mean that a network should (and in this case small-medium
ones simply can't) build all of their capacity to service a large number of
customer circuits at line rate at the same time for an extended
period, ESPECIALLY to the exact same endpoint. It's just not economically
reasonable to expect that. Remember we're talking about residential service
here, not enterprise circuits.

Therefore, how do you prevent this spike of [insert large number here]
gigabits traversing the network at the same time from causing issues? Build
more network? That sounds easy, but there are plenty of legitimate reasons
why ISPs can't or don't want to do that, particularly for an event that
only occurs once per quarter or so.

Does Akamai bear some burden here to make these rollouts less troublesome
for the ISPs they traverse through the last mile(s)? IMO yes, yes they do.
When you're doing something new and unprecedented, as Akamai frequently
brags about on Twitter, like having rapid, bursty growth of traffic, you
need to consider that just because you can generate it, doesn't mean it can
be delivered.  They've gotta be more sophisticated than a bunch of servers
with SSD arrays, ramdisks, and 100 gig interfaces, so there's no excuse for
them here to just blindly fill every link they have after sitting idle for
weeks/months at a time and expect everything to come out alright and nobody
to complain about it.

On Thu, Apr 1, 2021 at 1:21 PM Niels Bakker  wrote:

> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> CEST]:
> >An artificial roll out penalty somehow? Probably not at the ISP
> >level, but more at the game level. Well, ISP could also have some
> >mechanisms to reduce the impact or even Akamai could force a
> >progressive roll out.
>
> It's an online game. You can't play the game with outdated assets.
> You'd not see walls where other players would, for example.
>
> What you're suggesting is the ability of ISPs to market Internet access
> at a certain speed but not have to deliver it based on conditions they
> create.
>
>
> -- Niels.
>


-- 
Matt Erculiani
ERCUL-ARIN

Re: wow, lots of akamai

[Mike Hammett]

Well yes, but I am aware of the commercial pressure to release it ASAP. How 
much of that is real, remains to be seen. I'm also aware it's a lot more tricky 
when you set release dates before you have a firm grasp on your ability to 
produce a stable, desired product on time. 

I'm not sure what kind of time lines are expected or engineered for now, but it 
*seems* like its a 12 - 36 hour sprint to push the content out. If so, push it 
out to 36 - 72 hours? Adjust accordingly for however much off I am on the first 
time frame. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jean St-Laurent"  
To: "Mike Hammett" , "Niels Bakker"  
Cc: nanog@nanog.org 
Sent: Thursday, April 1, 2021 2:41:38 PM 
Subject: RE: wow, lots of akamai 



This would be a good compromises for all. 

Slowly deliver the assets few days/weeks ahead. 

Then, on April 1 st at this exact same second, you open the gate. 

@Mike: bull’s eye! 

Jean 



From: NANOG  On Behalf Of Mike 
Hammett 
Sent: April 1, 2021 3:31 PM 
To: Niels Bakker  
Cc: nanog@nanog.org 
Subject: Re: wow, lots of akamai 


There likely is some amount of time between the product being "done" and the 
activation date. That time could be used (and may very well be for some 
platforms) to distribute the content ahead of when people need it. If too many 
points of congestion arise, the above mentioned time would need to be longer. 





Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join 
IXes and push them bits at maximum speed! ;-) 





As an eyeball ISP, sometimes the congestion is in the home, creating a poor 
experience, yet no one above them is to blame. 






- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -


From: "Niels Bakker" < niels=na...@bakker.net > 
To: nanog@nanog.org 
Sent: Thursday, April 1, 2021 2:21:24 PM 
Subject: Re: wow, lots of akamai 

* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]: 
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out. 

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example. 

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create. 


-- Niels. 

Re: wow, lots of akamai

[Niels Bakker]

* j...@ddostest.me (Jean St-Laurent) [Thu 01 Apr 2021, 21:41 CEST]:

This would be a good compromises for all.
Slowly deliver the assets few days/weeks ahead.


Excellent compromise except for the people who paid for the game. 
Why do they need to spend storage to solve your bandwidth problem?


CoD is being played on lots of devices with limited storage space, 
like PlayStation 4. Needing to have two versions of the game would be 
a heavy burden on owners. And not everybody has infinite disk space in 
their gaming PCs either.



-- Niels.

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

This would be a good compromises for all.

 

Slowly deliver the assets few days/weeks ahead. 

 

Then, on April 1st at this exact same second, you open the gate.

 

@Mike: bull’s eye!

 

Jean

 

From: NANOG  On Behalf Of Mike Hammett
Sent: April 1, 2021 3:31 PM
To: Niels Bakker 
Cc: nanog@nanog.org
Subject: Re: wow, lots of akamai

 

There likely is some amount of time between the product being "done" and the 
activation date. That time could be used (and may very well be for some 
platforms) to distribute the content ahead of when people need it. If too many 
points of congestion arise, the above mentioned time would need to be longer.

 

 

Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join 
IXes and push them bits at maximum speed!  ;-)

 

 

As an eyeball ISP, sometimes the congestion is in the home, creating a poor 
experience, yet no one above them is to blame.






-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

 

  _  

From: "Niels Bakker" mailto:niels=na...@bakker.net> >
To: nanog@nanog.org <mailto:nanog@nanog.org> 
Sent: Thursday, April 1, 2021 2:21:24 PM
Subject: Re: wow, lots of akamai

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.


-- Niels.

 

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

It’s in fact better, because the one with the new asset can actually play and 
not be totally frozen at loading and/or suffering lag and being kill in action.

 

Progressive rolls out is the key to happiness 

 

From: NANOG  On Behalf Of Mehmet Akcin
Sent: April 1, 2021 3:25 PM
To: nanog@nanog.org
Subject: Re: wow, lots of akamai

 

 

 

On Thu, Apr 1, 2021 at 12:23 Niels Bakker mailto:na...@bakker.net> > wrote:

* nanog@nanog.org <mailto:nanog@nanog.org>  (Jean St-Laurent via NANOG) [Thu 01 
Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.

 

It’s actually worst. You can’t even login without having latest version to play 
multiplayer.

 



-- Niels.

-- 

Mehmet
+1-424-298-1903

Re: wow, lots of akamai

[Mike Hammett]

There likely is some amount of time between the product being "done" and the 
activation date. That time could be used (and may very well be for some 
platforms) to distribute the content ahead of when people need it. If too many 
points of congestion arise, the above mentioned time would need to be longer. 




Of course as an IX operator, I encourage everyone (CDNs and eyeballs) to join 
IXes and push them bits at maximum speed! ;-) 




As an eyeball ISP, sometimes the congestion is in the home, creating a poor 
experience, yet no one above them is to blame. 






- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Niels Bakker"  
To: nanog@nanog.org 
Sent: Thursday, April 1, 2021 2:21:24 PM 
Subject: Re: wow, lots of akamai 

* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]: 
>An artificial roll out penalty somehow? Probably not at the ISP 
>level, but more at the game level. Well, ISP could also have some 
>mechanisms to reduce the impact or even Akamai could force a 
>progressive roll out. 

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example. 

What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create. 


-- Niels. 

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

No I didn't suggest that.

-Original Message-
From: NANOG  On Behalf Of Niels
Bakker
Sent: April 1, 2021 3:21 PM
To: nanog@nanog.org
Subject: Re: wow, lots of akamai

* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
>An artificial roll out penalty somehow? Probably not at the ISP level, 
>but more at the game level. Well, ISP could also have some mechanisms 
>to reduce the impact or even Akamai could force a progressive roll out.

It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.

What you're suggesting is the ability of ISPs to market Internet access at a
certain speed but not have to deliver it based on conditions they create.


-- Niels.

Re: wow, lots of akamai

[Tom Beecher]

There are a couple things going on that all combine together.

- Competition between CDNs has pushed $/byte numbers down a lot. (Good or
bad, depending on which side you're on. :) )
- Game developers are under constant pressure to deliver content to users
quicker
- Games are graphically much higher resolution and multi resolution, which
means more assets that don't compress well.

The only real pressure on a developer to shrink their file sized comes from
users running out of disk space on their consoles. Otherwise it's cheaper
to just pay for the content delivery than hire more developers to improve
the file sizes.

On Thu, Apr 1, 2021 at 3:02 PM Jean St-Laurent via NANOG 
wrote:

> I remembered working for a big ISP in Europe offering cable tv + internet
> with +20M subscribers
>
> Every time there was a huge power outage in major cities, all tv`s would
> go off at the same time. I don`t have stats on power grid stability in
> Europe Vs N/A.
>
> The problem, was when the power was coming back in big cities, all the tv
> subscribers would come back online at the exact same second or minute.
> More or less the same 2 or 3 minutes.
>
> What happened is that it would create a kind of internal DDoS and they
> would all timed out and give a weird error message. Something very useful
> like Error Code 0x8098808 Please call our support line at this phone number.
>
> The server sysadmins would go on a panic because all systems were
> overloaded. They often needed to do overtime because DB crashed, key
> servers there crashed, DB here crashed, whatever... there was always
> something crashing.  This was before the cloud when you could just push a
> slider and have tons of VMs or containers to absorb the load in real time.
> (in my dream)
>
> This would every time create frustration from the clients, the help desk,
> the support teams and also the upper management. Every time the teams were
> really tired after that. It was draining juice.
>
> Anyway, after some years of talking internally (red tape), we finally
> managed to install a random artificial penalty in the setup boxes when they
> boot after a power outage. Nothing like 20 minutes, but just enough to
> spread the load over a longer period of time. For the end user, it went
> transparent for them because, if the setup box would boot in 206 seconds
> instead of the super aggressive 34 seconds, well it booted and they could
> watch tv.
> Vs
>
> my system is totally frozen and it`s been like that for 20 minutes with
> weird messages because all your systems are down and the error msg said to
> call the help desk.
>
> This simple change to add 3 lines of code to add a random artificial boot
> penalty of few seconds, completely solve the problem. This way, when a city
> would black out, we wouldn't be self DDoS, because the systems would slowly
> rampup. The setup boxes would all reboot but, wait randomly before asking
> for the DRM package to unlock the cable TV service and validate whether
> billing is right.
>
> I`m no Call of Duty expert nor Akamai, but it's been many times that I
> observe the same question here:
>
> What's happening?
> Call of Duty!
> Okay.
>
> Would a kind of throttle help here?
>
> An artificial roll out penalty somehow? Probably not at the ISP level, but
> more at the game level. Well, ISP could also have some mechanisms to reduce
> the impact or even Akamai could force a progressive roll out.
>
> I`m not sure that the proposed solutions could work, but it seems to
> impact NANOG frequently and/or at least generate a call overnight/weekend.
> It seems to also happens just before long holidays when operations are
> sometimes on reduce personnel.
>
> Are big games roll out really impacting NANOG? or it's more a: Hey I was
> curious what happened and I thought to ask here on NANOG?
>
> #JustCurious
>
> Jean
>
> -----Original Message-
> From: NANOG  On Behalf Of
> aar...@gvtc.com
> Sent: April 1, 2021 12:12 PM
> To: 'Jared Mauch' ; 'Töma Gavrichenkov' <
> xima...@gmail.com>
> Cc: 'NANOG' 
> Subject: RE: wow, lots of akamai
>
> Gaming update... I had a feeling.  Thanks for the feedback folks.
>
> Thanks Jared, it's running well, before, during and after.  We have a lot
> of capacity there.
>
> -Aaron
>
>
>

Re: wow, lots of akamai

[Mehmet Akcin]

On Thu, Apr 1, 2021 at 12:23 Niels Bakker  wrote:

> * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> CEST]:
> >An artificial roll out penalty somehow? Probably not at the ISP
> >level, but more at the game level. Well, ISP could also have some
> >mechanisms to reduce the impact or even Akamai could force a
> >progressive roll out.
>
> It's an online game. You can't play the game with outdated assets.
> You'd not see walls where other players would, for example.
>
> What you're suggesting is the ability of ISPs to market Internet access
> at a certain speed but not have to deliver it based on conditions they
> create.
>

It’s actually worst. You can’t even login without having latest version to
play multiplayer.


>
> -- Niels.
>
-- 
Mehmet
+1-424-298-1903

Re: wow, lots of akamai

[Niels Bakker]

* nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03 CEST]:
An artificial roll out penalty somehow? Probably not at the ISP 
level, but more at the game level. Well, ISP could also have some 
mechanisms to reduce the impact or even Akamai could force a 
progressive roll out.


It's an online game. You can't play the game with outdated assets. 
You'd not see walls where other players would, for example.


What you're suggesting is the ability of ISPs to market Internet access 
at a certain speed but not have to deliver it based on conditions they 
create.



-- Niels.

Re: wow, lots of akamai

[Luke Guillory]

IOS 7 seemed to be sent to everyone at once causing large spikes along with 
saturating many links for smaller ISPs.

I believe after that it went more to a distribution type of sorts though I 
could be wrong. Maybe it was that 7 was so vastly different everyone was 
itching to try it.




Sent from my iPhone

On Apr 1, 2021, at 2:02 PM, Jean St-Laurent via NANOG  wrote:

*External Email: Use Caution*

I remembered working for a big ISP in Europe offering cable tv + internet with 
+20M subscribers

Every time there was a huge power outage in major cities, all tv`s would go off 
at the same time. I don`t have stats on power grid stability in Europe Vs N/A.

The problem, was when the power was coming back in big cities, all the tv 
subscribers would come back online at the exact same second or minute.
More or less the same 2 or 3 minutes.

What happened is that it would create a kind of internal DDoS and they would 
all timed out and give a weird error message. Something very useful like Error 
Code 0x8098808 Please call our support line at this phone number.

The server sysadmins would go on a panic because all systems were overloaded. 
They often needed to do overtime because DB crashed, key servers there crashed, 
DB here crashed, whatever... there was always something crashing.  This was 
before the cloud when you could just push a slider and have tons of VMs or 
containers to absorb the load in real time. (in my dream)

This would every time create frustration from the clients, the help desk, the 
support teams and also the upper management. Every time the teams were really 
tired after that. It was draining juice.

Anyway, after some years of talking internally (red tape), we finally managed 
to install a random artificial penalty in the setup boxes when they boot after 
a power outage. Nothing like 20 minutes, but just enough to spread the load 
over a longer period of time. For the end user, it went transparent for them 
because, if the setup box would boot in 206 seconds instead of the super 
aggressive 34 seconds, well it booted and they could watch tv.
Vs

my system is totally frozen and it`s been like that for 20 minutes with weird 
messages because all your systems are down and the error msg said to call the 
help desk.

This simple change to add 3 lines of code to add a random artificial boot 
penalty of few seconds, completely solve the problem. This way, when a city 
would black out, we wouldn't be self DDoS, because the systems would slowly 
rampup. The setup boxes would all reboot but, wait randomly before asking for 
the DRM package to unlock the cable TV service and validate whether billing is 
right.

I`m no Call of Duty expert nor Akamai, but it's been many times that I observe 
the same question here:

What's happening?
Call of Duty!
Okay.

Would a kind of throttle help here?

An artificial roll out penalty somehow? Probably not at the ISP level, but more 
at the game level. Well, ISP could also have some mechanisms to reduce the 
impact or even Akamai could force a progressive roll out.

I`m not sure that the proposed solutions could work, but it seems to impact 
NANOG frequently and/or at least generate a call overnight/weekend. It seems to 
also happens just before long holidays when operations are sometimes on reduce 
personnel.

Are big games roll out really impacting NANOG? or it's more a: Hey I was 
curious what happened and I thought to ask here on NANOG?

#JustCurious

Jean

-Original Message-
From: NANOG  On Behalf Of 
aar...@gvtc.com
Sent: April 1, 2021 12:12 PM
To: 'Jared Mauch' ; 'Töma Gavrichenkov' 

Cc: 'NANOG' 
Subject: RE: wow, lots of akamai

Gaming update... I had a feeling.  Thanks for the feedback folks.

Thanks Jared, it's running well, before, during and after.  We have a lot of 
capacity there.

-Aaron

RE: wow, lots of akamai

[Jean St-Laurent via NANOG]

I remembered working for a big ISP in Europe offering cable tv + internet with 
+20M subscribers

Every time there was a huge power outage in major cities, all tv`s would go off 
at the same time. I don`t have stats on power grid stability in Europe Vs N/A.

The problem, was when the power was coming back in big cities, all the tv 
subscribers would come back online at the exact same second or minute.
More or less the same 2 or 3 minutes.

What happened is that it would create a kind of internal DDoS and they would 
all timed out and give a weird error message. Something very useful like Error 
Code 0x8098808 Please call our support line at this phone number.

The server sysadmins would go on a panic because all systems were overloaded. 
They often needed to do overtime because DB crashed, key servers there crashed, 
DB here crashed, whatever... there was always something crashing.  This was 
before the cloud when you could just push a slider and have tons of VMs or 
containers to absorb the load in real time. (in my dream)

This would every time create frustration from the clients, the help desk, the 
support teams and also the upper management. Every time the teams were really 
tired after that. It was draining juice.

Anyway, after some years of talking internally (red tape), we finally managed 
to install a random artificial penalty in the setup boxes when they boot after 
a power outage. Nothing like 20 minutes, but just enough to spread the load 
over a longer period of time. For the end user, it went transparent for them 
because, if the setup box would boot in 206 seconds instead of the super 
aggressive 34 seconds, well it booted and they could watch tv. 
Vs 

my system is totally frozen and it`s been like that for 20 minutes with weird 
messages because all your systems are down and the error msg said to call the 
help desk.

This simple change to add 3 lines of code to add a random artificial boot 
penalty of few seconds, completely solve the problem. This way, when a city 
would black out, we wouldn't be self DDoS, because the systems would slowly 
rampup. The setup boxes would all reboot but, wait randomly before asking for 
the DRM package to unlock the cable TV service and validate whether billing is 
right.

I`m no Call of Duty expert nor Akamai, but it's been many times that I observe 
the same question here:

What's happening?
Call of Duty!
Okay.

Would a kind of throttle help here? 

An artificial roll out penalty somehow? Probably not at the ISP level, but more 
at the game level. Well, ISP could also have some mechanisms to reduce the 
impact or even Akamai could force a progressive roll out. 

I`m not sure that the proposed solutions could work, but it seems to impact 
NANOG frequently and/or at least generate a call overnight/weekend. It seems to 
also happens just before long holidays when operations are sometimes on reduce 
personnel.

Are big games roll out really impacting NANOG? or it's more a: Hey I was 
curious what happened and I thought to ask here on NANOG?

#JustCurious

Jean

-Original Message-
From: NANOG  On Behalf Of 
aar...@gvtc.com
Sent: April 1, 2021 12:12 PM
To: 'Jared Mauch' ; 'Töma Gavrichenkov' 

Cc: 'NANOG' 
Subject: RE: wow, lots of akamai

Gaming update... I had a feeling.  Thanks for the feedback folks.

Thanks Jared, it's running well, before, during and after.  We have a lot of 
capacity there.

-Aaron

RE: wow, lots of akamai

[aaron1]

Gaming update... I had a feeling.  Thanks for the feedback folks.

Thanks Jared, it's running well, before, during and after.  We have a lot of 
capacity there.

-Aaron

Re: wow, lots of akamai

[Jared Mauch]

> On Apr 1, 2021, at 11:15 AM, Töma Gavrichenkov  wrote:
> 
> Peace,
> 
> On Thu, Apr 1, 2021, 6:09 PM  wrote:
> That was a lot of traffic coming out of akamai aanp clusters the last couple 
> nights!  What was it?
> 
> "Call of Duty" update again, obviously.
> 
> https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzone-update-is-over-50gb


Yes, we have had a number of big customer traffic in recent days.  Hopefully 
traffic is flowing well for many networks.  If this is negatively impacting 
you, please reach out.

- Jared

Re: wow, lots of akamai

[james jones]

On Thu, Apr 1, 2021 at 11:16 AM Töma Gavrichenkov  wrote:

> Peace,
>
> On Thu, Apr 1, 2021, 6:09 PM  wrote:
>
>> That was a lot of traffic coming out of akamai aanp clusters the last
>> couple nights!  What was it?
>>
> "Call of Duty" update again, obviously.
>
>
> https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzone-update-is-over-50gb
>
> --
> Töma
>

HAHA

Re: wow, lots of akamai

[Töma Gavrichenkov]

Peace,

On Thu, Apr 1, 2021, 6:09 PM  wrote:

> That was a lot of traffic coming out of akamai aanp clusters the last
> couple nights!  What was it?
>
"Call of Duty" update again, obviously.

https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzone-update-is-over-50gb

--
Töma