Re: Elephant in the room - Akamai

: "Mark Tinka" To: nanog@nanog.org Sent: Monday, December 9, 2019 3:57:55 AM Subject: Re: Elephant in the room - Akamai On 8/Dec/19 19:17, Rod Beck wrote: Last time I spoke with an Akamai engineer many years ago the network was purely transit. Is that evolving? I beli

Re: Elephant in the room - Akamai

On 8/Dec/19 19:17, Rod Beck wrote: > Last time I spoke with an Akamai engineer many years ago the network > was purely transit. Is that evolving? I believe Akamai are building, to a reasonable degree, an on-net backbone. Mark.

Re: Elephant in the room - Akamai

On 12/8/19 3:37 PM, Fred Baker wrote: Sent from my iPad On Dec 5, 2019, at 9:03 PM, Stephen Satchell wrote: For SP-grade routers, there isn't "code" that needs to be added to combat buffer bloat. All an admin has to do is cut back on the number of packet buffers on each interface -- an

Re: Elephant in the room - Akamai

Sent from my iPad > On Dec 5, 2019, at 9:03 PM, Stephen Satchell wrote: > > For SP-grade routers, there isn't "code" that needs to be added to combat > buffer bloat. All an admin has to do is cut back on the number of packet > buffers on each interface -- an interface setting, you see. A

Re: Elephant in the room - Akamai

Once upon a time, Brandon Martin said: > I guess what I'm getting at is that it sounds like, if you cannot > source the content locally to the peering link, there's not likely > to be an internal connection to the same site from somewhere else > within the Akamai network to deliver that content

Re: Elephant in the room - Akamai

* rod.b...@unitedcablecompany.com (Rod Beck) [Sun 08 Dec 2019, 18:18 CET]: Last time I spoke with an Akamai engineer many years ago the network was purely transit. Is that evolving? https://conference.apnic.net/data/41/ix_100-akamai-apricot2016-23feb2016_1456157526.pdf Per those slides, PAIX

Re: Elephant in the room - Akamai

Last time I spoke with an Akamai engineer many years ago the network was purely transit. Is that evolving? From: NANOG on behalf of Jared Mauch Sent: Sunday, December 8, 2019 6:10 PM To: Brandon Martin Cc: nanog@nanog.org Subject: Re: Elephant in the room

Re: Elephant in the room - Akamai

Yep. Real estate must be one of their largest expenses and unlike bandwidth it is not going down to price.  From: Owen DeLong Sent: Sunday, December 8, 2019 6:07 PM To: Rod Beck Cc: Jared Mauch ; nanog@nanog.org Subject: Re: Elephant in the room - Akamai My

Re: Elephant in the room - Akamai

> On Dec 8, 2019, at 11:48 AM, Brandon Martin wrote: > > I guess what I'm getting at is that it sounds like, if you cannot source the > content locally to the peering link, there's not likely to be an internal > connection to the same site from somewhere else within the Akamai network to >

Re: Elephant in the room - Akamai

with 50 billion peers. > > Disclaimer: Not picking a fight. Better things to do. > > Regards, > > Roderick. > > From: Jared Mauch > Sent: Sunday, December 8, 2019 1:19 AM > To: Rod Beck > Cc: Shawn L ; nanog@nanog.org > Subject: Re: Elephant in the room -

Re: Elephant in the room - Akamai

On 12/8/19 10:58 AM, Jared Mauch wrote: Not all content is suitable in all locations based on the physical security or market situation. We have some content that can not be served, an example is locations where there are licensing requirements (eg: ICP for China). You will see a different

Re: Elephant in the room - Akamai

Let's take a minute and thank Jared for taking the time and responding. thank you, Jared. On Sun, Dec 8, 2019 at 10:58 AM Jared Mauch wrote: > Not all content is suitable in all locations based on the physical > security or market situation. We have some content that can not be served, > an

Re: Elephant in the room - Akamai

> Have there been any fundamental change in their network architecture > that might explain pulling these caches? Maybe not network architecture, but what if the cache-to-content ratio is dropping dramatically due to changes in consumer behavior and/or a huge increase in the underlying content

Re: Elephant in the room - Akamai

Not all content is suitable in all locations based on the physical security or market situation. We have some content that can not be served, an example is locations where there are licensing requirements (eg: ICP for China). You will see a different mix from our 20940 vs 16625 as well. Those

Re: Elephant in the room - Akamai

On 12/7/19 7:19 PM, Jared Mauch wrote: Please see my email on Friday where I outlined a few of the dynamics at play. Akamai isn’t just one thing, it’s an entire basket of products that all have their own resulting behaviors. This is why even though you may peer with us directly you may not

Re: Elephant in the room - Akamai

+100, and thanks to Jared. -Ben > On Dec 7, 2019, at 10:08 PM, Mark Tinka wrote: > >  > >> On 8/Dec/19 02:19, Jared Mauch wrote: >> >> I’ve been working hard to provide value to our AANP partners as well. I’ll >> try to stop responding to the list at this point but don’t hesitate to >>

Re: Elephant in the room - Akamai

the cost of peering with 50 billion peers. Disclaimer: Not picking a fight. Better things to do. Regards, Roderick. From: Jared Mauch Sent: Sunday, December 8, 2019 1:19 AM To: Rod Beck Cc: Shawn L ; nanog@nanog.org Subject: Re: Elephant in the room - Akamai

Re: Elephant in the room - Akamai

On 8/Dec/19 02:19, Jared Mauch wrote: > I’ve been working hard to provide value to our AANP partners as well. I’ll > try to stop responding to the list at this point but don’t hesitate to > contact me here or via other means if you’re seeing something weird. I know > I resolved a problem

Re: Elephant in the room - Akamai

On Dec 7, 2019, at 5:34 PM, Rod Beck wrote: > > Have there been any fundamental change in their network architecture that > might explain pulling these caches? Please see my email on Friday where I outlined a few of the dynamics at play. Akamai isn’t just one thing, it’s an entire basket of

Re: Elephant in the room - Akamai

On 6/Dec/19 21:29, Keenan Tims wrote: > > We're currently seeing about 80% of our AS20940 origin traffic coming > from transit, and I'm certain there's a significant additional amount > which is difficult to identify coming from on-net caches at our > upstream providers (though it appears from

Re: Elephant in the room - Akamai

Cc: nanog@nanog.org Subject: Re: Elephant in the room - Akamai > On Dec 7, 2019, at 12:06 PM, Seth Mattinen wrote: > > On 12/6/19 06:46, Fawcett, Nick via NANOG wrote: >> We had three onsite Akamai caches a few months ago. They called us up and >> said they are removing

Re: Elephant in the room - Akamai

> On Dec 7, 2019, at 3:01 PM, Eric Kuhnke wrote: > > I think this thread might be a perfect example that when an organization > reaches a sufficiently large size, one part of its engineering/operations > team may no longer be fully aware of what other work groups are doing. > Definitely a

Re: Elephant in the room - Akamai

I think this thread might be a perfect example that when an organization reaches a sufficiently large size, one part of its engineering/operations team may no longer be fully aware of what other work groups are doing. Definitely a structural challenge for ISPs that span very large geographical

Re: Elephant in the room - Akamai

anog.org Subject: Re: Elephant in the room - Akamai > On Dec 7, 2019, at 12:06 PM, Seth Mattinen wrote: > > On 12/6/19 06:46, Fawcett, Nick via NANOG wrote: >> We had three onsite Akamai caches a few months ago. They called us up and >> said they are removing that service and

Re: Elephant in the room - Akamai

> On Dec 7, 2019, at 12:06 PM, Seth Mattinen wrote: > > On 12/6/19 06:46, Fawcett, Nick via NANOG wrote: >> We had three onsite Akamai caches a few months ago. They called us up and >> said they are removing that service and sent us boxes to pack up the >> hardware and ship back. We’ve

Re: Elephant in the room - Akamai

On 12/6/19 06:46, Fawcett, Nick via NANOG wrote: We had three onsite Akamai caches a few months ago.  They called us up and said they are removing that service and sent us boxes to pack up the hardware and ship back.  We’ve had quite the increase in DIA traffic as a result of it. Same

Re: Elephant in the room - Akamai

Speaking as a (very) small operator, we've also been seeing less and less of our Akamai traffic coming to us over peering over the last couple years. I've reached out to Akamai NOC as well as Jared directly on a few occasions and while they've been helpful and their changes usually have some

Re: Elephant in the room - Akamai

On 12/5/19 6:02 PM, Valdis Klētnieks wrote: On Thu, 05 Dec 2019 14:18:07 -0800, Michael Thomas said: My suspicion is that the root problem was buffer bloat -- i flashed a new router with openwrt and was a little dismayed that the bufferbloat code is a plugin you have to enable. The buffer

Re: Elephant in the room - Akamai

> On Dec 6, 2019, at 9:59 AM, Chris Adams wrote: > > Once upon a time, Fawcett, Nick said: >> We had three onsite Akamai caches a few months ago. They called us up and >> said they are removing that service and sent us boxes to pack up the >> hardware and ship back. We’ve had quite the

Re: Elephant in the room - Akamai

Once upon a time, Fawcett, Nick said: > We had three onsite Akamai caches a few months ago. They called us up and > said they are removing that service and sent us boxes to pack up the hardware > and ship back. We’ve had quite the increase in DIA traffic as a result of it. Same here. We'd

RE: Elephant in the room - Akamai

We had three onsite Akamai caches a few months ago. They called us up and said they are removing that service and sent us boxes to pack up the hardware and ship back. We’ve had quite the increase in DIA traffic as a result of it. ~Nick From: NANOG On Behalf Of Kaiser, Erich Sent: Wednesday,

Re: Elephant in the room - Akamai

On 12/5/19 6:02 PM, Valdis Klētnieks wrote: (I also admit having no idea what percentage of the intermediate routers in the ISP's networks have gotten de-bloating code. For SP-grade routers, there isn't "code" that needs to be added to combat buffer bloat. All an admin has to do is cut back

Re: Elephant in the room - Akamai

On Thu, 05 Dec 2019 14:18:07 -0800, Michael Thomas said: > My suspicion is that the root problem was buffer bloat -- i flashed a > new router with openwrt and was a little dismayed that the bufferbloat > code is a plugin you have to enable. The buffer bloat got a lot better Friends don't let

Re: Elephant in the room - Akamai

Once upon a time, Valdis Klētnieks said: > And it's only going to get worse. Sony has already announced that the > Playstation 5 will have a (probably) 1-2 terabyte SSD. And even with that, > the > game packaging is set up to support only downloading the single-player or > multi-player

Re: Elephant in the room - Akamai

On 12/5/19 1:44 PM, Valdis Klētnieks wrote: On Thu, 05 Dec 2019 14:41:30 -0600, "Aaron Gould" said: Tarko. wow, gaming again ! It's not going away. gaming traffic is growing in a big way it seems. And it's only going to get worse. Sony has already announced that the Playstation 5 will

Re: Elephant in the room - Akamai

On Thu, 05 Dec 2019 14:41:30 -0600, "Aaron Gould" said: > Tarko. wow, gaming again ! It's not going away. gaming traffic is growing > in a big way it seems. And it's only going to get worse. Sony has already announced that the Playstation 5 will have a (probably) 1-2 terabyte SSD. And even

RE: Elephant in the room - Akamai

Tarko. wow, gaming again ! It's not going away. gaming traffic is growing in a big way it seems. Clayton.. My thoughts exactly! I too have wondered how valuable these aanp's were, but lately I'm seeing good efficiency Thanks y'all -Aaron

RE: Elephant in the room - Akamai

Our AANP cache seems to have done the same in the past 2 nights. Lots of traffic that has never been there before. It has however not reduced the amount of traffic we're getting from AS20940 directly - still hit a new record level last night. We've got a request in with them for a PNI.

Re: Elephant in the room - Akamai

hey, I see my Akamai aanp cache utilization at all-time highs the last 2 nights as well.  Curious what it is. Halo Reach release. -- tarko

RE: Elephant in the room - Akamai

I see my Akamai aanp cache utilization at all-time highs the last 2 nights as well. Curious what it is. Jared, you can reply to my off-list if you wish, or on-list if it would benefit the community. Thanks, Aaron

Re: Elephant in the room - Akamai

Yes inbound. The patterns are not typical, we are talking gigs of traffic moved off of the IX side and onto our DIA side. They have reached out to me already. Will see what happens. Will post follow-up. On Thu, Dec 5, 2019 at 3:15 AM Bryan Holloway wrote: > > On 12/5/19 8:48 AM, Matthew

Re: Elephant in the room - Akamai

Good morning! If you are having Akamai issues you can reach out to me and I will help you out. - Jared > On Dec 4, 2019, at 10:04 PM, Kaiser, Erich wrote: > >  > Lets talk Akamai > > They have shifted 90% of their traffic off IXs and onto our full route DIA, > anyone else seeing this

Re: Elephant in the room - Akamai

On 12/5/19 8:48 AM, Matthew Petach wrote: On Wed, Dec 4, 2019, 19:05 Kaiser, Erich > wrote: Lets talk Akamai [...] The last two nights the traffic levels to them has skyrocketed as well. Any insight? Erich Kaiser The Fusion Network

Re: Elephant in the room - Akamai

On Wed, Dec 4, 2019, 19:05 Kaiser, Erich wrote: > Lets talk Akamai > [...] > The last two nights the traffic levels to them has skyrocketed as well. > > Any insight? > > > Erich Kaiser > The Fusion Network > As a CDN, I would usually expect to see traffic *from* Akamai to be the large

Re: Elephant in the room - Akamai

I don't have any insight but can confirm I am seeing the same thing. (Traffic shift back onto transit links) They did tell me they were having some bandwidth issues and are working on it. I am currently awaiting a direct PNI with them but haven't heard from them in some time.