Re: Elephant in the room - Akamai

[Mike Hammett]

There's no need for speculation. Jared has already said in this thread that's 
exactly what he was hired for. 

https://www.youtube.com/watch?v=KXBKnAbW4hQ 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Monday, December 9, 2019 3:57:55 AM 
Subject: Re: Elephant in the room - Akamai 




On 8/Dec/19 19:17, Rod Beck wrote: 





Last time I spoke with an Akamai engineer many years ago the network was purely 
transit. Is that evolving? 



I believe Akamai are building, to a reasonable degree, an on-net backbone. 

Mark. 

Re: Elephant in the room - Akamai

[Mark Tinka]

On 8/Dec/19 19:17, Rod Beck wrote:

> Last time I spoke with an Akamai engineer many years ago the network
> was purely transit. Is that evolving?

I believe Akamai are building, to a reasonable degree, an on-net backbone.

Mark.

Re: Elephant in the room - Akamai

[Michael Thomas]

On 12/8/19 3:37 PM, Fred Baker wrote:


Sent from my iPad


On Dec 5, 2019, at 9:03 PM, Stephen Satchell  wrote:

For SP-grade routers, there isn't "code" that needs to be added to combat 
buffer bloat.  All an admin has to do is cut back on the number of packet buffers on each 
interface -- an interface setting, you see.

A common misconception, and disagrees with the research on the topic.

Let me describe this conceptually. Think of a file transfer (a streaming flow 
can be thought of in those terms, as can web pages etc) as four groups of 
packets - those that have been delivered correctly and therefore don’t affect 
the window or flow rate, those that have been delivered out of order and 
therefore reduce the window and might get retransmitted even though they need 
not be resent, those that are sitting in a queue somewhere and therefore add 
latency, and those that haven’t been transmitted yet. If I have a large number 
of sessions transiting an interface, each one is likely to have a packet or two 
near the head of the queue; after that, it tends to thin out, with the sessions 
with the largest windows having packets deep in the queue, and sessions with 
smaller windows not so much.

If you reduce the queue depth, it does reduce that deep-in-the-queue group - 
there is no storage deep in the queue to hold it. What it does, however, is 
increase any given packet’s probability of loss (loss being the extreme case of 
delay, and when you reduce delay unintelligently is the byproduct), and 
therefore the second category of packets - the ones that managed to get through 
after a packet was lost, and therefore arrived out of order and have some 
probability of being retransmitted and therefore delivered multiple times.

What AQM technologies attempt to do (we can argue about the relative degree of 
success in different technologies; I’m talking about them as a class) is 
identify sessions in that deep-in-the-queue category and cause them to 
temporarily reduce their windows - keeping most of their outstanding packets 
near the head of the queue. Reducing their windows has the effect of moving 
packets out of the network buffers (bufferbloat) and reordering queues in the 
receiving host to “hasn’t been sent yet” in the sending host.  That also 
reduces median latency, meaning that the sessions with reduced windows don’t 
generally “slow down” - they simply keep less of their data streams in the 
network with reduced median latency.



So are saying in effect that the receiving host is, essentially, 
managing the sending host's queue by modulating the receiver's window? 
That seems really weird to me, and probably means I've got it wrong. How 
would the receiving host know when and why it should change the window, 
unless of course loss or other measurable things? If everything is 
chugging away, the receiver doesn't have any idea that the sender is 
starving other sessions, right?


It just seems to me that this is a sending hosts queuing problem?

Mike

Re: Elephant in the room - Akamai

[Fred Baker]

Sent from my iPad

> On Dec 5, 2019, at 9:03 PM, Stephen Satchell  wrote:
> 
> For SP-grade routers, there isn't "code" that needs to be added to combat 
> buffer bloat.  All an admin has to do is cut back on the number of packet 
> buffers on each interface -- an interface setting, you see.

A common misconception, and disagrees with the research on the topic.

Let me describe this conceptually. Think of a file transfer (a streaming flow 
can be thought of in those terms, as can web pages etc) as four groups of 
packets - those that have been delivered correctly and therefore don’t affect 
the window or flow rate, those that have been delivered out of order and 
therefore reduce the window and might get retransmitted even though they need 
not be resent, those that are sitting in a queue somewhere and therefore add 
latency, and those that haven’t been transmitted yet. If I have a large number 
of sessions transiting an interface, each one is likely to have a packet or two 
near the head of the queue; after that, it tends to thin out, with the sessions 
with the largest windows having packets deep in the queue, and sessions with 
smaller windows not so much.

If you reduce the queue depth, it does reduce that deep-in-the-queue group - 
there is no storage deep in the queue to hold it. What it does, however, is 
increase any given packet’s probability of loss (loss being the extreme case of 
delay, and when you reduce delay unintelligently is the byproduct), and 
therefore the second category of packets - the ones that managed to get through 
after a packet was lost, and therefore arrived out of order and have some 
probability of being retransmitted and therefore delivered multiple times.

What AQM technologies attempt to do (we can argue about the relative degree of 
success in different technologies; I’m talking about them as a class) is 
identify sessions in that deep-in-the-queue category and cause them to 
temporarily reduce their windows - keeping most of their outstanding packets 
near the head of the queue. Reducing their windows has the effect of moving 
packets out of the network buffers (bufferbloat) and reordering queues in the 
receiving host to “hasn’t been sent yet” in the sending host.  That also 
reduces median latency, meaning that the sessions with reduced windows don’t 
generally “slow down” - they simply keep less of their data streams in the 
network with reduced median latency.

Re: Elephant in the room - Akamai

[Chris Adams]

Once upon a time, Brandon Martin  said:
> I guess what I'm getting at is that it sounds like, if you cannot
> source the content locally to the peering link, there's not likely
> to be an internal connection to the same site from somewhere else
> within the Akamai network to deliver that content and, instead, the
> target network should expect it to come in over the "public
> Internet" via some other connection.  Is that accurate?

I believe this is true of multiple content networks.  For example, we
peer with Amazon in a couple of locations, but a significant amount of
traffic frmo their AS comes across transit rather than peering.

In old terms, this is "hot potato" routing - where the source gets the
traffic out of their network as soon as possible, rather than spend
internal resources to carry it as close to the destination as they can.
-- 
Chris Adams 

Re: Elephant in the room - Akamai

[Niels Bakker]

* rod.b...@unitedcablecompany.com (Rod Beck) [Sun 08 Dec 2019, 18:18 CET]:

Last time I spoke with an Akamai engineer many years ago the network was purely 
transit. Is that evolving?


https://conference.apnic.net/data/41/ix_100-akamai-apricot2016-23feb2016_1456157526.pdf

Per those slides, PAIX in 2000, LINX, DE-CIX, AMS-IX in 2001, JPIX in 
2002, so you must have spoken with an engineer shortly after the 
company was founded in 1998.



-- Niels.

Re: Elephant in the room - Akamai

[Rod Beck]

Last time I spoke with an Akamai engineer many years ago the network was purely 
transit. Is that evolving?


From: NANOG  on behalf of Jared Mauch 

Sent: Sunday, December 8, 2019 6:10 PM
To: Brandon Martin 
Cc: nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai



> On Dec 8, 2019, at 11:48 AM, Brandon Martin  wrote:
>
> I guess what I'm getting at is that it sounds like, if you cannot source the 
> content locally to the peering link, there's not likely to be an internal 
> connection to the same site from somewhere else within the Akamai network to 
> deliver that content and, instead, the target network should expect it to 
> come in over the "public Internet" via some other connection.  Is that 
> accurate?
>
> Thanks for the clarifications.



I was hired at Akamai to design the network architecture for a global backbone. 
 This is proving to be an interesting challenge taking a diverse set of 
products with various requirements and interconnecting them in a way that saves 
costs and improves performance while my employers traffic continue to grow.





Akamai is built to use the paths available to deliver traffic and meet our 
customers and our business goals.  Not all our sites are interconnected and 
it’s extremely unlikely (read: possibly never, but who knows) you will see all 
your traffic come over a direct link or cache.  With any sufficiently complex 
system, plus the acquisitions we have made over my short tenure it’s almost 
impractical to integrate them all quickly or possibly at all.

I personally want to make sure that we deliver the traffic in a way that makes 
sense, and a few people have seen those efforts but there’s also many things in 
progress that are not yet complete or ready for public consumption.  I believe 
there’s room here to improve and each time we can turn a switch or dial a knob 
to better serve our customers and the end-users that we are paid to serve, 
everyone wins.


Enterprises vs consumer ISPs have very different traffic profiles, and I think 
the genesis of this thread was a direct result of a very consumer oriented 
traffic profile that was unexpected.  People have wondered why I would spend so 
much time watching things like Apple rumor websites in the past, it’s because 
that would lead to high traffic events.  You go to where the data is.  The same 
can be said for other large download events or OTT launches.  Everyone knows a 
live event can be big but generally bound by the target audience size.

As software is attacked within minutes or hours after security patches are 
released, I don’t find it surprising these days that systems automatically 
download whatever they can the moment it’s released from gaming consoles to IoT 
and server and OS patches.

If the traffic is causing you pain, I encourage you to reach out so we can look 
at what might be improved.

- Jared
(I swear I’ll stop responding.. off to make lunch)

Re: Elephant in the room - Akamai

[Rod Beck]

Yep. Real estate must be one of their largest expenses and unlike bandwidth it 
is not going down to price. 


From: Owen DeLong 
Sent: Sunday, December 8, 2019 6:07 PM
To: Rod Beck 
Cc: Jared Mauch ; nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai

My guess (and it’s just this since I haven’t been inside Akamai for a couple of 
years now) is that they are culling the less effective AANPs (from Akamai’s 
perspective) in favor of redeploying the hardware to more effective locations 
and/or to eliminate the cost of supporting/refreshing said hardware.

I would guess that the traffic level required to justify the expense of 
maintaining an AANP (from Akamai’s perspective) probably depends on a great 
many factors not all of which would be obvious as viewed from the outside. I 
would guess that the density of AANPs and ISP interconnection in a given 
geography would be among the factors that would influence that number. I would 
also guess that the number would tend to rise over time.

Again, just external speculation on my part.

Owen


On Dec 8, 2019, at 06:39 , Rod Beck 
mailto:rod.b...@unitedcablecompany.com>> wrote:

Taking boxes out of a network does not sound like 'emergent behavior' or 
unintended consequences. Sounds like a policy change. Perhaps they are being 
redeployed for better performance or perhaps shut down to lower costs. Or may 
be the cost of transit for Akamai at the margin is less than the cost of 
peering with 50 billion peers.

Disclaimer: Not picking a fight. Better things to do.

Regards,

Roderick.


From: Jared Mauch mailto:ja...@puck.nether.net>>
Sent: Sunday, December 8, 2019 1:19 AM
To: Rod Beck 
mailto:rod.b...@unitedcablecompany.com>>
Cc: Shawn L mailto:sha...@up.net>>; 
nanog@nanog.org<mailto:nanog@nanog.org> 
mailto:nanog@nanog.org>>
Subject: Re: Elephant in the room - Akamai

On Dec 7, 2019, at 5:34 PM, Rod Beck 
mailto:rod.b...@unitedcablecompany.com>> wrote:
>
> Have there been any fundamental change in their network architecture that 
> might explain pulling these caches?


Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)

This is why I’m encouraging people to ping me off-list, because the dynamics at 
play for one provider don’t match across the board.  I know we have thousands 
of distinct sites that each have their own attributes and composition at play.

I’ve been working hard to provide value to our AANP partners as well.  I’ll try 
to stop responding to the list at this point but don’t hesitate to contact me 
here or via other means if you’re seeing something weird.  I know I resolved a 
problem a few days ago for someone quickly as there was a misconfiguration left 
around.. We all make mistakes and can all do better.

- jared

https://www.peeringdb.com/asn/20940

Re: Elephant in the room - Akamai

[Jared Mauch]

> On Dec 8, 2019, at 11:48 AM, Brandon Martin  wrote:
> 
> I guess what I'm getting at is that it sounds like, if you cannot source the 
> content locally to the peering link, there's not likely to be an internal 
> connection to the same site from somewhere else within the Akamai network to 
> deliver that content and, instead, the target network should expect it to 
> come in over the "public Internet" via some other connection.  Is that 
> accurate?
> 
> Thanks for the clarifications.



I was hired at Akamai to design the network architecture for a global backbone. 
 This is proving to be an interesting challenge taking a diverse set of 
products with various requirements and interconnecting them in a way that saves 
costs and improves performance while my employers traffic continue to grow.





Akamai is built to use the paths available to deliver traffic and meet our 
customers and our business goals.  Not all our sites are interconnected and 
it’s extremely unlikely (read: possibly never, but who knows) you will see all 
your traffic come over a direct link or cache.  With any sufficiently complex 
system, plus the acquisitions we have made over my short tenure it’s almost 
impractical to integrate them all quickly or possibly at all.

I personally want to make sure that we deliver the traffic in a way that makes 
sense, and a few people have seen those efforts but there’s also many things in 
progress that are not yet complete or ready for public consumption.  I believe 
there’s room here to improve and each time we can turn a switch or dial a knob 
to better serve our customers and the end-users that we are paid to serve, 
everyone wins.  


Enterprises vs consumer ISPs have very different traffic profiles, and I think 
the genesis of this thread was a direct result of a very consumer oriented 
traffic profile that was unexpected.  People have wondered why I would spend so 
much time watching things like Apple rumor websites in the past, it’s because 
that would lead to high traffic events.  You go to where the data is.  The same 
can be said for other large download events or OTT launches.  Everyone knows a 
live event can be big but generally bound by the target audience size.  

As software is attacked within minutes or hours after security patches are 
released, I don’t find it surprising these days that systems automatically 
download whatever they can the moment it’s released from gaming consoles to IoT 
and server and OS patches.

If the traffic is causing you pain, I encourage you to reach out so we can look 
at what might be improved.

- Jared
(I swear I’ll stop responding.. off to make lunch)

Re: Elephant in the room - Akamai

[Owen DeLong]

My guess (and it’s just this since I haven’t been inside Akamai for a couple of 
years now) is that they are culling the less effective AANPs (from Akamai’s 
perspective) in favor of redeploying the hardware to more effective locations 
and/or to eliminate the cost of supporting/refreshing said hardware.

I would guess that the traffic level required to justify the expense of 
maintaining an AANP (from Akamai’s perspective) probably depends on a great 
many factors not all of which would be obvious as viewed from the outside. I 
would guess that the density of AANPs and ISP interconnection in a given 
geography would be among the factors that would influence that number. I would 
also guess that the number would tend to rise over time.

Again, just external speculation on my part.

Owen


> On Dec 8, 2019, at 06:39 , Rod Beck  wrote:
> 
> Taking boxes out of a network does not sound like 'emergent behavior' or 
> unintended consequences. Sounds like a policy change. Perhaps they are being 
> redeployed for better performance or perhaps shut down to lower costs. Or may 
> be the cost of transit for Akamai at the margin is less than the cost of 
> peering with 50 billion peers. 
> 
> Disclaimer: Not picking a fight. Better things to do. 
> 
> Regards, 
> 
> Roderick. 
> 
> From: Jared Mauch 
> Sent: Sunday, December 8, 2019 1:19 AM
> To: Rod Beck 
> Cc: Shawn L ; nanog@nanog.org 
> Subject: Re: Elephant in the room - Akamai
>  
> On Dec 7, 2019, at 5:34 PM, Rod Beck  wrote:
> > 
> > Have there been any fundamental change in their network architecture that 
> > might explain pulling these caches?
> 
> 
> Please see my email on Friday where I outlined a few of the dynamics at play. 
>  Akamai isn’t just one thing, it’s an entire basket of products that all have 
> their own resulting behaviors.  This is why even though you may peer with us 
> directly you may not see 100% of the traffic from that interconnection.  
> (Take SSL for example, it’s often not served via the clusters in an ISP due 
> to the security requirements we place on those racks, and this is something 
> we treat very seriously!)
> 
> This is why I’m encouraging people to ping me off-list, because the dynamics 
> at play for one provider don’t match across the board.  I know we have 
> thousands of distinct sites that each have their own attributes and 
> composition at play.
> 
> I’ve been working hard to provide value to our AANP partners as well.  I’ll 
> try to stop responding to the list at this point but don’t hesitate to 
> contact me here or via other means if you’re seeing something weird.  I know 
> I resolved a problem a few days ago for someone quickly as there was a 
> misconfiguration left around.. We all make mistakes and can all do better.
> 
> - jared
> 
> https://www.peeringdb.com/asn/20940 <https://www.peeringdb.com/asn/20940>

Re: Elephant in the room - Akamai

[Brandon Martin]

On 12/8/19 10:58 AM, Jared Mauch wrote:

Not all content is suitable in all locations based on the physical security or 
market situation. We have some content that can not be served, an example is 
locations where there are licensing requirements (eg: ICP for China).

You will see a different mix from our 20940 vs 16625 as well. Those have 
different requirements on the security side. If you treat your PKI data 
seriously you will appreciate what is done here.

In Marquette Michigan there will be different opportunities compared with 
Amsterdam or Ashburn as well.

Our customers and traffic mix makes it challenging to serve from a platform 
where you do capital planning for several year depreciation cycle. We have 
thousands of unique sites and that scale is quite different from serving on a 
few distinct IXPs and transit providers.

So yes you will see a difference and there are things we can do to improve it 
when there is a variance in the behavior.
I guess what I'm getting at is that it sounds like, if you cannot source 
the content locally to the peering link, there's not likely to be an 
internal connection to the same site from somewhere else within the 
Akamai network to deliver that content and, instead, the target network 
should expect it to come in over the "public Internet" via some other 
connection.  Is that accurate?


Thanks for the clarifications.
--
Brandon Martin

Re: Elephant in the room - Akamai

[Mehmet Akcin]

Let's take a minute and thank Jared for taking the time and responding.

thank you, Jared.

On Sun, Dec 8, 2019 at 10:58 AM Jared Mauch  wrote:

> Not all content is suitable in all locations based on the physical
> security or market situation. We have some content that can not be served,
> an example is locations where there are licensing requirements (eg: ICP for
> China).
>
> You will see a different mix from our 20940 vs 16625 as well. Those have
> different requirements on the security side. If you treat your PKI data
> seriously you will appreciate what is done here.
>
> In Marquette Michigan there will be different opportunities compared with
> Amsterdam or Ashburn as well.
>
> Our customers and traffic mix makes it challenging to serve from a
> platform where you do capital planning for several year depreciation cycle.
> We have thousands of unique sites and that scale is quite different from
> serving on a few distinct IXPs and transit providers.
>
> So yes you will see a difference and there are things we can do to improve
> it when there is a variance in the behavior.
>
> - Jared
>
> > On Dec 8, 2019, at 10:33 AM, Brandon Martin 
> wrote:
> >
> > On 12/7/19 7:19 PM, Jared Mauch wrote:
> >> Please see my email on Friday where I outlined a few of the dynamics at
> play.  Akamai isn’t just one thing, it’s an entire basket of products that
> all have their own resulting behaviors.  This is why even though you may
> peer with us directly you may not see 100% of the traffic from that
> interconnection.  (Take SSL for example, it’s often not served via the
> clusters in an ISP due to the security requirements we place on those
> racks, and this is something we treat very seriously!)
> >
> > Does this mean that, if you peer with Akamai at some location, only
> content locally available at that location will come over that peering
> session with the rest coming via other means?  Does Akamai not have private
> connectivity to their public peering points?
> > --
> > Brandon Martin
>

Re: Elephant in the room - Akamai

[Mark Delany]

> Have there been any fundamental change in their network architecture
> that might explain pulling these caches?

Maybe not network architecture, but what if the cache-to-content ratio
is dropping dramatically due to changes in consumer behavior and/or a
huge increase in the underlying content (such as adoption of higher
and multiple-resolution videos)?

There has to be a tipping point at which a proportionally small cache
becomes almost worthless from a traffic saving perspective.

If you run a cluster one presumes you can see what your in/out ratio
looks like and where the trend-line is headed.


Another possibility might be security. It may be that they need
additional security credentials for newer services which they are
reluctant to load into remote cache clusters they don't physically
control.


Mark.

Re: Elephant in the room - Akamai

[Jared Mauch]

Not all content is suitable in all locations based on the physical security or 
market situation. We have some content that can not be served, an example is 
locations where there are licensing requirements (eg: ICP for China). 

You will see a different mix from our 20940 vs 16625 as well. Those have 
different requirements on the security side. If you treat your PKI data 
seriously you will appreciate what is done here. 

In Marquette Michigan there will be different opportunities compared with 
Amsterdam or Ashburn as well. 

Our customers and traffic mix makes it challenging to serve from a platform 
where you do capital planning for several year depreciation cycle. We have 
thousands of unique sites and that scale is quite different from serving on a 
few distinct IXPs and transit providers. 

So yes you will see a difference and there are things we can do to improve it 
when there is a variance in the behavior. 

- Jared 

> On Dec 8, 2019, at 10:33 AM, Brandon Martin  wrote:
> 
> On 12/7/19 7:19 PM, Jared Mauch wrote:
>> Please see my email on Friday where I outlined a few of the dynamics at 
>> play.  Akamai isn’t just one thing, it’s an entire basket of products that 
>> all have their own resulting behaviors.  This is why even though you may 
>> peer with us directly you may not see 100% of the traffic from that 
>> interconnection.  (Take SSL for example, it’s often not served via the 
>> clusters in an ISP due to the security requirements we place on those racks, 
>> and this is something we treat very seriously!)
> 
> Does this mean that, if you peer with Akamai at some location, only content 
> locally available at that location will come over that peering session with 
> the rest coming via other means?  Does Akamai not have private connectivity 
> to their public peering points?
> -- 
> Brandon Martin

Re: Elephant in the room - Akamai

[Brandon Martin]

On 12/7/19 7:19 PM, Jared Mauch wrote:

Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)


Does this mean that, if you peer with Akamai at some location, only 
content locally available at that location will come over that peering 
session with the rest coming via other means?  Does Akamai not have 
private connectivity to their public peering points?

--
Brandon Martin

Re: Elephant in the room - Akamai

[Ben Cannon]

+100, and thanks to Jared.

-Ben

> On Dec 7, 2019, at 10:08 PM, Mark Tinka  wrote:
> 
> 
> 
>> On 8/Dec/19 02:19, Jared Mauch wrote:
>> 
>> I’ve been working hard to provide value to our AANP partners as well.  I’ll 
>> try to stop responding to the list at this point but don’t hesitate to 
>> contact me here or via other means if you’re seeing something weird.  I know 
>> I resolved a problem a few days ago for someone quickly as there was a 
>> misconfiguration left around.. We all make mistakes and can all do better.
> 
> Problems are part of the gig - otherwise we'd have no reason to get up
> in the morning.
> 
> What matters is that there is someone you can find to help you fix them.
> That's what makes all the difference.
> 
> So kudos to you, Jared, and the entire team out there at Akamai.
> 
> Mark.

Re: Elephant in the room - Akamai

[Rod Beck]

Taking boxes out of a network does not sound like 'emergent behavior' or 
unintended consequences. Sounds like a policy change. Perhaps they are being 
redeployed for better performance or perhaps shut down to lower costs. Or may 
be the cost of transit for Akamai at the margin is less than the cost of 
peering with 50 billion peers.

Disclaimer: Not picking a fight. Better things to do.

Regards,

Roderick.


From: Jared Mauch 
Sent: Sunday, December 8, 2019 1:19 AM
To: Rod Beck 
Cc: Shawn L ; nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai

On Dec 7, 2019, at 5:34 PM, Rod Beck  wrote:
>
> Have there been any fundamental change in their network architecture that 
> might explain pulling these caches?


Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)

This is why I’m encouraging people to ping me off-list, because the dynamics at 
play for one provider don’t match across the board.  I know we have thousands 
of distinct sites that each have their own attributes and composition at play.

I’ve been working hard to provide value to our AANP partners as well.  I’ll try 
to stop responding to the list at this point but don’t hesitate to contact me 
here or via other means if you’re seeing something weird.  I know I resolved a 
problem a few days ago for someone quickly as there was a misconfiguration left 
around.. We all make mistakes and can all do better.

- jared

https://www.peeringdb.com/asn/20940

Re: Elephant in the room - Akamai

[Mark Tinka]

On 8/Dec/19 02:19, Jared Mauch wrote:

> I’ve been working hard to provide value to our AANP partners as well.  I’ll 
> try to stop responding to the list at this point but don’t hesitate to 
> contact me here or via other means if you’re seeing something weird.  I know 
> I resolved a problem a few days ago for someone quickly as there was a 
> misconfiguration left around.. We all make mistakes and can all do better.

Problems are part of the gig - otherwise we'd have no reason to get up
in the morning.

What matters is that there is someone you can find to help you fix them.
That's what makes all the difference.

So kudos to you, Jared, and the entire team out there at Akamai.

Mark.

Re: Elephant in the room - Akamai

[Jared Mauch]

On Dec 7, 2019, at 5:34 PM, Rod Beck  wrote:
> 
> Have there been any fundamental change in their network architecture that 
> might explain pulling these caches?


Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)

This is why I’m encouraging people to ping me off-list, because the dynamics at 
play for one provider don’t match across the board.  I know we have thousands 
of distinct sites that each have their own attributes and composition at play.

I’ve been working hard to provide value to our AANP partners as well.  I’ll try 
to stop responding to the list at this point but don’t hesitate to contact me 
here or via other means if you’re seeing something weird.  I know I resolved a 
problem a few days ago for someone quickly as there was a misconfiguration left 
around.. We all make mistakes and can all do better.

- jared

https://www.peeringdb.com/asn/20940

Re: Elephant in the room - Akamai

[Mark Tinka]

On 6/Dec/19 21:29, Keenan Tims wrote:

>
> We're currently seeing about 80% of our AS20940 origin traffic coming
> from transit, and I'm certain there's a significant additional amount
> which is difficult to identify coming from on-net caches at our
> upstream providers (though it appears from the thread that may be
> reducing as well). Only about 20% is coming from peering where we have
> significantly more capacity and lower costs. Whatever the algorithm is
> doing, from my perspective it doesn't make a lot of sense and is
> pretty frustrating, and I'm somewhat concerned about busting commits
> and possibly running into congestion for the next big event that does
> hit us, which would not be a problem if it were delivered over peering.

We've had 2 or 3 customers, in the last 3 months, complain about the
same thing - where they are seeing Akamai traffic drop over peering but
preferred via their transit service with us. We run a number of Akamai
AANP caches across our backbone.

We are working very closely with Akamai - and the customers - to resolve
this, I'll add.

Mark.

Re: Elephant in the room - Akamai

[Rod Beck]

Have there been any fundamental change in their network architecture that might 
explain pulling these caches?


From: NANOG  on behalf of Shawn L via NANOG 

Sent: Saturday, December 7, 2019 8:20 PM
To: Jared Mauch 
Cc: nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai


Same -- we had an Akamai cache for 15+ years.  Then we were notified that it 
was done and were sent boxes to pack our stuff up and send it back.





-Original Message-
From: "Jared Mauch" 
Sent: Saturday, December 7, 2019 2:05pm
To: "Seth Mattinen" 
Cc: nanog@nanog.org
Subject: Re: Elephant in the room - Akamai



> On Dec 7, 2019, at 12:06 PM, Seth Mattinen  wrote:
>
> On 12/6/19 06:46, Fawcett, Nick via NANOG wrote:
>> We had three onsite Akamai caches a few months ago. They called us up and 
>> said they are removing that service and sent us boxes to pack up the 
>> hardware and ship back. We’ve had quite the increase in DIA traffic as a 
>> result of it.
>
>
> Same here, removed last month, and no more Akamai traffic over peering since.

This last part doesn’t sound right.

Can you send me details in private?

Thanks,

- Jared

Re: Elephant in the room - Akamai

[Jared Mauch]

> On Dec 7, 2019, at 3:01 PM, Eric Kuhnke  wrote:
> 
> I think this thread might be a perfect example that when an organization 
> reaches a sufficiently large size, one part of its engineering/operations 
> team may no longer be fully aware of what other work groups are doing. 
> Definitely a structural challenge for ISPs that span very large geographical 
> areas and services/roles. 

We are a decent sized (public) company.  You can look at the # of employees if 
you are curious.  I am but one person who can try to influence things.  I’ll 
say that if you have a few servers from us, it’s not going to serve the entire 
content set that our customers have.

I do remain open to look at your individual cases and see what can be done to 
improve though.  The answer may be nothing, but an e-mail also costs you 
little.  I’ve got several people I’m corresponding with and will continue to do 
so at least until the paychecks dry up.  Given the number of questions I still 
field about my prior employer, it may even last beyond that point :-)

- Jared

Re: Elephant in the room - Akamai

[Eric Kuhnke]

I think this thread might be a perfect example that when an organization
reaches a sufficiently large size, one part of its engineering/operations
team may no longer be fully aware of what other work groups are doing.
Definitely a structural challenge for ISPs that span very large
geographical areas and services/roles.





On Sat, Dec 7, 2019 at 11:06 AM Jared Mauch  wrote:

>
>
> > On Dec 7, 2019, at 12:06 PM, Seth Mattinen  wrote:
> >
> > On 12/6/19 06:46, Fawcett, Nick via NANOG wrote:
> >> We had three onsite Akamai caches a few months ago.  They called us up
> and said they are removing that service and sent us boxes to pack up the
> hardware and ship back.  We’ve had quite the increase in DIA traffic as a
> result of it.
> >
> >
> > Same here, removed last month, and no more Akamai traffic over peering
> since.
>
> This last part doesn’t sound right.
>
> Can you send me details in private?
>
> Thanks,
>
> - Jared

Re: Elephant in the room - Akamai

[Shawn L via NANOG]

Same -- we had an Akamai cache for 15+ years.  Then we were notified that it 
was done and were sent boxes to pack our stuff up and send it back.
 
 
-Original Message-
From: "Jared Mauch" 
Sent: Saturday, December 7, 2019 2:05pm
To: "Seth Mattinen" 
Cc: nanog@nanog.org
Subject: Re: Elephant in the room - Akamai




> On Dec 7, 2019, at 12:06 PM, Seth Mattinen  wrote:
> 
> On 12/6/19 06:46, Fawcett, Nick via NANOG wrote:
>> We had three onsite Akamai caches a few months ago. They called us up and 
>> said they are removing that service and sent us boxes to pack up the 
>> hardware and ship back. We’ve had quite the increase in DIA traffic as a 
>> result of it.
> 
> 
> Same here, removed last month, and no more Akamai traffic over peering since.

This last part doesn’t sound right.

Can you send me details in private?

Thanks,

- Jared

Re: Elephant in the room - Akamai

[Jared Mauch]

> On Dec 7, 2019, at 12:06 PM, Seth Mattinen  wrote:
> 
> On 12/6/19 06:46, Fawcett, Nick via NANOG wrote:
>> We had three onsite Akamai caches a few months ago.  They called us up and 
>> said they are removing that service and sent us boxes to pack up the 
>> hardware and ship back.  We’ve had quite the increase in DIA traffic as a 
>> result of it.
> 
> 
> Same here, removed last month, and no more Akamai traffic over peering since.

This last part doesn’t sound right.

Can you send me details in private?

Thanks,

- Jared

Re: Elephant in the room - Akamai

[Seth Mattinen]

On 12/6/19 06:46, Fawcett, Nick via NANOG wrote:
We had three onsite Akamai caches a few months ago.  They called us up 
and said they are removing that service and sent us boxes to pack up the 
hardware and ship back.  We’ve had quite the increase in DIA traffic as 
a result of it.





Same here, removed last month, and no more Akamai traffic over peering 
since.

Re: Elephant in the room - Akamai

[Keenan Tims]

Speaking as a (very) small operator, we've also been seeing less and 
less of our Akamai traffic coming to us over peering over the last 
couple years. I've reached out to Akamai NOC as well as Jared directly 
on a few occasions and while they've been helpful and their changes 
usually have some short-term impact, the balance has always shifted back 
some weeks/months later. I've more or less resigned myself to this being 
how Akamai wants things, and as we so often have to as small fish, just 
dealing with it.


We're currently seeing about 80% of our AS20940 origin traffic coming 
from transit, and I'm certain there's a significant additional amount 
which is difficult to identify coming from on-net caches at our upstream 
providers (though it appears from the thread that may be reducing as 
well). Only about 20% is coming from peering where we have significantly 
more capacity and lower costs. Whatever the algorithm is doing, from my 
perspective it doesn't make a lot of sense and is pretty frustrating, 
and I'm somewhat concerned about busting commits and possibly running 
into congestion for the next big event that does hit us, which would not 
be a problem if it were delivered over peering.


Luckily we're business focussed, so we're not getting hit by these 
gaming events.


Keenan Tims
Stargate Connections Inc (AS19171)

On 2019-12-06 8:13 a.m., Jared Mauch wrote:



On Dec 6, 2019, at 9:59 AM, Chris Adams  wrote:

Once upon a time, Fawcett, Nick  said:

We had three onsite Akamai caches a few months ago.  They called us up and said 
they are removing that service and sent us boxes to pack up the hardware and 
ship back.  We’ve had quite the increase in DIA traffic as a result of it.

Same here.  We'd had Akamai servers for many years, replaced as needed
(including one failed servre replaced right before they turned them
off).  Now about 50% of our Akamai traffic comes across transit links,
not peering.  This seems like it would be rather inefficient for them
too…

There’s an element of scale when it comes to certain content that makes it not 
viable if the majority of traffic is VOD with variable bitrates it requires a 
lot more capital.

Things like downloads of software updates (eg: patch Tuesday) lend themselves 
to different optimizations.  The hardware has a cost as well as the bandwidth 
as well.

I’ll say that most places that have a few servers may only see a minor 
improvement in their in:out.  If you’re not peering with us or are and see 
significant traffic via transit, please do reach out.

I’m happy to discuss in private or at any NANOG/IETF meeting people are at.  We 
generally have someone at most of the other NOG meetings as well, including 
RIPE, APRICOT and even GPF etc.

I am personally always looking for better ways to serve the medium (or small) 
size providers better.

- Jared

Re: Elephant in the room - Akamai

[Michael Thomas]

On 12/5/19 6:02 PM, Valdis Klētnieks wrote:

On Thu, 05 Dec 2019 14:18:07 -0800, Michael Thomas said:


My suspicion is that the root problem was buffer bloat -- i flashed a
new router with openwrt and was a little dismayed that the bufferbloat
code is a plugin you have to enable. The buffer bloat got a lot better

Friends don't let friends run factory firmware. :)

Hopefully sometime soon the SQM stuff will be added to the default  openwrt
configs for most of the supported routers, if it hasn't been already. It's been
in my config since before the Luci support for SQM got created

The big problem is that a lot of eyeball networks have a lot of CPE boxes that
were created before the bufferbloat work was done, and often have no real
motivation to push software updates to the CPE (if they even have the ability),
and a lot of customers have routers that they bought at Best Buy or Walmart
that will *never* get a software update.

(I also admit having no idea what percentage of the intermediate routers in the
ISP's networks have gotten de-bloating code.



So I tested this out again after I sent out my message and it does 
indeed seem to be just fine: it wasn't an identical test since my friend 
was over wifi, but that really shouldn't affect things, I'd think.


The thing I don't get is that buffer bloat is a creature of the 
upstream, right? I wouldn't think that the stream of acks sent from 
downloading the file would put much pressure on the upstream. Which 
makes me wonder if it's just that the old router itself was saturated 
and couldn't keep up. Or something.


In any case, there are probably zillions of 10 year old routers out in 
the world, and no matter what exactly caused this for me it will 
probably happen for zillions of other people too. Hope support desks are 
ready for the deluge.


Mike

Re: Elephant in the room - Akamai

[Jared Mauch]

> On Dec 6, 2019, at 9:59 AM, Chris Adams  wrote:
> 
> Once upon a time, Fawcett, Nick  said:
>> We had three onsite Akamai caches a few months ago.  They called us up and 
>> said they are removing that service and sent us boxes to pack up the 
>> hardware and ship back.  We’ve had quite the increase in DIA traffic as a 
>> result of it.
> 
> Same here.  We'd had Akamai servers for many years, replaced as needed
> (including one failed servre replaced right before they turned them
> off).  Now about 50% of our Akamai traffic comes across transit links,
> not peering.  This seems like it would be rather inefficient for them
> too…

There’s an element of scale when it comes to certain content that makes it not 
viable if the majority of traffic is VOD with variable bitrates it requires a 
lot more capital.  

Things like downloads of software updates (eg: patch Tuesday) lend themselves 
to different optimizations.  The hardware has a cost as well as the bandwidth 
as well.

I’ll say that most places that have a few servers may only see a minor 
improvement in their in:out.  If you’re not peering with us or are and see 
significant traffic via transit, please do reach out.

I’m happy to discuss in private or at any NANOG/IETF meeting people are at.  We 
generally have someone at most of the other NOG meetings as well, including 
RIPE, APRICOT and even GPF etc.

I am personally always looking for better ways to serve the medium (or small) 
size providers better.

- Jared

Re: Elephant in the room - Akamai

[Chris Adams]

Once upon a time, Fawcett, Nick  said:
> We had three onsite Akamai caches a few months ago.  They called us up and 
> said they are removing that service and sent us boxes to pack up the hardware 
> and ship back.  We’ve had quite the increase in DIA traffic as a result of it.

Same here.  We'd had Akamai servers for many years, replaced as needed
(including one failed servre replaced right before they turned them
off).  Now about 50% of our Akamai traffic comes across transit links,
not peering.  This seems like it would be rather inefficient for them
too...

-- 
Chris Adams 

RE: Elephant in the room - Akamai

[Fawcett, Nick via NANOG]

We had three onsite Akamai caches a few months ago.  They called us up and said 
they are removing that service and sent us boxes to pack up the hardware and 
ship back.  We’ve had quite the increase in DIA traffic as a result of it.

~Nick

From: NANOG  On Behalf Of Kaiser, Erich
Sent: Wednesday, December 4, 2019 9:03 PM
To: NANOG list 
Subject: Elephant in the room - Akamai

Lets talk Akamai

They have shifted 90% of their traffic off IXs and onto our full route DIA, 
anyone else seeing this issue or have insight as to what is going on over 
there?  We have been asking for help on resolution for weeks and all we get is 
we are working on it and now we get no response.  We were even sent an LOA and 
when the DC went to go put in the x-connect their patch panel was full.  How do 
they not know if they have ports open or not?  I have even reached out to an 
engineer who is on this list and he does not even respond.

The last two nights the traffic levels to them has skyrocketed as well.

Any insight?


Erich Kaiser
The Fusion Network



--

Checked by SOPHOS http://www.sophos.com

-- 
Checked by SOPHOS http://www.sophos.com

Re: Elephant in the room - Akamai

[Stephen Satchell]

On 12/5/19 6:02 PM, Valdis Klētnieks wrote:

(I also admit having no idea what percentage of the intermediate routers in the
ISP's networks have gotten de-bloating code.


For SP-grade routers, there isn't "code" that needs to be added to 
combat buffer bloat.  All an admin has to do is cut back on the number 
of packet buffers on each interface -- an interface setting, you see.


The reason that comsumer-grade devices can contribute to buffer bloat is 
because the vendor doesn't expose a knob to adjust buffering.  At least 
in most instances with Best Buy and Office Depot routers.

Re: Elephant in the room - Akamai

[Valdis Klētnieks]

On Thu, 05 Dec 2019 14:18:07 -0800, Michael Thomas said:

> My suspicion is that the root problem was buffer bloat -- i flashed a
> new router with openwrt and was a little dismayed that the bufferbloat
> code is a plugin you have to enable. The buffer bloat got a lot better

Friends don't let friends run factory firmware. :)

Hopefully sometime soon the SQM stuff will be added to the default  openwrt
configs for most of the supported routers, if it hasn't been already. It's been
in my config since before the Luci support for SQM got created

The big problem is that a lot of eyeball networks have a lot of CPE boxes that
were created before the bufferbloat work was done, and often have no real
motivation to push software updates to the CPE (if they even have the ability),
and a lot of customers have routers that they bought at Best Buy or Walmart
that will *never* get a software update.

(I also admit having no idea what percentage of the intermediate routers in the
ISP's networks have gotten de-bloating code.



pgpipUcFxDede.pgp
Description: PGP signature

Re: Elephant in the room - Akamai

[Chris Adams]

Once upon a time, Valdis Klētnieks  said:
> And it's only going to get worse.  Sony has already announced that the
> Playstation 5 will have a (probably) 1-2 terabyte SSD.  And even with that, 
> the
> game packaging is set up to support only downloading the single-player or
> multi-player portions of a game because images are going to be pushing 100
> gigabytes RSN (some are already well over 40gig).

Xbox One X games are already there... I'm a pretty casual gamer, and I
have multiple games over 90GB (one is 117GB).

-- 
Chris Adams 

Re: Elephant in the room - Akamai

[Michael Thomas]

On 12/5/19 1:44 PM, Valdis Klētnieks wrote:

On Thu, 05 Dec 2019 14:41:30 -0600, "Aaron Gould" said:


Tarko. wow, gaming again !  It's not going away. gaming traffic is growing
in a big way it seems.

And it's only going to get worse.  Sony has already announced that the
Playstation 5 will have a (probably) 1-2 terabyte SSD.  And even with that, the
game packaging is set up to support only downloading the single-player or
multi-player portions of a game because images are going to be pushing 100
gigabytes RSN (some are already well over 40gig).

So even with the download restructuring, we're probably going to be seeing a
lot of people downloading lots of gigabytes on Day 1 (or a few days before, for
games that support it), and re-downloading smaller (but still large) amounts
when they want to re-play the game...



I suspect that it's going to be even worse on the home side. A while ago 
a friend was here and unbeknownst to me, he was downloading a big game. 
The rest of the home network was rendered unusable, and it took me over 
an hour to figure out what was going on. I knew what to look for -- and 
even then giving the awful tools that routers support it was hard -- but 
just about anybody else would have been on the phone to their provider 
saying that "INTERTOOBS ARE SLOW!".


My suspicion is that the root problem was buffer bloat -- i flashed a 
new router with openwrt and was a little dismayed that the bufferbloat 
code is a plugin you have to enable. The buffer bloat got a lot better 
after that, but I forgot to retest the downloading after so I'm not 100% 
positive. But if it was the problem, we're probably in for a world of 
hurt as I doubt that many home routers implement it.


Mike

Re: Elephant in the room - Akamai

[Valdis Klētnieks]

On Thu, 05 Dec 2019 14:41:30 -0600, "Aaron Gould" said:

> Tarko. wow, gaming again !  It's not going away. gaming traffic is growing
> in a big way it seems.

And it's only going to get worse.  Sony has already announced that the
Playstation 5 will have a (probably) 1-2 terabyte SSD.  And even with that, the
game packaging is set up to support only downloading the single-player or
multi-player portions of a game because images are going to be pushing 100
gigabytes RSN (some are already well over 40gig).

So even with the download restructuring, we're probably going to be seeing a
lot of people downloading lots of gigabytes on Day 1 (or a few days before, for
games that support it), and re-downloading smaller (but still large) amounts
when they want to re-play the game...



pgpZd9dnF7KSd.pgp
Description: PGP signature

RE: Elephant in the room - Akamai

[Aaron Gould]

Tarko. wow, gaming again !  It's not going away. gaming traffic is growing
in a big way it seems.

 

Clayton.. My thoughts exactly!  I too have wondered how valuable these
aanp's were, but lately I'm seeing good efficiency

 

Thanks y'all

 

-Aaron

 

 

RE: Elephant in the room - Akamai

[Clayton Zekelman]

Our AANP cache seems to have done the same in the 
past 2 nights. Lots of traffic that has never 
been there before.  It has however not reduced 
the amount of traffic we're getting from AS20940 
directly - still hit a new record level last night.


We've got a request in with them for a PNI.   If 
things keep growing at this rate, we might need two!


Over the years, I've questioned how much the AANP 
boxes really did for us, as their in:out ratio 
seemed almost balanced.  If the last two nights 
are an indication, then they're worth keeping.




At 09:39 AM 05/12/2019, Aaron Gould wrote:
I see my Akamai aanp cache utilization at 
all-time highs the last 2 nights as well.  Curious what it is.


Jared, you can reply to my off-list if you wish, 
or on-list if it would benefit the community.


Thanks,
Aaron



--

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409

Re: Elephant in the room - Akamai

[Tarko Tikan]

hey,

I see my Akamai aanp cache utilization at all-time highs the last 2 
nights as well.  Curious what it is.


Halo Reach release.

--
tarko

RE: Elephant in the room - Akamai

[Aaron Gould]

I see my Akamai aanp cache utilization at all-time highs the last 2 nights as 
well.  Curious what it is.

 

Jared, you can reply to my off-list if you wish, or on-list if it would benefit 
the community.

 

Thanks,

Aaron

 

Re: Elephant in the room - Akamai

[Kaiser, Erich]

Yes inbound.  The patterns are not typical, we are talking gigs of traffic
moved off of the IX side and onto our DIA side.  They have reached out to
me already.  Will see what happens.  Will post follow-up.


On Thu, Dec 5, 2019 at 3:15 AM Bryan Holloway  wrote:

>
> On 12/5/19 8:48 AM, Matthew Petach wrote:
> >
> >
> > On Wed, Dec 4, 2019, 19:05 Kaiser, Erich  > > wrote:
> >
> > Lets talk Akamai
> >
> >
> > [...]
> >
> >
> > The last two nights the traffic levels to them has skyrocketed as
> well.
> >
> > Any insight?
> >
> >
> > Erich Kaiser
> > The Fusion Network
> >
> >
> > As a CDN, I would usually expect to see traffic *from* Akamai to be the
> > large direction.
> >
> > If you're seeing your traffic *to* them skyrocketing, are you sure you
> > aren't carrying DDoS attack traffic at them?
> >
> > CDNs aren't known for being large traffic sinks.   ^_^;;
> >
> > Matt
>
>
> I think he meant inbound (from). We also saw the same thing.
>

Re: Elephant in the room - Akamai

[Jared Mauch]

Good morning!

If you are having Akamai issues you can reach out to me and I will help you 
out. 

- Jared 

> On Dec 4, 2019, at 10:04 PM, Kaiser, Erich  wrote:
> 
> 
> Lets talk Akamai
> 
> They have shifted 90% of their traffic off IXs and onto our full route DIA, 
> anyone else seeing this issue or have insight as to what is going on over 
> there?  We have been asking for help on resolution for weeks and all we get 
> is we are working on it and now we get no response.  We were even sent an LOA 
> and when the DC went to go put in the x-connect their patch panel was full.  
> How do they not know if they have ports open or not?  I have even reached out 
> to an engineer who is on this list and he does not even respond.  
> 
> The last two nights the traffic levels to them has skyrocketed as well.  
> 
> Any insight?
> 
> 
> Erich Kaiser
> The Fusion Network

Re: Elephant in the room - Akamai

[Bryan Holloway]

On 12/5/19 8:48 AM, Matthew Petach wrote:



On Wed, Dec 4, 2019, 19:05 Kaiser, Erich > wrote:


Lets talk Akamai


[...]


The last two nights the traffic levels to them has skyrocketed as well.

Any insight?


Erich Kaiser
The Fusion Network


As a CDN, I would usually expect to see traffic *from* Akamai to be the 
large direction.


If you're seeing your traffic *to* them skyrocketing, are you sure you 
aren't carrying DDoS attack traffic at them?


CDNs aren't known for being large traffic sinks.   ^_^;;

Matt



I think he meant inbound (from). We also saw the same thing.

Re: Elephant in the room - Akamai

[Matthew Petach]

On Wed, Dec 4, 2019, 19:05 Kaiser, Erich  wrote:

> Lets talk Akamai
>

[...]


> The last two nights the traffic levels to them has skyrocketed as well.
>
> Any insight?
>
>
> Erich Kaiser
> The Fusion Network
>

As a CDN, I would usually expect to see traffic *from* Akamai to be the
large direction.

If you're seeing your traffic *to* them skyrocketing, are you sure you
aren't carrying DDoS attack traffic at them?

CDNs aren't known for being large traffic sinks.   ^_^;;

Matt

Re: Elephant in the room - Akamai

[craig washington]

I don't have any insight but can confirm I am seeing the same thing. (Traffic 
shift back onto transit links)
They did tell me they were having some bandwidth issues and are working on it.
I am currently awaiting a direct PNI with them but haven't heard from them in 
some time.


From: NANOG  on behalf of Kaiser, Erich 

Sent: Thursday, December 5, 2019 3:03 AM
To: NANOG list 
Subject: Elephant in the room - Akamai

Lets talk Akamai

They have shifted 90% of their traffic off IXs and onto our full route DIA, 
anyone else seeing this issue or have insight as to what is going on over 
there?  We have been asking for help on resolution for weeks and all we get is 
we are working on it and now we get no response.  We were even sent an LOA and 
when the DC went to go put in the x-connect their patch panel was full.  How do 
they not know if they have ports open or not?  I have even reached out to an 
engineer who is on this list and he does not even respond.

The last two nights the traffic levels to them has skyrocketed as well.

Any insight?


Erich Kaiser
The Fusion Network

Elephant in the room - Akamai

[Kaiser, Erich]

Lets talk Akamai

They have shifted 90% of their traffic off IXs and onto our full route DIA,
anyone else seeing this issue or have insight as to what is going on over
there?  We have been asking for help on resolution for weeks and all we get
is we are working on it and now we get no response.  We were even sent an
LOA and when the DC went to go put in the x-connect their patch panel was
full.  How do they not know if they have ports open or not?  I have even
reached out to an engineer who is on this list and he does not even
respond.

The last two nights the traffic levels to them has skyrocketed as well.

Any insight?


Erich Kaiser
The Fusion Network