Re: Elephant in the room - Akamai

[Michael Thomas]

On 12/8/19 3:37 PM, Fred Baker wrote:


Sent from my iPad


On Dec 5, 2019, at 9:03 PM, Stephen Satchell  wrote:

For SP-grade routers, there isn't "code" that needs to be added to combat 
buffer bloat.  All an admin has to do is cut back on the number of packet buffers on each 
interface -- an interface setting, you see.

A common misconception, and disagrees with the research on the topic.

Let me describe this conceptually. Think of a file transfer (a streaming flow 
can be thought of in those terms, as can web pages etc) as four groups of 
packets - those that have been delivered correctly and therefore don’t affect 
the window or flow rate, those that have been delivered out of order and 
therefore reduce the window and might get retransmitted even though they need 
not be resent, those that are sitting in a queue somewhere and therefore add 
latency, and those that haven’t been transmitted yet. If I have a large number 
of sessions transiting an interface, each one is likely to have a packet or two 
near the head of the queue; after that, it tends to thin out, with the sessions 
with the largest windows having packets deep in the queue, and sessions with 
smaller windows not so much.

If you reduce the queue depth, it does reduce that deep-in-the-queue group - 
there is no storage deep in the queue to hold it. What it does, however, is 
increase any given packet’s probability of loss (loss being the extreme case of 
delay, and when you reduce delay unintelligently is the byproduct), and 
therefore the second category of packets - the ones that managed to get through 
after a packet was lost, and therefore arrived out of order and have some 
probability of being retransmitted and therefore delivered multiple times.

What AQM technologies attempt to do (we can argue about the relative degree of 
success in different technologies; I’m talking about them as a class) is 
identify sessions in that deep-in-the-queue category and cause them to 
temporarily reduce their windows - keeping most of their outstanding packets 
near the head of the queue. Reducing their windows has the effect of moving 
packets out of the network buffers (bufferbloat) and reordering queues in the 
receiving host to “hasn’t been sent yet” in the sending host.  That also 
reduces median latency, meaning that the sessions with reduced windows don’t 
generally “slow down” - they simply keep less of their data streams in the 
network with reduced median latency.



So are saying in effect that the receiving host is, essentially, 
managing the sending host's queue by modulating the receiver's window? 
That seems really weird to me, and probably means I've got it wrong. How 
would the receiving host know when and why it should change the window, 
unless of course loss or other measurable things? If everything is 
chugging away, the receiver doesn't have any idea that the sender is 
starving other sessions, right?


It just seems to me that this is a sending hosts queuing problem?

Mike

Re: Elephant in the room - Akamai

[Fred Baker]

Sent from my iPad

> On Dec 5, 2019, at 9:03 PM, Stephen Satchell  wrote:
> 
> For SP-grade routers, there isn't "code" that needs to be added to combat 
> buffer bloat.  All an admin has to do is cut back on the number of packet 
> buffers on each interface -- an interface setting, you see.

A common misconception, and disagrees with the research on the topic.

Let me describe this conceptually. Think of a file transfer (a streaming flow 
can be thought of in those terms, as can web pages etc) as four groups of 
packets - those that have been delivered correctly and therefore don’t affect 
the window or flow rate, those that have been delivered out of order and 
therefore reduce the window and might get retransmitted even though they need 
not be resent, those that are sitting in a queue somewhere and therefore add 
latency, and those that haven’t been transmitted yet. If I have a large number 
of sessions transiting an interface, each one is likely to have a packet or two 
near the head of the queue; after that, it tends to thin out, with the sessions 
with the largest windows having packets deep in the queue, and sessions with 
smaller windows not so much.

If you reduce the queue depth, it does reduce that deep-in-the-queue group - 
there is no storage deep in the queue to hold it. What it does, however, is 
increase any given packet’s probability of loss (loss being the extreme case of 
delay, and when you reduce delay unintelligently is the byproduct), and 
therefore the second category of packets - the ones that managed to get through 
after a packet was lost, and therefore arrived out of order and have some 
probability of being retransmitted and therefore delivered multiple times.

What AQM technologies attempt to do (we can argue about the relative degree of 
success in different technologies; I’m talking about them as a class) is 
identify sessions in that deep-in-the-queue category and cause them to 
temporarily reduce their windows - keeping most of their outstanding packets 
near the head of the queue. Reducing their windows has the effect of moving 
packets out of the network buffers (bufferbloat) and reordering queues in the 
receiving host to “hasn’t been sent yet” in the sending host.  That also 
reduces median latency, meaning that the sessions with reduced windows don’t 
generally “slow down” - they simply keep less of their data streams in the 
network with reduced median latency.

Spoofer Report for NANOG for Nov 2019

[CAIDA Spoofer Project]

In response to feedback from operational security communities,
CAIDA's source address validation measurement project
(https://spoofer.caida.org) is automatically generating monthly
reports of ASes originating prefixes in BGP for systems from which
we received packets with a spoofed source address.
We are publishing these reports to network and security operations
lists in order to ensure this information reaches operational
contacts in these ASes.

This report summarises tests conducted within usa, can.

Inferred improvements during Nov 2019:
   ASN Name   Fixed-By
  8047 GCI2019-11-20

Further information for the inferred remediation is available at:
https://spoofer.caida.org/remedy.php

Source Address Validation issues inferred during Nov 2019:
   ASN Name   First-Spoofed Last-Spoofed
   209 CENTURYLINK-US-LEGACY-QWEST   2016-08-16   2019-11-29
  6128 CABLE-NET-1   2016-09-03   2019-11-28
  7459 GRANDECOM-AS1 2016-09-26   2019-11-06
 20412 CLARITY-TELECOM   2016-09-30   2019-11-29
  6181 FUSE-NET  2016-10-10   2019-11-30
 25787 ROWE-NETWORKS 2016-10-21   2019-11-11
   174 COGENT-1742016-10-21   2019-11-26
 32440 LONI  2016-11-03   2019-11-29
 12083 WOW-INTERNET  2016-11-09   2019-11-30
 18451 LESNET2017-02-22   2019-11-05
   701 UUNET 2017-06-14   2019-11-23
 63296 AMARILLO-WIRELESS 2017-09-01   2019-11-26
   546 PARSONS-PGS-1 2017-11-20   2019-11-25
 11827 WSU   2018-05-07   2019-11-29
393564 SPOKANE   2018-06-05   2019-11-19
   225 VIRGINIA  2018-06-18   2019-11-20
 33452 RW2018-09-19   2019-11-27
 20448 VPNTRANET-LLC 2018-09-20   2019-11-11
 63275 RADIOWIRE 2019-02-07   2019-11-05
  8047 GCI   2019-04-11   2019-11-26
 10745 ARIN-ASH-CHA  2019-04-29   2019-11-15
 20381 CABLELABS-LV  2019-05-01   2019-11-01
 46300 HSC-WAP   2019-10-30   2019-11-28
 53424 PUREPAGES 2019-11-05   2019-11-05
  2886 EC-813-2886   2019-11-05   2019-11-29
 26827 EPBTELECOM2019-11-07   2019-11-07
 54646 HARDYNET  2019-11-10   2019-11-10
 11293 UCOP-ASN  2019-11-23   2019-11-23
 53641 TOMALESBAYLAN 2019-11-29   2019-11-29

Further information for these tests where we received spoofed
packets is available at:
https://spoofer.caida.org/recent_tests.php?country_include=usa,can&no_block=1

Please send any feedback or suggestions to spoofer-i...@caida.org

Re: Elephant in the room - Akamai

[Chris Adams]

Once upon a time, Brandon Martin  said:
> I guess what I'm getting at is that it sounds like, if you cannot
> source the content locally to the peering link, there's not likely
> to be an internal connection to the same site from somewhere else
> within the Akamai network to deliver that content and, instead, the
> target network should expect it to come in over the "public
> Internet" via some other connection.  Is that accurate?

I believe this is true of multiple content networks.  For example, we
peer with Amazon in a couple of locations, but a significant amount of
traffic frmo their AS comes across transit rather than peering.

In old terms, this is "hot potato" routing - where the source gets the
traffic out of their network as soon as possible, rather than spend
internal resources to carry it as close to the destination as they can.
-- 
Chris Adams 

Re: Elephant in the room - Akamai

[Niels Bakker]

* rod.b...@unitedcablecompany.com (Rod Beck) [Sun 08 Dec 2019, 18:18 CET]:

Last time I spoke with an Akamai engineer many years ago the network was purely 
transit. Is that evolving?


https://conference.apnic.net/data/41/ix_100-akamai-apricot2016-23feb2016_1456157526.pdf

Per those slides, PAIX in 2000, LINX, DE-CIX, AMS-IX in 2001, JPIX in 
2002, so you must have spoken with an engineer shortly after the 
company was founded in 1998.



-- Niels.

Re: Elephant in the room - Akamai

[Rod Beck]

Last time I spoke with an Akamai engineer many years ago the network was purely 
transit. Is that evolving?


From: NANOG  on behalf of Jared Mauch 

Sent: Sunday, December 8, 2019 6:10 PM
To: Brandon Martin 
Cc: nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai



> On Dec 8, 2019, at 11:48 AM, Brandon Martin  wrote:
>
> I guess what I'm getting at is that it sounds like, if you cannot source the 
> content locally to the peering link, there's not likely to be an internal 
> connection to the same site from somewhere else within the Akamai network to 
> deliver that content and, instead, the target network should expect it to 
> come in over the "public Internet" via some other connection.  Is that 
> accurate?
>
> Thanks for the clarifications.



I was hired at Akamai to design the network architecture for a global backbone. 
 This is proving to be an interesting challenge taking a diverse set of 
products with various requirements and interconnecting them in a way that saves 
costs and improves performance while my employers traffic continue to grow.





Akamai is built to use the paths available to deliver traffic and meet our 
customers and our business goals.  Not all our sites are interconnected and 
it’s extremely unlikely (read: possibly never, but who knows) you will see all 
your traffic come over a direct link or cache.  With any sufficiently complex 
system, plus the acquisitions we have made over my short tenure it’s almost 
impractical to integrate them all quickly or possibly at all.

I personally want to make sure that we deliver the traffic in a way that makes 
sense, and a few people have seen those efforts but there’s also many things in 
progress that are not yet complete or ready for public consumption.  I believe 
there’s room here to improve and each time we can turn a switch or dial a knob 
to better serve our customers and the end-users that we are paid to serve, 
everyone wins.


Enterprises vs consumer ISPs have very different traffic profiles, and I think 
the genesis of this thread was a direct result of a very consumer oriented 
traffic profile that was unexpected.  People have wondered why I would spend so 
much time watching things like Apple rumor websites in the past, it’s because 
that would lead to high traffic events.  You go to where the data is.  The same 
can be said for other large download events or OTT launches.  Everyone knows a 
live event can be big but generally bound by the target audience size.

As software is attacked within minutes or hours after security patches are 
released, I don’t find it surprising these days that systems automatically 
download whatever they can the moment it’s released from gaming consoles to IoT 
and server and OS patches.

If the traffic is causing you pain, I encourage you to reach out so we can look 
at what might be improved.

- Jared
(I swear I’ll stop responding.. off to make lunch)

Re: Elephant in the room - Akamai

[Rod Beck]

Yep. Real estate must be one of their largest expenses and unlike bandwidth it 
is not going down to price. 😃


From: Owen DeLong 
Sent: Sunday, December 8, 2019 6:07 PM
To: Rod Beck 
Cc: Jared Mauch ; nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai

My guess (and it’s just this since I haven’t been inside Akamai for a couple of 
years now) is that they are culling the less effective AANPs (from Akamai’s 
perspective) in favor of redeploying the hardware to more effective locations 
and/or to eliminate the cost of supporting/refreshing said hardware.

I would guess that the traffic level required to justify the expense of 
maintaining an AANP (from Akamai’s perspective) probably depends on a great 
many factors not all of which would be obvious as viewed from the outside. I 
would guess that the density of AANPs and ISP interconnection in a given 
geography would be among the factors that would influence that number. I would 
also guess that the number would tend to rise over time.

Again, just external speculation on my part.

Owen


On Dec 8, 2019, at 06:39 , Rod Beck 
mailto:rod.b...@unitedcablecompany.com>> wrote:

Taking boxes out of a network does not sound like 'emergent behavior' or 
unintended consequences. Sounds like a policy change. Perhaps they are being 
redeployed for better performance or perhaps shut down to lower costs. Or may 
be the cost of transit for Akamai at the margin is less than the cost of 
peering with 50 billion peers.

Disclaimer: Not picking a fight. Better things to do.

Regards,

Roderick.


From: Jared Mauch mailto:ja...@puck.nether.net>>
Sent: Sunday, December 8, 2019 1:19 AM
To: Rod Beck 
mailto:rod.b...@unitedcablecompany.com>>
Cc: Shawn L mailto:sha...@up.net>>; 
nanog@nanog.org 
mailto:nanog@nanog.org>>
Subject: Re: Elephant in the room - Akamai

On Dec 7, 2019, at 5:34 PM, Rod Beck 
mailto:rod.b...@unitedcablecompany.com>> wrote:
>
> Have there been any fundamental change in their network architecture that 
> might explain pulling these caches?


Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)

This is why I’m encouraging people to ping me off-list, because the dynamics at 
play for one provider don’t match across the board.  I know we have thousands 
of distinct sites that each have their own attributes and composition at play.

I’ve been working hard to provide value to our AANP partners as well.  I’ll try 
to stop responding to the list at this point but don’t hesitate to contact me 
here or via other means if you’re seeing something weird.  I know I resolved a 
problem a few days ago for someone quickly as there was a misconfiguration left 
around.. We all make mistakes and can all do better.

- jared

https://www.peeringdb.com/asn/20940

Re: Elephant in the room - Akamai

[Jared Mauch]

> On Dec 8, 2019, at 11:48 AM, Brandon Martin  wrote:
> 
> I guess what I'm getting at is that it sounds like, if you cannot source the 
> content locally to the peering link, there's not likely to be an internal 
> connection to the same site from somewhere else within the Akamai network to 
> deliver that content and, instead, the target network should expect it to 
> come in over the "public Internet" via some other connection.  Is that 
> accurate?
> 
> Thanks for the clarifications.



I was hired at Akamai to design the network architecture for a global backbone. 
 This is proving to be an interesting challenge taking a diverse set of 
products with various requirements and interconnecting them in a way that saves 
costs and improves performance while my employers traffic continue to grow.





Akamai is built to use the paths available to deliver traffic and meet our 
customers and our business goals.  Not all our sites are interconnected and 
it’s extremely unlikely (read: possibly never, but who knows) you will see all 
your traffic come over a direct link or cache.  With any sufficiently complex 
system, plus the acquisitions we have made over my short tenure it’s almost 
impractical to integrate them all quickly or possibly at all.

I personally want to make sure that we deliver the traffic in a way that makes 
sense, and a few people have seen those efforts but there’s also many things in 
progress that are not yet complete or ready for public consumption.  I believe 
there’s room here to improve and each time we can turn a switch or dial a knob 
to better serve our customers and the end-users that we are paid to serve, 
everyone wins.  


Enterprises vs consumer ISPs have very different traffic profiles, and I think 
the genesis of this thread was a direct result of a very consumer oriented 
traffic profile that was unexpected.  People have wondered why I would spend so 
much time watching things like Apple rumor websites in the past, it’s because 
that would lead to high traffic events.  You go to where the data is.  The same 
can be said for other large download events or OTT launches.  Everyone knows a 
live event can be big but generally bound by the target audience size.  

As software is attacked within minutes or hours after security patches are 
released, I don’t find it surprising these days that systems automatically 
download whatever they can the moment it’s released from gaming consoles to IoT 
and server and OS patches.

If the traffic is causing you pain, I encourage you to reach out so we can look 
at what might be improved.

- Jared
(I swear I’ll stop responding.. off to make lunch)

Re: Elephant in the room - Akamai

[Owen DeLong]

My guess (and it’s just this since I haven’t been inside Akamai for a couple of 
years now) is that they are culling the less effective AANPs (from Akamai’s 
perspective) in favor of redeploying the hardware to more effective locations 
and/or to eliminate the cost of supporting/refreshing said hardware.

I would guess that the traffic level required to justify the expense of 
maintaining an AANP (from Akamai’s perspective) probably depends on a great 
many factors not all of which would be obvious as viewed from the outside. I 
would guess that the density of AANPs and ISP interconnection in a given 
geography would be among the factors that would influence that number. I would 
also guess that the number would tend to rise over time.

Again, just external speculation on my part.

Owen


> On Dec 8, 2019, at 06:39 , Rod Beck  wrote:
> 
> Taking boxes out of a network does not sound like 'emergent behavior' or 
> unintended consequences. Sounds like a policy change. Perhaps they are being 
> redeployed for better performance or perhaps shut down to lower costs. Or may 
> be the cost of transit for Akamai at the margin is less than the cost of 
> peering with 50 billion peers. 
> 
> Disclaimer: Not picking a fight. Better things to do. 
> 
> Regards, 
> 
> Roderick. 
> 
> From: Jared Mauch 
> Sent: Sunday, December 8, 2019 1:19 AM
> To: Rod Beck 
> Cc: Shawn L ; nanog@nanog.org 
> Subject: Re: Elephant in the room - Akamai
>  
> On Dec 7, 2019, at 5:34 PM, Rod Beck  wrote:
> > 
> > Have there been any fundamental change in their network architecture that 
> > might explain pulling these caches?
> 
> 
> Please see my email on Friday where I outlined a few of the dynamics at play. 
>  Akamai isn’t just one thing, it’s an entire basket of products that all have 
> their own resulting behaviors.  This is why even though you may peer with us 
> directly you may not see 100% of the traffic from that interconnection.  
> (Take SSL for example, it’s often not served via the clusters in an ISP due 
> to the security requirements we place on those racks, and this is something 
> we treat very seriously!)
> 
> This is why I’m encouraging people to ping me off-list, because the dynamics 
> at play for one provider don’t match across the board.  I know we have 
> thousands of distinct sites that each have their own attributes and 
> composition at play.
> 
> I’ve been working hard to provide value to our AANP partners as well.  I’ll 
> try to stop responding to the list at this point but don’t hesitate to 
> contact me here or via other means if you’re seeing something weird.  I know 
> I resolved a problem a few days ago for someone quickly as there was a 
> misconfiguration left around.. We all make mistakes and can all do better.
> 
> - jared
> 
> https://www.peeringdb.com/asn/20940 

Re: Puerto Rico IX (operational)

[Mehmet Akcin]

thank you all for the wonderful feedback. We are in the process of
implementing https://www.ixpmanager.org/ as many recommended. IPv6 enabled.

We were able to connect 4 major data centers within san Juan metro using n
x 10G ring thanks to various partners and special thanks this time goes to
OSnet of Puerto Rico for providing lots of onsite support.

Still trying to grow the number of critical ASNs in the IX, if you have any
suggestions, please let us know.

Mehmet

On Wed, Nov 13, 2019 at 5:56 PM Mehmet Akcin  wrote:

> Hey there,
>
> Puerto Rico IX , famously known as PRIX , is now operational.
>
> You can visit www.puertoricoix.net to see sites you can connect to PRIX,
> members and join Ix mailing list.
>
> There are no fees to join the IX. We hope to keep this, this way until
> there is enough interest to form a non profit org.
>
> I would like to thank once again Aeronet, Arista and Cloudsmash for their
> donations of equipment , time and energy!
>
> Now, we are looking for those who want to deploy 1-2RU caching boxes
> anything from DNS to Content.
>
> We are working on more stats/looking glass, etc soon!
>
> Mehmet
> --
> Mehmet
> +1-424-298-1903
>

Re: Elephant in the room - Akamai

[Brandon Martin]

On 12/8/19 10:58 AM, Jared Mauch wrote:

Not all content is suitable in all locations based on the physical security or 
market situation. We have some content that can not be served, an example is 
locations where there are licensing requirements (eg: ICP for China).

You will see a different mix from our 20940 vs 16625 as well. Those have 
different requirements on the security side. If you treat your PKI data 
seriously you will appreciate what is done here.

In Marquette Michigan there will be different opportunities compared with 
Amsterdam or Ashburn as well.

Our customers and traffic mix makes it challenging to serve from a platform 
where you do capital planning for several year depreciation cycle. We have 
thousands of unique sites and that scale is quite different from serving on a 
few distinct IXPs and transit providers.

So yes you will see a difference and there are things we can do to improve it 
when there is a variance in the behavior.
I guess what I'm getting at is that it sounds like, if you cannot source 
the content locally to the peering link, there's not likely to be an 
internal connection to the same site from somewhere else within the 
Akamai network to deliver that content and, instead, the target network 
should expect it to come in over the "public Internet" via some other 
connection.  Is that accurate?


Thanks for the clarifications.
--
Brandon Martin

Re: Elephant in the room - Akamai

[Mehmet Akcin]

Let's take a minute and thank Jared for taking the time and responding.

thank you, Jared.

On Sun, Dec 8, 2019 at 10:58 AM Jared Mauch  wrote:

> Not all content is suitable in all locations based on the physical
> security or market situation. We have some content that can not be served,
> an example is locations where there are licensing requirements (eg: ICP for
> China).
>
> You will see a different mix from our 20940 vs 16625 as well. Those have
> different requirements on the security side. If you treat your PKI data
> seriously you will appreciate what is done here.
>
> In Marquette Michigan there will be different opportunities compared with
> Amsterdam or Ashburn as well.
>
> Our customers and traffic mix makes it challenging to serve from a
> platform where you do capital planning for several year depreciation cycle.
> We have thousands of unique sites and that scale is quite different from
> serving on a few distinct IXPs and transit providers.
>
> So yes you will see a difference and there are things we can do to improve
> it when there is a variance in the behavior.
>
> - Jared
>
> > On Dec 8, 2019, at 10:33 AM, Brandon Martin 
> wrote:
> >
> > On 12/7/19 7:19 PM, Jared Mauch wrote:
> >> Please see my email on Friday where I outlined a few of the dynamics at
> play.  Akamai isn’t just one thing, it’s an entire basket of products that
> all have their own resulting behaviors.  This is why even though you may
> peer with us directly you may not see 100% of the traffic from that
> interconnection.  (Take SSL for example, it’s often not served via the
> clusters in an ISP due to the security requirements we place on those
> racks, and this is something we treat very seriously!)
> >
> > Does this mean that, if you peer with Akamai at some location, only
> content locally available at that location will come over that peering
> session with the rest coming via other means?  Does Akamai not have private
> connectivity to their public peering points?
> > --
> > Brandon Martin
>

Re: Elephant in the room - Akamai

[Mark Delany]

> Have there been any fundamental change in their network architecture
> that might explain pulling these caches?

Maybe not network architecture, but what if the cache-to-content ratio
is dropping dramatically due to changes in consumer behavior and/or a
huge increase in the underlying content (such as adoption of higher
and multiple-resolution videos)?

There has to be a tipping point at which a proportionally small cache
becomes almost worthless from a traffic saving perspective.

If you run a cluster one presumes you can see what your in/out ratio
looks like and where the trend-line is headed.


Another possibility might be security. It may be that they need
additional security credentials for newer services which they are
reluctant to load into remote cache clusters they don't physically
control.


Mark.

Re: Elephant in the room - Akamai

[Jared Mauch]

Not all content is suitable in all locations based on the physical security or 
market situation. We have some content that can not be served, an example is 
locations where there are licensing requirements (eg: ICP for China). 

You will see a different mix from our 20940 vs 16625 as well. Those have 
different requirements on the security side. If you treat your PKI data 
seriously you will appreciate what is done here. 

In Marquette Michigan there will be different opportunities compared with 
Amsterdam or Ashburn as well. 

Our customers and traffic mix makes it challenging to serve from a platform 
where you do capital planning for several year depreciation cycle. We have 
thousands of unique sites and that scale is quite different from serving on a 
few distinct IXPs and transit providers. 

So yes you will see a difference and there are things we can do to improve it 
when there is a variance in the behavior. 

- Jared 

> On Dec 8, 2019, at 10:33 AM, Brandon Martin  wrote:
> 
> On 12/7/19 7:19 PM, Jared Mauch wrote:
>> Please see my email on Friday where I outlined a few of the dynamics at 
>> play.  Akamai isn’t just one thing, it’s an entire basket of products that 
>> all have their own resulting behaviors.  This is why even though you may 
>> peer with us directly you may not see 100% of the traffic from that 
>> interconnection.  (Take SSL for example, it’s often not served via the 
>> clusters in an ISP due to the security requirements we place on those racks, 
>> and this is something we treat very seriously!)
> 
> Does this mean that, if you peer with Akamai at some location, only content 
> locally available at that location will come over that peering session with 
> the rest coming via other means?  Does Akamai not have private connectivity 
> to their public peering points?
> -- 
> Brandon Martin

Re: Elephant in the room - Akamai

[Brandon Martin]

On 12/7/19 7:19 PM, Jared Mauch wrote:

Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)


Does this mean that, if you peer with Akamai at some location, only 
content locally available at that location will come over that peering 
session with the rest coming via other means?  Does Akamai not have 
private connectivity to their public peering points?

--
Brandon Martin

Re: Elephant in the room - Akamai

[Ben Cannon]

+100, and thanks to Jared.

-Ben

> On Dec 7, 2019, at 10:08 PM, Mark Tinka  wrote:
> 
> 
> 
>> On 8/Dec/19 02:19, Jared Mauch wrote:
>> 
>> I’ve been working hard to provide value to our AANP partners as well.  I’ll 
>> try to stop responding to the list at this point but don’t hesitate to 
>> contact me here or via other means if you’re seeing something weird.  I know 
>> I resolved a problem a few days ago for someone quickly as there was a 
>> misconfiguration left around.. We all make mistakes and can all do better.
> 
> Problems are part of the gig - otherwise we'd have no reason to get up
> in the morning.
> 
> What matters is that there is someone you can find to help you fix them.
> That's what makes all the difference.
> 
> So kudos to you, Jared, and the entire team out there at Akamai.
> 
> Mark.

Re: Elephant in the room - Akamai

[Rod Beck]

Taking boxes out of a network does not sound like 'emergent behavior' or 
unintended consequences. Sounds like a policy change. Perhaps they are being 
redeployed for better performance or perhaps shut down to lower costs. Or may 
be the cost of transit for Akamai at the margin is less than the cost of 
peering with 50 billion peers.

Disclaimer: Not picking a fight. Better things to do.

Regards,

Roderick.


From: Jared Mauch 
Sent: Sunday, December 8, 2019 1:19 AM
To: Rod Beck 
Cc: Shawn L ; nanog@nanog.org 
Subject: Re: Elephant in the room - Akamai

On Dec 7, 2019, at 5:34 PM, Rod Beck  wrote:
>
> Have there been any fundamental change in their network architecture that 
> might explain pulling these caches?


Please see my email on Friday where I outlined a few of the dynamics at play.  
Akamai isn’t just one thing, it’s an entire basket of products that all have 
their own resulting behaviors.  This is why even though you may peer with us 
directly you may not see 100% of the traffic from that interconnection.  (Take 
SSL for example, it’s often not served via the clusters in an ISP due to the 
security requirements we place on those racks, and this is something we treat 
very seriously!)

This is why I’m encouraging people to ping me off-list, because the dynamics at 
play for one provider don’t match across the board.  I know we have thousands 
of distinct sites that each have their own attributes and composition at play.

I’ve been working hard to provide value to our AANP partners as well.  I’ll try 
to stop responding to the list at this point but don’t hesitate to contact me 
here or via other means if you’re seeing something weird.  I know I resolved a 
problem a few days ago for someone quickly as there was a misconfiguration left 
around.. We all make mistakes and can all do better.

- jared

https://www.peeringdb.com/asn/20940