Re: Traffic engineering and peering for CDNs

[Wolff, Nick]

On 6/7/16, 2:46 AM, "NANOG on behalf of Mark Tinka"
 wrote:

>
>
>On 6/Jun/16 20:03, Tom Smyth wrote:
>
>> as far as im aware ... a friend of mine on INEX in Ireland said most
>>cdns
>> use source ip of the DNS requests to determine which network to direct
>>them
>> to ... so if you use you have your own resolver on  an ip address  in
>>your
>> network range cdns can accurately determine what network the request is
>> comming from and determine what  ip address / what network that the cdn
>>has
>> nearest to your network...
>>
>> ff you use 3rd party  dns servers for your clients... you may not get an
>> optimal ip answer for your dns queries from the CDNS involved
>
>Some CDN's use DNS (in addition to latency, congestion levels, busy
>state, e.t.c.).
>
>Others use Anycast routing, which I tend to prefer. The problem is the
>latter run a network while the former may typically not.
>
>Mark.

Also some companies make layer 7 decisions for their CDN¹s in conjunction
with these other methods. Their applications makes a decision on what host
to send you to based on routing information, your source address, and
other accumulated data.

Re: Traffic engineering and peering for CDNs

[Mark Tinka]

On 6/Jun/16 20:03, Tom Smyth wrote:

> as far as im aware ... a friend of mine on INEX in Ireland said most cdns
> use source ip of the DNS requests to determine which network to direct them
> to ... so if you use you have your own resolver on  an ip address  in your
> network range cdns can accurately determine what network the request is
> comming from and determine what  ip address / what network that the cdn has
> nearest to your network...
>
> ff you use 3rd party  dns servers for your clients... you may not get an
> optimal ip answer for your dns queries from the CDNS involved

Some CDN's use DNS (in addition to latency, congestion levels, busy
state, e.t.c.).

Others use Anycast routing, which I tend to prefer. The problem is the
latter run a network while the former may typically not.

Mark.

Re: Traffic engineering and peering for CDNs

[Tom Smyth]

as far as im aware ... a friend of mine on INEX in Ireland said most cdns
use source ip of the DNS requests to determine which network to direct them
to ... so if you use you have your own resolver on  an ip address  in your
network range cdns can accurately determine what network the request is
comming from and determine what  ip address / what network that the cdn has
nearest to your network...

ff you use 3rd party  dns servers for your clients... you may not get an
optimal ip answer for your dns queries from the CDNS involved

I hope this helps

Tom Smyth

On Mon, Jun 6, 2016 at 6:53 PM, Mike Hammett <na...@ics-il.net> wrote:

> Some rely on performance testing to the client's DNS resolver and if
> they're not using on-net ones, they'll be directed to use a different CDN
> node.
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> Midwest Internet Exchange
> http://www.midwest-ix.com
>
>
> - Original Message -
>
> From: "Graham Johnston" <johnst...@westmancom.com>
> To: "nanog@nanog.org" <nanog@nanog.org>
> Sent: Monday, June 6, 2016 8:36:43 AM
> Subject: Traffic engineering and peering for CDNs
>
> Lately I have been putting in some effort to maximize our IX connections
> by trying to work with the top 5-ish list of ASNs that still send us
> traffic via a paid transit connection despite the fact that we are both
> present on the same IX(s). In one case I missed the fact that one ASN
> wasn't using the IXs route-servers, that's on me for not spotting that one.
>
> Even with proper IX peering in place though it seems like some CDNs are
> better at using the IX connections than others. ASN 15169 for instance does
> an excellent job sending more than 99.99% of traffic via the IX connection;
> thank you. While others only seem to manage to send 60 - 80% of traffic via
> the IX. What I am not understanding about the respective CDN's network
> wherein they don't send traffic to me through a consistent path? Is the
> content coming from widely different places and rather than transport it
> across their own network from a remote site they would rather hot-potato it
> out a local transit connection? Are their transit costs so low that they
> don't care about using an IX connection over transit unlike a small
> operator like me? Is this just a non-obvious issue wherein they maybe just
> can't originate enough of the traffic near the IX and therefore don't make
> use of the IX connection, again a hot-potato phenomenon?
>
> Secondly can someone explain to me why some CDNs want a gigabit or two of
> traffic to be exchanged between our respective networks before they would
> peer with me via a public IX? I totally get those kinds of thresholds
> before engaging in a private interconnect but I don't understand the
> reluctance with regard to a public IX, that they are already established
> at. Is it again just a simple case of bandwidth economics that operate at a
> different scale than I can comprehend?
>
> I'm hoping the community can shed some light on this for me as I'm trying
> to avoid grilling the operators that are working with me as I don't expect
> those front line individuals to necessarily have a full view of the factors
> at play.
>
> Thanks,
> Graham Johnston
> Network Planner
> Westman Communications Group
> 204.717.2829
> johnst...@westmancom.com<mailto:johnst...@westmancom.com>
> P think green; don't print this email.
>
>
>


-- 
Kindest regards,
Tom Smyth

Mobile: +353 87 6193172
-
PLEASE CONSIDER THE ENVIRONMENT BEFORE YOU PRINT THIS E-MAIL
This email contains information which may be confidential or privileged.
The information is intended solely for the use of the individual or entity
named above.  If you are not the intended recipient, be aware that
any disclosure, copying, distribution or use of the contents of this
information is prohibited. If you have received this electronic
transmission in error, please notify me by telephone or by electronic mail
immediately. Any opinions expressed are those of the author, not the
company's  .This email does not constitute either offer or acceptance of
any contractually binding agreement. Such offer or acceptance must be
communicated in
writing. You are requested to carry out your own virus check before opening
any attachment. Thomas Smyth accepts no liability for any loss or damage
which may be caused by malicious software or attachments.

AW: Traffic engineering and peering for CDNs

[Bernd Spiess]

Hi Graham!

In addition to the other two comments, I´d like to add some topics:

> Lately I have been putting in some effort to maximize our IX connections
by 
> trying to work with the top 5-ish list of ASNs that still send us traffic
via a paid 
> transit connection despite the fact that we are both present on the same
IX(s). 
> In one case I missed the fact that one ASN wasn't using the IXs
route-servers, 
> that's on me for not spotting that one.

This brings up some ideas ... see here:

1) Check if the CDN is on the routeserver
2) Check if the CDN has maybe tagged his prefixes with a "do-not-announce"
tag (verify at looking glass) (relevant of course only for outbound traffic)
3) Try to establish a direct peering session with the CDN over the IXP so
that you are known to the CDN 
   ... some CDN´s could maybe also prefer or have higher priority on direct
sessions than via only routeserver...
   ... quite some networks give you a larger set of prefixes on a direct
session...
4) Talk with the CDN and check his geolocation tagging's for your prefixes
and maybe let correct them after you have found out what they are doing
5) Think on the fact, that CDN´s could take their routing decision on the
geo-location of the used dns resolver server and not on the users ip address
6) Check, that your network or your customers are not referring to public
dns resolvers
7) Think on the fact, that ipv6 could maybe be in place and active too ...
so don´t forget your ipv6 path and ipv6 dns resolvers...
8) Check your RADB/RIPE/AFRINIC/... routing db entries - if you did
something wrong, maybe you are filtered... (check:
http://irrexplorer.nlnog.net/ and looking glasses)
9) Check that you have your set of IP prefixes in good order and do not
implement strange magic with asymmetric more specifics ... reduce everything
to supernetworks on all edges!
10) Check if you are not handing over bgp tagging's which could cause some
prevention automatics and check if 
   you do not send high metric values which are maybe causing negative
routing decisions on the other side
11) Think on the fact, that CDN´s have the geographic distance as one of
their parameters - if the CDN node is too 
   far away from your network - maybe a closer located content box via
transit is rated higher than maybe the long 
   distance via peering (if there is no node or no node with the right
content next to you) ... typically CDN are very close to larger IXP´s...
12) Think on the fact, that your ip-transit upstream could be a peering for
the CDN ... This neutralizes the peering or transit rating
   from the point of view of the CDN, but of course not for you ... so: talk
with the CDN ... therefore you must be known ... 
   therefore, you should have done a peering session ...
13) Talk with your IXP ... maybe he can help ...

Anything missing here? Anything wrong with my ideas?

(disclaimer: yes, I do work for an IXP ... but this is my personal opinion)

happy peering...
Bernd

Bernd Spiess (Manager Peering Service / DE-CIX Management GmbH)




smime.p7s
Description: S/MIME cryptographic signature

Re: Traffic engineering and peering for CDNs

[Mike Hammett]

Some rely on performance testing to the client's DNS resolver and if they're 
not using on-net ones, they'll be directed to use a different CDN node. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Graham Johnston" <johnst...@westmancom.com> 
To: "nanog@nanog.org" <nanog@nanog.org> 
Sent: Monday, June 6, 2016 8:36:43 AM 
Subject: Traffic engineering and peering for CDNs 

Lately I have been putting in some effort to maximize our IX connections by 
trying to work with the top 5-ish list of ASNs that still send us traffic via a 
paid transit connection despite the fact that we are both present on the same 
IX(s). In one case I missed the fact that one ASN wasn't using the IXs 
route-servers, that's on me for not spotting that one. 

Even with proper IX peering in place though it seems like some CDNs are better 
at using the IX connections than others. ASN 15169 for instance does an 
excellent job sending more than 99.99% of traffic via the IX connection; thank 
you. While others only seem to manage to send 60 - 80% of traffic via the IX. 
What I am not understanding about the respective CDN's network wherein they 
don't send traffic to me through a consistent path? Is the content coming from 
widely different places and rather than transport it across their own network 
from a remote site they would rather hot-potato it out a local transit 
connection? Are their transit costs so low that they don't care about using an 
IX connection over transit unlike a small operator like me? Is this just a 
non-obvious issue wherein they maybe just can't originate enough of the traffic 
near the IX and therefore don't make use of the IX connection, again a 
hot-potato phenomenon? 

Secondly can someone explain to me why some CDNs want a gigabit or two of 
traffic to be exchanged between our respective networks before they would peer 
with me via a public IX? I totally get those kinds of thresholds before 
engaging in a private interconnect but I don't understand the reluctance with 
regard to a public IX, that they are already established at. Is it again just a 
simple case of bandwidth economics that operate at a different scale than I can 
comprehend? 

I'm hoping the community can shed some light on this for me as I'm trying to 
avoid grilling the operators that are working with me as I don't expect those 
front line individuals to necessarily have a full view of the factors at play. 

Thanks, 
Graham Johnston 
Network Planner 
Westman Communications Group 
204.717.2829 
johnst...@westmancom.com<mailto:johnst...@westmancom.com> 
P think green; don't print this email. 

Re: Traffic engineering and peering for CDNs

[Phil Rosenthal]

Hello,

> On Jun 6, 2016, at 7:36 AM, Graham Johnston  wrote:
> 
> Lately I have been putting in some effort to maximize our IX connections by 
> trying to work with the top 5-ish list of ASNs that still send us traffic via 
> a paid transit connection despite the fact that we are both present on the 
> same IX(s). In one case I missed the fact that one ASN wasn't using the IXs 
> route-servers, that's on me for not spotting that one.
> 
> Even with proper IX peering in place though it seems like some CDNs are 
> better at using the IX connections than others.  ASN 15169 for instance does 
> an excellent job sending more than 99.99% of traffic via the IX connection; 
> thank you. While others only seem to manage to send 60 - 80% of traffic via 
> the IX.  What I am not understanding about the respective CDN's network 
> wherein they don't send traffic to me through a consistent path? Is the 
> content coming from widely different places and rather than transport it 
> across their own network from a remote site they would rather hot-potato it 
> out a local transit connection?  Are their transit costs so low that they 
> don't care about using an IX connection over transit unlike a small operator 
> like me? Is this just a non-obvious issue wherein they maybe just can't 
> originate enough of the traffic near the IX and therefore don't make use of 
> the IX connection, again a hot-potato phenomenon?

Most CDN’s do not have a backbone.  Transit costs are not free, but as most 
traffic is served by local nodes from cache, the costs of transport between 
locations in many cases is higher than just sending via transit. In some cases, 
the CDN may not have good mapping and may not be certain which node is best to 
serve your customers. In other cases, not all content exists on all nodes, and 
they may redirect to serve from the nodes which have the content. Finally, 
there may be an outage or capacity limits from the closest location, and 
another location may be serving to make up the shortfall.

> 
> Secondly can someone explain to me why some CDNs want a gigabit or two of 
> traffic to be exchanged between our respective networks before they would 
> peer with me via a public IX? I totally get those kinds of thresholds before 
> engaging in a private interconnect but I don't understand the reluctance with 
> regard to a public IX, that they are already established at. Is it again just 
> a simple case of bandwidth economics that operate at a different scale than I 
> can comprehend?
> 

This sounds like a surprisingly high threshold, but to some extent it boils 
down like this — setting up sessions requires some time. In the ideal world, 
the peer is intelligent and has everything set up properly, but even in this 
case, it still requires some time for making sure things go up properly. Some 
(but not all) CDN’s have it automated to reduce this time. Some potential 
peering networks are poorly run, and will leak routes, not announce all of 
their routes, will not configure the sessions properly, etc — this adds up to 
significantly more time. Before the CDN starts setting up peering with another 
network, it is not necessarily obvious if the potential peer is run by 
competent people or not. Many CDN’s are members of the route servers.  If you 
are exchanging a small amount of traffic, and both you and the CDN are on the 
Route Server for the IX, there maybe no reason to set up direct sessions which 
will require both more coordination time for configuration, and more router cpu 
time/ram on an ongoing basis. From the perspective of the CDN, most likely, 
1Gbps or less is a perfectly reasonable amount of traffic to exchange to peers 
who are learned only via the route server, and not directly.
 
> I'm hoping the community can shed some light on this for me as I'm trying to 
> avoid grilling the operators that are working with me as I don't expect those 
> front line individuals to necessarily have a full view of the factors at play.
> 
> Thanks,
> Graham Johnston
> Network Planner
> Westman Communications Group
> 204.717.2829
> johnst...@westmancom.com
> P think green; don't print this email.
> 

Best Regards,
-Phil Rosenthal
ISPrime

Re: Traffic engineering and peering for CDNs

[Jon Lewis]

On Mon, 6 Jun 2016, Graham Johnston wrote:

What I am not understanding about the respective CDN's network wherein 
they don't send traffic to me through a consistent path? Is the content 
coming from widely different places and rather than transport it across 
their own network from a remote site they would rather hot-potato it out 
a local transit connection?


Depends on the CDN, but its possible the traffic is coming from different 
locations and not all CDNs even have a network, so if you don't have 
peering with their location serving the traffic (and they don't have a 
network), the traffic will have to come to you via other paths.


Are their transit costs so low that they don't care about using an IX 
connection over transit unlike a small operator like me? Is this just a


Maybe.  Or maybe the traffic to you is small enough (to them) that you're 
not on their radar as a desirable peer.  Or maybe they just haven't gotten 
around to sending you a peering request yet.


Secondly can someone explain to me why some CDNs want a gigabit or two 
of traffic to be exchanged between our respective networks before they 
would peer with me via a public IX?


Which ones want that much?  We like to see some traffic before moving from 
"IX route-server peering" to direct peering via the IX, just because there 
are so many possible peers and only so much router resources.  It's really 
not worth the resources (router or management) to direct peer with a 
network with which there's virtually no traffic being exchanged, just 
because we're on the same IX(s).  1-2G to peer seems kind of high.  Some 
might insist that you move peering to PNI if you're doing >1-2G across an 
IX.


--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Traffic engineering and peering for CDNs

[Graham Johnston]

Lately I have been putting in some effort to maximize our IX connections by 
trying to work with the top 5-ish list of ASNs that still send us traffic via a 
paid transit connection despite the fact that we are both present on the same 
IX(s). In one case I missed the fact that one ASN wasn't using the IXs 
route-servers, that's on me for not spotting that one.

Even with proper IX peering in place though it seems like some CDNs are better 
at using the IX connections than others.  ASN 15169 for instance does an 
excellent job sending more than 99.99% of traffic via the IX connection; thank 
you. While others only seem to manage to send 60 - 80% of traffic via the IX.  
What I am not understanding about the respective CDN's network wherein they 
don't send traffic to me through a consistent path? Is the content coming from 
widely different places and rather than transport it across their own network 
from a remote site they would rather hot-potato it out a local transit 
connection?  Are their transit costs so low that they don't care about using an 
IX connection over transit unlike a small operator like me? Is this just a 
non-obvious issue wherein they maybe just can't originate enough of the traffic 
near the IX and therefore don't make use of the IX connection, again a 
hot-potato phenomenon?

Secondly can someone explain to me why some CDNs want a gigabit or two of 
traffic to be exchanged between our respective networks before they would peer 
with me via a public IX? I totally get those kinds of thresholds before 
engaging in a private interconnect but I don't understand the reluctance with 
regard to a public IX, that they are already established at. Is it again just a 
simple case of bandwidth economics that operate at a different scale than I can 
comprehend?

I'm hoping the community can shed some light on this for me as I'm trying to 
avoid grilling the operators that are working with me as I don't expect those 
front line individuals to necessarily have a full view of the factors at play.

Thanks,
Graham Johnston
Network Planner
Westman Communications Group
204.717.2829
johnst...@westmancom.com
P think green; don't print this email.