Re: CDN node locations
There are a number of us here. Best, -M On Sun, Nov 17, 2013 at 3:21 AM, Warren Bailey wbai...@satelliteintelligencegroup.com wrote: U.. So who gets to field the Akamai questions now? ;) Sent from my Mobile Device. Original message From: Patrick W. Gilmore patr...@ianai.net Date: 11/16/2013 4:10 PM (GMT-09:00) To: NANOG list nanog@nanog.org Subject: Re: CDN node locations On Nov 16, 2013, at 19:36 , Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8, and 4.2.2.1, and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs were only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? #1: I could not possibly comment on customers. But since I've only worked at Markley Group for 3 weeks, I don't know all the customers, so I couldn't tell you even if they were customers at all, more or less how they do things. Besides, Markley Group ain't a CDN. #2: Assuming you are assuming I still work at Akamai (I don't), and are asking me if that's how Akamai does things, I couldn't possibly comment on customers at a previous position. Everything I've said up to now was either public knowledge or something I was more than happy to give out publicly if asked while I was at Akamai. The query above, specifically is XXX how customer YYY does things, is neither of those. But in the more general sense, your hypothesis does not really fit the circumstances completely. DNS is orthogonal to serving bits. If Sprint's DNS is f00bar'ed, then you can't resolve anything, CDN-ififed or not. It is true some CDNs put some name servers inside other networks, but that is still a race condition, because (for instance) Akamai's DNS TTL is 20 seconds. You have to go back 'outside' eventually to get stuff, which means relying on Sprint's recursive NSes. Plus the two sites you list (YouTube DailyShow) are not on the same infrastructure. Google hosts its own videos, DailyShow is not hosted on Google (AFAIK), therefore they must be two different companies using two different pieces of equipment and two different name server algorithms / topologies. It would be weird that Sprint's failure mode worked fine for those two and nothing else. Sorry. -- TTFN, patrick P.S. I wasn't chuckling. :)
Re: CDN node locations
On Nov 16, 2013, at 19:36 , Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8, and 4.2.2.1, and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs were only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? #1: I could not possibly comment on customers. But since I've only worked at Markley Group for 3 weeks, I don't know all the customers, so I couldn't tell you even if they were customers at all, more or less how they do things. Besides, Markley Group ain't a CDN. #2: Assuming you are assuming I still work at Akamai (I don't), and are asking me if that's how Akamai does things, I couldn't possibly comment on customers at a previous position. Everything I've said up to now was either public knowledge or something I was more than happy to give out publicly if asked while I was at Akamai. The query above, specifically is XXX how customer YYY does things, is neither of those. But in the more general sense, your hypothesis does not really fit the circumstances completely. DNS is orthogonal to serving bits. If Sprint's DNS is f00bar'ed, then you can't resolve anything, CDN-ififed or not. It is true some CDNs put some name servers inside other networks, but that is still a race condition, because (for instance) Akamai's DNS TTL is 20 seconds. You have to go back 'outside' eventually to get stuff, which means relying on Sprint's recursive NSes. Plus the two sites you list (YouTube DailyShow) are not on the same infrastructure. Google hosts its own videos, DailyShow is not hosted on Google (AFAIK), therefore they must be two different companies using two different pieces of equipment and two different name server algorithms / topologies. It would be weird that Sprint's failure mode worked fine for those two and nothing else. Sorry. -- TTFN, patrick P.S. I wasn't chuckling. :) signature.asc Description: Message signed with OpenPGP using GPGMail
Re: CDN node locations
On 11/16/13, 7:36 PM, Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8, and 4.2.2.1, and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs were only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? It seems more likely the Sprint resolvers you were using were having difficulty reaching external authoratative servers but the devices they proxy all the media content through wasn't... All major media content these days is CDN'd but I don't think that had anything to do with it. Phil
Re: CDN node locations
Maybe, but I don't use their proxies, I've overriden them for speed. Phil Bedard bedard.p...@gmail.com wrote: On 11/16/13, 7:36 PM, Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8, and 4.2.2.1, and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs were only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? It seems more likely the Sprint resolvers you were using were having difficulty reaching external authoratative servers but the devices they proxy all the media content through wasn't... All major media content these days is CDN'd but I don't think that had anything to do with it. Phil -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Re: CDN node locations
http://www.sprint.com/legal/open_internet_information.html Maybe proxy was the wrong word, and transparent video optimization are better words. :) I'm not speaking with any internal knowledge of Sprint Wireless's network but I wouldn't be surprised if you had no choice in the matter. Phil From: Jay Ashworth j...@baylink.com Date: Saturday, November 16, 2013 at 8:56 PM To: Phil Bedard bedard.p...@gmail.com, NANOG nanog@nanog.org Subject: Re: CDN node locations Maybe, but I don't use their proxies, I've overriden them for speed. Phil Bedard bedard.p...@gmail.com wrote: On 11/16/13, 7:36 PM, Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8 http://8.8.8.8 , and 4.2.2.1 http://4.2.2.1 , and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs wer e only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? It seems more likely the Sprint resolvers you were using were having difficulty reaching external authoratative servers but the devices they proxy all the media content through wasn't... All major media content these days is CDN'd but I don't think that had anything to do with it. Phil -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
Re: CDN node locations
U.. So who gets to field the Akamai questions now? ;) Sent from my Mobile Device. Original message From: Patrick W. Gilmore patr...@ianai.net Date: 11/16/2013 4:10 PM (GMT-09:00) To: NANOG list nanog@nanog.org Subject: Re: CDN node locations On Nov 16, 2013, at 19:36 , Jay Ashworth j...@baylink.com wrote: Second, a list of CDN nodes is likely impossible to gather maintain without the help of the CDNs themselves. There are literally thousands of them, most do not serve the entire Internet, and they change frequently. And before you ask, I know at least Akamai will _not_ give you their list, so don't even try to ask them. I find myself unsurprised. I was led to a very interesting failure case involving CDN's a couple weeks ago, that I thought you might find amusing. I have a Samsung Galaxy S4, with Sprint. On a semi-regular basis, the networking gets flaky around 1-2am ish local time, but 3 weekends ago, the symptom I saw was DNS lookups failed -- and it wasn't clear to me whether it was just some lookups failed, or that Big Sites were cached at the provider, and *all* outgoing 53 traffic to the greater internet wasn't being forwarded by Sprint's customer resolvers. I know that it was their resolvers, though, as I grabbed a copy of Set DNS, and pointed my phone to 8.8.8.8, and 4.2.2.1, and OpenDNS, and like that, and everything worked ok. Except media. (Patrick is starting to nod and chuckle, now :-) Both YouTube and The Daily Show's apps worked ok, but refused to play video clips for me. If I reset the DNS to normal, I went back to not all sites are reachable, but media plays fine. My diagnosis was that those sites were CDNed, and the DNS names to *which* they were CDNs were only visible inside Sprint's event horizon, so when I was on alternate DNS resolution, I couldn't get to them. But that took me over a day to figure out. Don't get old. :-) Patrick? Is that how (at least some) customers do it? #1: I could not possibly comment on customers. But since I've only worked at Markley Group for 3 weeks, I don't know all the customers, so I couldn't tell you even if they were customers at all, more or less how they do things. Besides, Markley Group ain't a CDN. #2: Assuming you are assuming I still work at Akamai (I don't), and are asking me if that's how Akamai does things, I couldn't possibly comment on customers at a previous position. Everything I've said up to now was either public knowledge or something I was more than happy to give out publicly if asked while I was at Akamai. The query above, specifically is XXX how customer YYY does things, is neither of those. But in the more general sense, your hypothesis does not really fit the circumstances completely. DNS is orthogonal to serving bits. If Sprint's DNS is f00bar'ed, then you can't resolve anything, CDN-ififed or not. It is true some CDNs put some name servers inside other networks, but that is still a race condition, because (for instance) Akamai's DNS TTL is 20 seconds. You have to go back 'outside' eventually to get stuff, which means relying on Sprint's recursive NSes. Plus the two sites you list (YouTube DailyShow) are not on the same infrastructure. Google hosts its own videos, DailyShow is not hosted on Google (AFAIK), therefore they must be two different companies using two different pieces of equipment and two different name server algorithms / topologies. It would be weird that Sprint's failure mode worked fine for those two and nothing else. Sorry. -- TTFN, patrick P.S. I wasn't chuckling. :)
