GigaOm has begin tracking this story: http://gigaom.com/2010/11/30/a-play-by-play-on-the-comcast-and-level-3-spat
On Tue, Nov 30, 2010 at 1:02 PM, Leo Bicknell <bickn...@ufp.org> wrote: > > Having been involved with a few peering spats in the past I know > what is said publically rarely matches the reality behind the scenes. > In this particular case my spidy sense tells me there is absolutely > something interesting behind the scenes, but the question is what. > > I'd never really paid attention to how Netflix delivers its content. > It's obviously a lot of bandwidth, and likely part of the issue > here so I thought I would investigate. > > Apparently Akamai has been the primary Netflix streaming source > since March. LimeLight Networks has been a secondary provider, and > it would appear those two make up the vast majority of Netflix's > actual streaming traffic. I can't tell if Netflix does any streaming > out of their own ASN, but if they do it appears to be minor. > > Here's a reference from the business side of things: > http://www.fool.com/investing/general/2010/11/11/netflix-takes-streaming-to-a-new-level.aspx > > This is also part of the reason I went back to the very first message in > this thread to reply: > > In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. > Gilmore wrote: >> <http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp> >> >> I understand that politics is off-topic, but this policy affects operational >> aspects of the 'Net. > > Patrick works for Akamai, it seems likely he might know more about > what is going on. Likely he can't discuss the details, but wanted > to seed a discussion. I'd say that worked well. > > I happen to be a Comcast cable modem customer. Gooling for people > who had issues getting to Netflix streaming turned up plenty of > forum posts with traceroutes to Netflix servers on Akamai and > Limelight. I did traceroutes to about 20 of them from my cable > modem, and it's clear Comcast and Akamai and Comcast and Limelight > are interconnected quite well. Akamai does not sell IP Transit, > and I'm thinking it is extremely unlikely that Comcast is buying > transit from Limelight. I will thus conclude that these are either > peering relationships, or that they have cut some sort of special > "CDN Interconnect" deal with Comcast. > > But what about Level 3? One of my friends I was chatting with on AIM > said they thought Comcast was a Level 3 customer, at least at one time. > Google to the rescue again. > > Level 3 provides fiber to Comcast (20 year deal in 2004): > http://blog.tmcnet.com/blog/tom-keating/voip/level-3-and-comcast.asp > > Level 3 provides voice services/support to Comcast: > http://cable.tmcnet.com/news/2005/jul/1168088.htm > > Perhaps the most interesting though is looking up an IP on Comcast's > local network here in my city in L3's looking glass: > http://lg.level3.net/bgp/bgp.cgi?site=sjo1&target=68.86.240.141 > > Slightly reformatting for your viewing pleasure, along with my comments: > > Community: North_America > Lclprf_100 > Level3_Customer # Level 3 thinks they are a customer > United_States > San_Jose > EU_Suppress_to_Peers > Suppress_to_AS174 # Cogent > Suppress_to_AS1239 # Sprint > Suppress_to_AS1280 # ISC > Suppress_to_AS1299 # Telia > Suppress_to_AS1668 # AOL > Suppress_to_AS2828 # XO > Suppress_to_AS2914 # NTT > Suppress_to_AS3257 # TiNet > Suppress_to_AS3320 # DTAG > Suppress_to_AS3549 # GBLX > Suppress_to_AS3561 # Savvis > Suppress_to_AS3786 # LG DACOM > Suppress_to_AS4637 # Reach > Suppress_to_AS5511 # OpenTransit > Suppress_to_AS6453 # Tata > Suppress_to_AS6461 # AboveNet > Suppress_to_AS6762 # Seabone > Suppress_to_AS7018 # AT&T > Suppress_to_AS7132 # AT&T (ex SBC) > > So it would appear Comcast is a transit customer of Level 3 (along with > buying a lot of other services from them). I'm going to speculate that > the list of supressed ASN's are peers of both Level 3 and Comcast, and > Comcast is going that so those peers can't send some traffic through > Level 3 in attempt to game the ratios on their direct connections to > Comcast. > > Now a more interesting picture emerges. Let me emphasize that this is > AN EDUCATED GUESS on my part, and I can't prove any of it. > > Level 3 starts talking to Netflix, and offers them a sweetheart deal to > move traffic from Akamai to Level 3. Part of the reason they are > willing to go so low on the price to Netflix is they will get to double > dip by charging Netflix for the bits and charging Comcast for the bits, > since Comcast is a customer! But wait, they also get to triple dip, > they provide the long haul fiber to Comcast, so when Comcast needs more > capacity to get to the peering points to move the traffic that money > also goes back to Level 3! Patrick, from Akamai, is unhappy at losing > all the business, and/or bemused at the ruckus this will cause and > chooses to kick the hornets nest on NANOG. > > One last thing, before we do some careful word parsing. CDN's like > Akamai and LimeLight want to be close to the end user, and the > networks with end users want them to be close to the end user. It > doesn't make sense to haul the bits across the country for any party > involved. Akamai does this by locating clusters inside providers > networks, LimeLight does it by provisioning bandwidth from their > data centers directly to distribution points on eyeball networks. > > So let's go back and look at the public statements now: > > Level 3 said: > http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions > -2010-11-29?reflink=MW_news_stmp > > "On November 19, 2010, Comcast informed Level 3 that, for the first > time, it will demand a recurring fee from Level 3 to transmit Internet > online movies and other content to Comcast's customers who request such > content." > > Comcast said: > http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html > > "Comcast has long established and mutually acceptable commercial > arrangements with Level 3's Content Delivery Network (CDN) competitors > in delivering the same types of traffic to our customers. Comcast > offered Level 3 the same terms it offers to Level 3's CDN competitors > for the same traffic." > > You can make both of these statements make sense if the real situation > is that Comcast told Level 3 they needed to act like a CDN if they were > going to host Netflix. Rather than having Comcast pay as a customer, > they needed to show up in various Comcast distribution centers around > the country where they could drop traffic off "locally". To Comcast > this is the same deal other CDN's get, it matches their statement. To > Level 3, this means paying a fee for bandwidth to these locations, and > being that they are Comcast locations it may even mean colocation fees > or other charges directly to Comcast. Comcast said if you're not going > to show up and do things like the CDN players then we're going to hold > you to a reasonable peering policy, like we would anyone else making us > run the bits the old way. > > The most interesting thing to me about all of this is these companies > clearly had a close relationship, fiber, voice, and IP transit all on > long term deals. If my speculation is right I'm a bit surprised Level 3 > would choose to piss off such a long term large customer by bringing > Netflix to the party like this, which is one of the reasons I doubt my > speculation a bit. > > But, to bring things full circle, neither of the public statements tell > the whole story here. They each tell one small nugget, the nugget that > side wants the press to run with so they can score political points. > > Business is messy, and as I've said throughout this thread this isn't > about peering policies or ratios, there are deeper business interests > on both sides. This article: > http://www.marketwatch.com/story/level-3-outlines-network-expansion-on-netflix-pact-lvlt-nflx-llnw-akam-2010-11-11 > > Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix. > I'm sure a lot of that is going to Comcast users, since they are the > largest residential broadband ISP. > > Messy. Very messy. > > -- > Leo Bicknell - bickn...@ufp.org - CCIE 3440 > PGP keys at http://www.ufp.org/~bicknell/ >