RE: Cogent/Level 3 depeering (philosophical solution)
[EMAIL PROTECTED] (David Schwartz) writes: I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. It is? For everybody? For always? That's a BIG statement. Can you justify? In those cases where it in fact is and there's nothing you can do about it, you need to accept it. You should not expect to be able to shift the burden of carrying your customers' traffic on your network to others. (The fact that you can sometimes bully or blackmail and get away with it doesn't justify it.) ... The question is whether the benefit to each side exceeds their cost. Yea, verily. But I don't think you'll find a one-cost-fits-all model. When one person's costs are lower than another and they're doing similar things, it's often called efficiency or competitiveness. (Just as one example.) I heartily agree. My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. There can certainly be legitimate peering disputes about where to peer and whether there are enough peering points. If someone wants you to peer with them at just one place, it would certainly be more cost-effective for you to reach them through a transit provider you meet in multiple places, for example. (You could definitely refuse settlement-free peering if it actually increases your costs to reach the peer.) I am not making the pie-in-the-sky argument that everyone should peer with everyone else. I am specifically rejecting the argument that a traffic direction imbalance is grounds for rejecting settlement-free peering. If your customers want to receive traffic and receiving is more expensive, then that's what they're paying you for. Again, carrying *your* customers' traffic over *your* network is what *your* customers are paying *you* for. If your customers want more expensive traffic, you should bear that greater burden. A traffic direction imbalance is not reasonable grounds for rejecting SFI. The direction your customers want their traffic to go is more valuable and it's okay if it costs more. DS
Re: Cogent/Level 3 depeering (philosophical solution)
Peering Ratios? It is very timely that the upcoming NANOG Peering BOF X in Los Angeles will have a debate on this very subject: Traffic Ratios - a valid settlement metric or dinosaur from the dot.bomb past. I'm sure the strongest arguments from these threads will be clearly articulated (in a bullet point/summarized form I hope) during the debate by the debaters. At the end of the day, as with most things peering, the focus of this discussion is a meld of business and technical interests. The heat we have witnessed is probably more related to the friction of the business interests. We get very upset about the notion of fair don't we. Perhaps in the few structured minutes of the Peering BOF debate we can objectively hear both sides of this argument and provide a little light as well. Defending Traffic Ratios as a valid peering prereq: Peter Cohen Attacking Traffic Ratios as peering prereq: Richard Steenbergen Should be good fun. Bill On 10/10/05, David Schwartz [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (David Schwartz) writes: I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. It is? For everybody? For always? That's a BIG statement. Can you justify? In those cases where it in fact is and there's nothing you can do about it, you need to accept it. You should not expect to be able to shift the burden of carrying your customers' traffic on your network to others. (The fact that you can sometimes bully or blackmail and get away with it doesn't justify it.) ... The question is whether the benefit to each side exceeds their cost. Yea, verily. But I don't think you'll find a one-cost-fits-all model. When one person's costs are lower than another and they're doing similar things, it's often called efficiency or competitiveness. (Just as one example.) I heartily agree. My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. There can certainly be legitimate peering disputes about where to peer and whether there are enough peering points. If someone wants you to peer with them at just one place, it would certainly be more cost-effective for you to reach them through a transit provider you meet in multiple places, for example. (You could definitely refuse settlement-free peering if it actually increases your costs to reach the peer.) I am not making the pie-in-the-sky argument that everyone should peer with everyone else. I am specifically rejecting the argument that a traffic direction imbalance is grounds for rejecting settlement-free peering. If your customers want to receive traffic and receiving is more expensive, then that's what they're paying you for. Again, carrying *your* customers' traffic over *your* network is what *your* customers are paying *you* for. If your customers want more expensive traffic, you should bear that greater burden. A traffic direction imbalance is not reasonable grounds for rejecting SFI. The direction your customers want their traffic to go is more valuable and it's okay if it costs more. DS -- // // William B. Norton [EMAIL PROTECTED] // Co-Founder and Chief Technical Liaison, Equinix // GSM Mobile: 650-315-8635 // Skype, Y!IM: williambnorton
Re: Cogent/Level 3 depeering (philosophical solution)
[EMAIL PROTECTED] (David Schwartz) writes: My point is simply that the your customers are getting more out of our network that our customers are argument is bull. Your customers are paying you to carry their traffic over your network. whenever you think you have a reasonable design, you can concept-test it for the internet by asking, what if six million people did this? i suspect that absent peering requirements, there would be a lot of WAN ISO-L2 and on-net ISO-L3 sold, a lot more ASN's on the hoof, and a bit less stability in the BGP core. since most of the transit ISO-L3 providers are also in the on-net ISO-L3 or WAN ISO-L2 (or both) business, the end result would be the same people getting paid the same amounts by the same other people, but called something else than what we call it now. maybe this would be better than my network is bigger!, no it ain't!, etc? -- Paul Vixie
Re: Cogent move without renumbering (was: Cogent/Level 3 depeering)
in a pay-me-now-or-pay-me-later scenario, you have to pick now vs. later. (it's a pity that the internet, for all its power, cannot alter that rule.) It should be noted that if one opts for 'later', you can do quick and dirty games with NAT. Do not renumber, change providers and put a NAT between yourself and your provider. This will continue to work until such time as your original PA space is reassigned and then you will not be able to reach the new assignee. This allows for quick moves, but creates the mortgage of an eventual renumbering. Folks who take this approach are likely to renumber into RFC 1918 space. Before you break out the blowtorches, I'm *not* claiming that this a good way of doing things. It's a hack. It's expedient. ;-) Regards, Tony
Re: Cogent/Level 3 depeering
On Sat, 8 Oct 2005, [EMAIL PROTECTED] wrote: On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Sure, but consider is it worse to have a very small number of complaining customers who cant get to a bit of the web for 2 or 3 days, or a complete outage to the Internet for a few hours because of a problem you cant fix. I see the latter occurring quite frequently, in particular I see support queries about loss of connectivity to large parts of the Internet which on inspection was caused by dampening because the ISP was flapping. I'm just saying, you fix one problem and create a whole bunch of new ones and it depends on the customer as to which results in the optimum situation. Steve
How to multihome endusers [was: Cogent/Level 3 depeering]
Yes, indeed, I think it makes sense to multihome my humble enduser pc. Right now all I can get is aDSL and it does not matter what provider because they all use DTAG.DE infrastructure. Maybe cable will be choce. It is not as fast as aDSL at least not here and it will take another two or three years until they deploy it. If it does not get shot on site again by the regulation office or the cartell office again. So I will end up having a cable-modem speaking ethernet/PPPoE and an aDSL-modem speaking ethernet/ tcp/ip and DHCP. My ip adresses probably will be 84.167.xxx.xxx for aDSL and 24.xxx.xxx.xxx for the cable. I can talk to no-ip.com, they will allow a second ip for host_look(84.167.252.166,echnaton.serveftp.com,1420295334). host_name(84.167.252.166,p54A7FCA6.dip.t-dialin.net). Its entry will look a bit like this one: host_look(81.88.34.51,Kunden2.KONTENT.de,1364730419). host_name(81.88.34.51,kunden2-1.kontent.de). host_look(81.88.34.52,Kunden2.KONTENT.de,1364730420). host_name(81.88.34.52,kunden2-2.kontent.de). So I will end up with 3 names and 2 ip addresses for my humble host. Do I need BGP now or OSPF or can I rely on RIP. Do I need an AS number? How do I get it? Imagine not a fool like me is asking this but some 32K end users of DTAG.DE connected to a DSLAM at Franfurt/Main in germany. I guess the number of end users disconnected be Cogent and Level 3 is not much smaller. Asbestos parapluis opened. Shoot now! Peter and Karin Dambier :) [EMAIL PROTECTED] wrote: On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Anybody want to start a pool on how many new AS numbers will get issued as a result of this tiff, and what percent will commit a BGP whoopsie that impacts more than just themselves within the first 6 months? On the other hand, I see a business opportunity to sell new customers insurance against self-inflicted gunshot wounds to the feet here. Some providers might even consider selling a managed service at a slight loss, just for self-defense.. :) -- Peter and Karin Dambier Public-Root Graeffstrasse 14 D-64646 Heppenheim +49-6252-671788 (Telekom) +49-179-108-3978 (O2 Genion) +49-6252-750308 (VoIP: sipgate.de) +1-360-448-1275 (VoIP: freeworldialup.com) mail: [EMAIL PROTECTED] http://iason.site.voila.fr http://www.kokoom.com/iason
Re: How to multihome endusers [was: Cogent/Level 3 depeering]
Look into multi6 - which basically proposes new network layer above ip but below tcp and that new layer would provide common end-point for system with multiple ip addresses. A closer possibility right now is dns multi-homing based on incoming request ip, i.e. dns server would answer with one provider ip address if they are coming from cogent routed ip space and for another from l3 routed ip space. This requires integration of bgp routing data with dns which has only been done by private implementations (I'm sure you all know who I mean) so far, but it would be a worthy project to do a open-source implementation of this technique if fragmentation of the internet continues to happen or becomes permanent. On Sun, 9 Oct 2005, Peter Dambier wrote: Yes, indeed, I think it makes sense to multihome my humble enduser pc. Right now all I can get is aDSL and it does not matter what provider because they all use DTAG.DE infrastructure. Maybe cable will be choce. It is not as fast as aDSL at least not here and it will take another two or three years until they deploy it. If it does not get shot on site again by the regulation office or the cartell office again. So I will end up having a cable-modem speaking ethernet/PPPoE and an aDSL-modem speaking ethernet/ tcp/ip and DHCP. My ip adresses probably will be 84.167.xxx.xxx for aDSL and 24.xxx.xxx.xxx for the cable. I can talk to no-ip.com, they will allow a second ip for host_look(84.167.252.166,echnaton.serveftp.com,1420295334). host_name(84.167.252.166,p54A7FCA6.dip.t-dialin.net). Its entry will look a bit like this one: host_look(81.88.34.51,Kunden2.KONTENT.de,1364730419). host_name(81.88.34.51,kunden2-1.kontent.de). host_look(81.88.34.52,Kunden2.KONTENT.de,1364730420). host_name(81.88.34.52,kunden2-2.kontent.de). So I will end up with 3 names and 2 ip addresses for my humble host. Do I need BGP now or OSPF or can I rely on RIP. Do I need an AS number? How do I get it? Imagine not a fool like me is asking this but some 32K end users of DTAG.DE connected to a DSLAM at Franfurt/Main in germany. I guess the number of end users disconnected be Cogent and Level 3 is not much smaller. Asbestos parapluis opened. Shoot now! Peter and Karin Dambier :) [EMAIL PROTECTED] wrote: On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Anybody want to start a pool on how many new AS numbers will get issued as a result of this tiff, and what percent will commit a BGP whoopsie that impacts more than just themselves within the first 6 months? On the other hand, I see a business opportunity to sell new customers insurance against self-inflicted gunshot wounds to the feet here. Some providers might even consider selling a managed service at a slight loss, just for self-defense.. :) -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: How to multihome endusers [was: Cogent/Level 3 depeering]
On Sun, 9 Oct 2005, william(at)elan.net wrote: Look into multi6 - which basically proposes new network layer above ip multi6 is dead, long live shim6... attend and discuss in Vancouver. (also, I'm fairly sure it's not going to help if you only have a single provider)
Re: Cogent/Level 3 depeering (philosophical solution)
[EMAIL PROTECTED] (David Schwartz) writes: I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. It is? For everybody? For always? That's a BIG statement. Can you justify? ... The question is whether the benefit to each side exceeds their cost. Yea, verily. But I don't think you'll find a one-cost-fits-all model. When one person's costs are lower than another and they're doing similar things, it's often called efficiency or competitiveness. (Just as one example.) -- Paul Vixie
Re: Cogent/Level 3 depeering (philosophical solution)
On Oct 8, 2005, at 7:02 AM, David Schwartz wrote: Various people have stated that uneven data flows (e.g. from mostly-content networks to mostly-eyeball networks) is a good reason to not peer. I think the industry simply needs to accept that it's more expensive to receive traffic than to send it. But it is not. It is more expensive to carry a large packet a long way than to carry a small packet a long way. Because of things like hot-potato routing, that frequently means the sender has less cost than the receiver, depending on where they meet. The rest of your argument is based on the premise that none of this is changeable. Which is clearly wrong. Receivers have been de-peering Senders for over half a decade. (I.e. Forever in Internet time.) These fights have been fixed by things like sending MEDs or intentionally recruiting customers to balance traffic for a long, long time. Of course, there is nothing wrong with an eyeball network saying I'll carry it, gimme gimme! But that doesn't mean they have to. Yes, that can't possibly work. It's way too simple and actually makes sense. No, it can't work because you assume things which are not necessarily factual. Not to mention, it doesn't make sense. -- TTFN, patrick
Re: Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
On Fri, 7 Oct 2005, William Allen Simpson wrote: Stephen J. Wilcox wrote: On Fri, 7 Oct 2005, William Allen Simpson wrote: Rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, Mr Wilcox. the contract talks of on-net traffic, off-net traffic and excused outages excused outages includes that of third party network providers off-net traffic has a 99% SLA excluding excused outages. Again, rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, not your summary of an index of definitions, Mr Wilcox. that was it, i just shortened it For instance, I rather doubt that the contract language defines a decision of L(3) to terminate connectivity to a third party as an excused outage. But we won't know without the contract. Enlighten us. excused outages ... includes ... third party network providers it doesnt go anywhere talking about peerings or specifics of the connectivity, but it seems to me that the ability to pass traffic to cogent falls right in this get out clause as it is a third party ianal but i'd push to break contract rather than sue Level3 as the latter seems to be a very big gamble Steve
Re: Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
On Fri, 7 Oct 2005, [EMAIL PROTECTED] wrote: Seems to me that the ideal here would be for the industry to agree on a dispute resolution mechanism and for all bilateral peering agreements to include the same arbitration clause. For this kind of arbitration to function well, the arbitrators need to have some understanding of the industry and the technology. This can only be accomplished by selecting one arbitration organization to handle all the arbitration duties for the whole industry. the trouble is that there is no regulatory requirement of peering, there is no accepted standard for peering, the definition of fair varies greatly and the policies that exist are based on many criteria and personalities the problem that would arise as i see it is that such an arbitrator would be consistent with its decisions but that would be consistently right for one player and consistently wrong for another.. and if we apply that to the current scenario we can see arguments for both cogent and level3s positions Airing dirty landry in public like this hurts the whole industry, not just Level 3 and Cogent in particular. The solution is to use binding arbitration clauses in all interconnect agreements whether settlement-free, paid peering or settlement-based. i'm not sure the industry does get hurt, to us this is a major incident, but in reality there appears to only be a handful of affected customers and its not getting much attention from the press someone implied this might work in the favour of non-tier-1 networks so if that were true that would be a benefit to such networks! Steve
Re: Cogent/Level 3 depeering
On Fri, 7 Oct 2005, Daniel Golding wrote: On 10/6/05 10:37 AM, Patrick W. Gilmore [EMAIL PROTECTED] wrote: On Oct 6, 2005, at 10:19 AM, tony sarendal wrote: This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. s/every time it happens/every time it happens to YOUR upstream People on Sprint, ATT, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity. Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) disagree. i know networks who multihome to avoid this kind of problem but introduce new problems with greater risk because they are unable to run bgp properly (be it from inadequate hardware, bad config, bad administration) my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management Steve
Re: Cogent/Level 3 depeering
On Sat, 08 Oct 2005 20:41:55 BST, Stephen J. Wilcox said: my rule would be if your provider can manage an autonomous system better than you and multihoming isnt a requirement of your business then let them take on the management I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. Anybody want to start a pool on how many new AS numbers will get issued as a result of this tiff, and what percent will commit a BGP whoopsie that impacts more than just themselves within the first 6 months? On the other hand, I see a business opportunity to sell new customers insurance against self-inflicted gunshot wounds to the feet here. Some providers might even consider selling a managed service at a slight loss, just for self-defense.. :) pgpNa6EDhFhJX.pgp Description: PGP signature
Re: Cogent/Level 3 depeering
Take-away: Do not single home. ... so, CIDR was a bad idea, and we should push forward with one AS per end-site and a global routing table of 500 million entries? I think that's unnecessarily one dimensional. The needs of business to be connected in a reliable fashion are above and beyond being for or against CIDR. Rather, they are the requirements for the routing architecture that the Internet has yet to fulfill. well, sure, my answer is only valid if pigs do not have rocket boosters. if you're going to talk about how fast pigs could fly if they had rocket boosters then i'm very interested but i consider it a change of subject. Single homing is bad simply from a reliability standpoint, and the only true technological impediment to everyone multi-homing is cost and the routing architecture. Consider the ability of the average consumer to make use of WiFi to provide mutual backup connectivity to his neighbors with alternate last mile providers. As the cost goes to zero, everyone will want to multi-home. yea, verily, that is so. but do not single home is not practical advice as of the date on this particular milk carton. unless you can do it without bgp. see http://www.google.com/search?client=safarirls=en-usq=vixie+multihoming+without+bgpie=UTF-8oe=UTF-8 if you're wondering how long we've been fiddling around with THAT tune here. -- Paul Vixie
Re: Cogent/Level 3 depeering
I'm willing to bet there's a lot of single-homed customers of both Cogent and L3 that 2 weeks ago didn't think multihoming was a requirement of their business either, who now are contemplating it. Plus possibly some single-homed customers of other large providers as well. any ISP likely to be involved in a peering dispute is a reliability risk, and whether it's because others keep de-peering them or because they keep de-peering others, doesn't matter. i liked the advice heard here the other day-- if you have to single home, do it through a tier-2 or tier-1.5 ISP without transit-free aspirations. they'll remain connected to the riskier ISP's no matter what the riskier ISP's are doing to each other this week. -- Paul Vixie
Re: Cogent/Level 3 depeering
On Thu, Oct 06, 2005 at 10:54:37PM -0700, JC Dill wrote: Various people have stated that uneven data flows (e.g. from mostly-content networks to mostly-eyeball networks) is a good reason to not peer. I'd love to know how it improves Level 3's network to have data from Cogent arrive over some *other* connection rather than directly from a peering connection. Do they suddenly, magically, no longer have backhaul that mostly-content data across their own backbone to their users who have requested it if it should come in from one of their *other* peers who (in normal peering fashion) hot-potato hands it off to them at the first opportunity, rather than coming in directly from Cogent? I don't think so. So why break off peering??? AFAICT there's only one reason to break off peering, and it's to force Cogent to pay (anyone) to transit the data. Why does L3 care if Cogent sends the data for free via peering, or pays someone ELSE to transit the data? First off, why do you assume that peering is a right to which people are entitled? Level 3 operates a network, a pretty darn big and successful one (well big at any rate), and it apparently does a good job delivering the bits or it wouldn't have as many customers as it does. Why *must* they give another network free access to their network if they don't feel that it is mutually beneficial? You seem to be making the argument that because YOU think it is mutually beneficial, therefore (3) must be doing something evil. Beneficial is in the eye of the beholder, and Cogent could just as easily have been the one to decide it wasn't beneficial to them. Second, there are serious some serious fallacies in the argument that the bits have to go there anyways. The vast majority of the Internet is multihomed in one way or another, especially the closer you get to the big Tier 1's. By my count, Level 3 has over 57,000 customer prefixes. I don't have any data about how many of those Cogent was actually using before-hand (though I'm sure someone does), but we know that roughly 4,000 some prefixes were single homed and couldn't be reached any other way. I'd be willing to venture a guess that while Cogent was probably not using anywhere near 57,000 prefixes from (3), they were using a heck of a lot more than 4,000 before the depeering notice and traffic depref, and that those prefixes were substantially trafficked. There is also this thing called BGP path selection, and one of the important criteria is AS-PATH length, so making the path to 3356 longer may have diverted a significant amount of traffic away from Cogent at the source of their multihomed customers. A depeering would combine both effects, Cogent would immediately depref, and the longer AS-PATH would shift traffic away from this path. Even if reaching (3) didn't cost Cogent a dime, depeering may still have been a viable method of traffic engineering. The bottom line is that we have no idea what was really going on, but there are loads of reasons why (3) would want to depeer Cogent that don't have anything to do with forcing them to pay for transit. I don't think anyone who depeers another network actually expects to see a dime in transit business from the depeered network any time in the immediate future anyways. I think this is about a big bully trying to force a smaller player off of the big guys' playing field (tier 1 peering). From where I sit it looks like an anti-competitive move that is not a best effort to serve their customers but a specific effort to put another (smaller) competitor out of business (of being a transit-free or mostly transit-free backbone) by forcing them to pay (someone), forcing their costs up. Level 3 must know they are no longer putting for a best effort for their own customers to connect them to the internet (as their customers see it, the complete internet that their customers have come to expect). If Cogent can't stay in business if (3) decides that there is no benefit to giving them free access to all of their customers, and infrastructure necessary to deliver it, they were never a peer to begin with. Peering is about mutual benefit, not entitlement. Cogent makes the same value judgements when it decides if it is going to peer with another network or not, it is no different here than it is there. Cogent routinely turns away smaller peers for peering and suggests that they buy transit instead, are you going to accuse them of anti-competetive practices next? I also believe that Cogent has a valid argument that Level 3's behavior is anti-competitive in a market where the tier 1 networks *collectively* have a 100% complete monopoly on the business of offering transit-free backbone internet services. As such, L3's behavior might fall into anti-trust territory - because if Cogent caves in over this and buys transit for the traffic destined for L3 then what's to stop the rest of the
Re: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, JC Dill wrote: to pay (anyone) to transit the data. Why does L3 care if Cogent sends the data for free via peering, or pays someone ELSE to transit the data? Anything to increase a competitors spending must be good, right? The more expenses a competitor has, the higher their price when they sell to customers, and the less likely they are to take potential customers? -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
RE: Peering vs SFI (was Re: Cogent/Level 3 depeering)
Would you care to speculate on which party receives the greater benefit: the sender of bytes, or the receiver of bytes? Nope! I'll let the economists argue about that question. Probably on some other list where people know a lot more about the issue of value than on this list. --Michael Dillon
Re: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, tony sarendal wrote: On 06/10/05, Patrick W. Gilmore [EMAIL PROTECTED] wrote: On Oct 6, 2005, at 10:19 AM, tony sarendal wrote: This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. s/every time it happens/every time it happens to YOUR upstream People on Sprint, ATT, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity. Now, is being a tier-1 now a good or bad sales argument when selling internet access ? It's still a good argument, because Marketing != Reality. :) Patrick, it happens to every PA customer who buys his service from one of the Tier-1 providers active in the de-peering. If a PA customer buys his service from a non-tier1 this will most likely not happen, unless that provider has bought transit in a very unwise way. The entire point is that it's not always good to be too close to tier-1 space. See my other post tho, connectivity disputes and problems can arise between any networks, being tier-1 isnt special.. anyone can choose not to give access or send routes to any other network. Steve
Re: Cogent/Level 3 depeering
On 05 Oct 2005, at 13:44 , Charles Gucker wrote: Oh man, I have to jump in here for a moment. Not that I agree with what happened, but to refute your claim that Cogent can get L3 elsewhere, it goes both ways. L3 can also get Cogent connectivity elsewhere. This is a big game of chicken, it will be interesting to see who backs down first. Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? I mean, Cogent can pay someone to route to L3 or L3 can fix what they did on their side, they have no need to go anywhere but their own routers, right? -- Lewis Butler, Owner Covisp.net 240 S Broadway #203, 80209 mobile: 303.564.2512 fx: 303.282.1515 AIM/ichat: covisp xdi: http://public.xdi.org/=lewisbutler
Re: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, JC Dill wrote: Alex Rubenstein wrote: Further, the internet has always been a best-effort medium. Can someone please explain how Level 3 is making a best effort to connect their customers to Cogent's customers? thats not what alex means as you know. and Level(3)/Cogent are playing a pain game here, its 'no effort' not 'best effort' Various people have stated that uneven data flows (e.g. from mostly-content networks to mostly-eyeball networks) is a good reason to not peer. I'd love to know how it improves Level 3's network to have data from Cogent arrive over some *other* connection rather than directly from a peering connection. Do perhaps the other connection is already carrying significant outbound so this extra inbound is a small net cost, that would support L3's argument So why break off peering??? this is about politics not engineering, dont try to confuse them. peering often is. AFAICT there's only one reason to break off peering, and it's to force Cogent to pay (anyone) to transit the data. Why does L3 care if Cogent sends the data for free via peering, or pays someone ELSE to transit the data? the economics are different for cogent, cogent loses some marketing advantage.. i can think of other reasons I think this is about a big bully trying to force a smaller player off of the big guys' playing field (tier 1 peering). From where I sit it cogent isnt a small player, they are a real threat to L(3).. dont feel sorry for them, they're not being bullied! looks like an anti-competitive move that is not a best effort to serve their customers but a specific effort to put another (smaller) competitor out of business (of being a transit-free or mostly transit-free backbone) by forcing them to pay (someone), forcing their really? you mean one company wants to take business from the other company? thats amazing.. and i thought ISPs existed together in harmony never looking at each others customer bases IMHO all L3 customers have a valid argument that Level 3 is in default of any service contract that calls for best effort or similar on L3's part. can you cite the relevant clause in your Level3 contract that brings you to this conclusion.. hint: you might be looking a long time because it doesnt exist and they're not in breach I also believe that Cogent has a valid argument that Level 3's behavior is anti-competitive in a market where the tier 1 networks *collectively* have a 100% complete monopoly on the business of offering transit-free backbone internet services. As such, L3's behavior might fall into anti-trust territory - because if Cogent caves in over this and buys transit for the traffic destined for L3 then what's to stop the rest of the tier 1 guys from following suit and forcing Cogent to buy transit to get to *all* tier 1 networks? Then who will they (TINT) force out next? these are big companies, they can fight their own battles. there is no tier-1 monopoly. in many cases its cheaper to send data via transit than peering so why do you care about transit-free anyway? What's to stop a big government (like the US) from stepping in and attempting to regulate peering agreements, using the argument that internet access is too important to allow individual networks to bully other networks out of the market - at the expense of customers - and ultimately resulting in less competition and higher rates? Is this type of regulation good for the internet? OTOH is market consolidation good for the internet? they're not acting illegally or as a monopoly, and theres no anti-trust so theres no reason to expect any government interventions. Steve
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
While I realize that the nuke survivable thing is probably an old wives tale, it seems ridiculous that the Internet can't adjust by [...] It's not a myth. If the Internet were running RIP instead of BGP For the Internet, I believe it was indeed a myth. I wasn't there, but according to someone who was: http://www.postel.org/pipermail/end2end-interest/2004-April/003940.html We'll probably never resolve this question entirely, but a simple internetwork (partial mesh, not too big) running RIP does seem to be able to survive in the face of multiple failures. Presumably, the network view of a nuclear war would be multiple failures. In any case, I think that you have to go further back to find the roots of this story. Paul Baran came up with the basic ideas of packet-switching and partial mesh networks which are the foundation of the Internet. There is a nice explanation of this on his bio page here: http://www.ibiblio.org/pioneers/baran.html I think Dave Reed should have just said to the reporter that the Internet survived 9/11 so well because it was largely a non-centralized network that does not depend on any kind of central traffic control. It's like a road network where every driver(packet) is free to detour around obstructions. Remember the information highway? --Michael Dillon
Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
Stephen J. Wilcox wrote: On Thu, 6 Oct 2005, JC Dill wrote: IMHO all L3 customers have a valid argument that Level 3 is in default of any service contract that calls for best effort or similar on L3's part. can you cite the relevant clause in your Level3 contract that brings you to this conclusion.. hint: you might be looking a long time because it doesnt exist and they're not in breach Rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, Mr Wilcox. I don't have a L(3) contract. Merit seems to have one with Cogent (and mine is with Merit), but I don't have ready access to that one, either. We'll need to see the contractual language before embarking on a concerted effort. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
Re: Cogent/Level 3 depeering
On Fri, Oct 07, 2005 at 01:29:06AM -0400, Patrick W. Gilmore wrote: You also forgot that Providers A B have to pay cab fare to get to those geographically dispersed corners. One might have to take the cab a lot longer than the other, incurring more time money. You also forgot ... well, about 14 other things. Ok Patrick, the analogies are killing me. -- Jay Adelson
Re: Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
On Fri, 7 Oct 2005, William Allen Simpson wrote: Stephen J. Wilcox wrote: On Thu, 6 Oct 2005, JC Dill wrote: IMHO all L3 customers have a valid argument that Level 3 is in default of any service contract that calls for best effort or similar on L3's part. can you cite the relevant clause in your Level3 contract that brings you to this conclusion.. hint: you might be looking a long time because it doesnt exist and they're not in breach Rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, Mr Wilcox. the contract talks of on-net traffic, off-net traffic and excused outages excused outages includes that of third party network providers off-net traffic has a 99% SLA excluding excused outages. Steve
Re: Cogent/Level 3 depeering
Jay Adelson wrote: On Fri, Oct 07, 2005 at 01:29:06AM -0400, Patrick W. Gilmore wrote: You also forgot that Providers A B have to pay cab fare to get to those geographically dispersed corners. One might have to take the cab a lot longer than the other, incurring more time money. You also forgot ... well, about 14 other things. Ok Patrick, the analogies are killing me. [EMAIL PROTECTED] echo godwins law | sed -e s/nazis/analogies/g
Re: Cogent/Level 3 depeering
On Fri, Oct 07, 2005 at 03:17:53AM -0400, Richard A Steenbergen wrote: On Thu, Oct 06, 2005 at 10:54:37PM -0700, JC Dill wrote: AFAICT there's only one reason to break off peering, and it's to force Cogent to pay (anyone) to transit the data. Why does L3 care if Cogent sends the data for free via peering, or pays someone ELSE to transit the data? First off, why do you assume that peering is a right to which people are entitled? I didn't read that... I think the point is more why not, not Level3 is evil. The bottom line is that we have no idea what was really going on, but there are loads of reasons why (3) would want to depeer Cogent that don't have anything to do with forcing them to pay for transit. I don't think anyone who depeers another network actually expects to see a dime in transit business from the depeered network any time in the immediate future anyways. There is the issue of precedent, which neither network wants to set. If you cave on a game of chicken, others will follow suit, like a bad mass adoption of norton's art of peering. What caused the initial chess move, as you say, is hard to fathom, until we hear a statement on it from Level3. not, it is no different here than it is there. Cogent routinely turns away smaller peers for peering and suggests that they buy transit instead, are you going to accuse them of anti-competetive practices next? I'm not sure the analogy applies here, given the size of Cogent. Hopefully, sanity. Whether you believe it or not, the Internet *IS* best effort. If you don't like it, or you aren't happy with it, buy from someone else who makes a better effort. Speaking of slippery slopes, how would you like to not be able to block traffic to your network from people you consider to be spam sites, because you are harming global reachability and potentially trying to force those sites out of business? Same argument, different benefit for you. Unfortunately, Richard, you are using logic. What happens when depeering hits the public press is that the federal government starts watching it again. We've seen this over and over, and unfortunately with govt. churn, the same questions and hearings transpire. While I agree with the best effort statement, historically the govt./doj has started to get involved when mass standoffs occur like this one. -- Jay Adelson
Re: Cogent/Level 3 depeering
In a message written on Thu, Oct 06, 2005 at 06:36:00PM -0400, Lamar Owen wrote: All philosophy aside, it does bother me that a simple single depeering can cause such an uproar in a network supposedly immune to nuclear war (even though the Internet was not designed from the start to survive nuclear war; I point you back a few weeks to when the hurricane hit. You need look no further to see people offering up their assistance to those in need. Look back further to 9-11, and people offering networking help to those who's infrastructure was damaged. In both of those cases, how much core infrastructure was damaged? (I've read the threads on this list (with interest, particularly some of the comments about many core paths from Atlanta to Houston not tranisting NOLA), and have read the archives of several lists on the earlier event; when the backbone providers want to do so, they can cooperate very nicely. When they don't want to do so, it gets ugly). If a tornado took out a major peering point, that would be different. I have no doubt that if the Level 3 / Cogent issue had been caused by a pre-emptive nuclear strike and the nation was called to arms that virtually every ISP that connects to both would be offering them free transit to get them reconnected. Yes, you would be correct. Which offers an interesting thought: why would it be important for you then but not now? If the issue impacts your customers, then why not spend the 3 minutes reconfiguring your router(s)? (obviously, if it doesn't impact your customers, then ignore that). Indeed, I could log into my routers now and fix the Cogent / Level 3 problem with about 3 minutes of typing. It would cost my company thousands of dollars to do so, so I'm not going to do it. In other words, this problem is a problem simply because people can't be bothered to fix the problem because it's just a customer service issue, and not 'helping out fellow backbone providers?' Shades of the old backbone cabal here. (yes, a healthy dose of cynicism there) As I said before, right now this is a business problem. Absolutely. But who's business? Hint: the two parties involved aren't the only ones with a business stake in this issue. Of course, I'm not telling you something you don't already know. I've got a new set of rules to add to this thread: If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment. Again, shades of the old backbone cabal. How do you know that I have neither? I have both, in fact, even though the negotiations for the SFI didn't pan out due to regulated carrier issues (after all, one must arrange transit to a peering point, and, while I have the Cisco 12000's sitting here, they are here and not there). You really don't understand what's going on here, and it's not, I repeat, not a technical problem. Of course. I fully understand that. There is nothing wrong with the technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies. There is something very wrong with the whole business if two players' business models and business decisions can make this much of an uproar. When other businesses won't help their customers see the illusion of it being fixed, something is wrong. That's why you multihome in the first place. And if you have customers, and you are single-homed, your business plan stinks anyway. That goes for content consumer as well as content producer customers. But you are very right; it is not a technical problem and never was. Why? Because technical problems don't typically take this long to fix, and, as you said, if enough people cared enough this could easily be a nonissue. Aside: love the domain name, Leo. -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
Re: Cogent/Level 3 depeering
In a message written on Fri, Oct 07, 2005 at 10:40:50AM -0400, Lamar Owen wrote: Yes, you would be correct. Which offers an interesting thought: why would it be important for you then but not now? If the issue impacts your customers, then why not spend the 3 minutes reconfiguring your router(s)? (obviously, if it doesn't impact your customers, then ignore that). I venture any other ISP of importance either has direct connectivity to Cogent and Level 3, and/or buys transit from someone who does. All but the smallest most trivial ISP's are multi-homed. Those that are have seen no result from this, by and large. I can all my customers can get to both. In other words, this problem is a problem simply because people can't be bothered to fix the problem because it's just a customer service issue, and not 'helping out fellow backbone providers?' Shades of the old backbone cabal here. (yes, a healthy dose of cynicism there) No, it doesn't affect anyone else's customers. Period. Fixing it in this case would be offering Charity to Level 3 and Cogent, and offering your competitors Charity, particularly for their own mistake is not high on most business plans. There's a very large difference between offering charity to a competitor, and keeping the industry going in the face of disaster. To suggest the two are related at all is just absurd. If someone wants to cut their network off from someone else for a business reason they will be able to do that whatever the design of the network may be. Level 3 and Cogent are both actively causing this outage. It's not some grand design failure. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org
Re: Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
Rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, Mr Wilcox. the contract talks of on-net traffic, off-net traffic and excused outages excused outages includes that of third party network providers off-net traffic has a 99% SLA excluding excused outages. One interesting point that came up in an off-list message that I received was the topic of arbitration. Would anyone be in a position to estimate how many peering agreements include an arbitration clause such as this one: http://www.cidra.org/modelarb.htm I have no relationship with CIDRA, but since they are in Chicago, a major trading center, and they offer international arbitration, I thought they were an appropriate example. In the non-Internet world of telecommunications, the state PUCs handle arbitration of interconnect disputes but that is because this role is embedded in the Telecommunications Act of 1966. Other arbitration services include http://www.adr.org who are tied in to the NAFTA agreements, http://www.cpradr.org/ http://www.usam.com/ http://www.acrnet.org/ Here is a quick summary of US arbitration law http://www.arbitration.co.nz/content.asp?section=Arbitrationcountry=USA Seems to me that the ideal here would be for the industry to agree on a dispute resolution mechanism and for all bilateral peering agreements to include the same arbitration clause. For this kind of arbitration to function well, the arbitrators need to have some understanding of the industry and the technology. This can only be accomplished by selecting one arbitration organization to handle all the arbitration duties for the whole industry. Airing dirty landry in public like this hurts the whole industry, not just Level 3 and Cogent in particular. The solution is to use binding arbitration clauses in all interconnect agreements whether settlement-free, paid peering or settlement-based. --Michael Dillon
Re: Cogent/Level 3 depeering
On 10/6/05 10:30 AM, Randy Bush [EMAIL PROTECTED] wrote: Is being a tier-1 now a good or bad sales argument when selling internet access ? Its a great sales argument. That's why everyone claims to be one. It just sounds SO good. And its not like the Peering Police are going to enforce it. What does it mean in real life? Nothing. Nada. An organization's SFI status is a particularly poor criteria for choosing a transit provider. There are so many better factors to use - support, packet loss, price, latency, availability, provisioning speed - you name it, its a better criteria than SFI status. packet loss and latency to *where*? before replying, consider that most of a leaf's traffic is either to/from another leaf of a tier-1 to which they're (possibly indirectly) downstream, or to/from the tree of a tier-1 which peers with the tier-1 to which they're attached. Consider this: A Tier 1 (SFI network) with congested peering links vs a non-SFI network with wide open transit pipes. I know I'd pick the latter. Latency when all inter-network links are uncongested is going to be pretty low in any case. if tier-n, where n 1, is buying transit from tier-1s, which they have to do, then the price game seems to be pretty determined unless one likes to run at a loss or is cross- subsidizing from some other product line. all your bases are belong to us. :-) randy Dan
Re: Cogent/Level 3 depeering
On 10/6/05 10:37 AM, Patrick W. Gilmore [EMAIL PROTECTED] wrote: On Oct 6, 2005, at 10:19 AM, tony sarendal wrote: This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. s/every time it happens/every time it happens to YOUR upstream People on Sprint, ATT, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity. Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) Now, is being a tier-1 now a good or bad sales argument when selling internet access ? It's still a good argument, because Marketing != Reality. :) Dan
Re: Cogent/Level 3 depeering
*Leo Bicknell [EMAIL PROTECTED] wrote: If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment. You really don't understand what's going on here, and it's not, I repeat, not a technical problem. There is nothing wrong with the technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies. Well, I disagree. I have no clue. I am not an ISP I do, however, have enable. We do buy transit There are lots of folks that have replied to this thread. A number of them have a lot of "clue". Some, like me,don't. If this list didn't have suchlively discussions I would not have the education about how the "Internet" works that this list has given me. If only people with "clue" commented on things this would be a dull place indeed. Might just as well be a closed list. For this, I thank all of you that have contributed.
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
I think Dave Reed should have just said to the reporter that the Internet survived 9/11 so well because it was largely a non-centralized network that does not depend on any kind of central traffic control. It's like a road network where every driver(packet) is free to detour around obstructions. He should have given the the real reason: most of the Internet routers were at the old WUTCO building at 60 Hudson St, a safe distance away from the WTC, while the phone switches were across the street on West St in a building that was severely damaged. Swap those two buildings and the myth would be that the phone system is robust and the Internet is fragile. The phone network reroutes pretty well when the switching equipment that does the routing hasn't been smashed. Regards, John Levine, [EMAIL PROTECTED], Primary Perpetrator of The Internet for Dummies, Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor I shook hands with Senators Dole and Inouye, said Tom, disarmingly.
RE: Cogent/Level 3 depeering
What is Internet? Let's channel Seth Breidbart briefly and call it the largest equivalence class in the reflexive transitive symmetric closure of the relationship can be reached by an IP packet from. It should be clear that the nature and extent of this network depends very much on the perspective of the connected device from which is it measured. At last, a definition we can all agree on! ;) Honestly this might be closest to the truth, but it's not quite the perception that the marauding forces of marketing have encouraged over the previous 10 years. Rather, the market which exists to support ISPs tends not to include people who understand the nature of the network, and its instability. Sadly, for many of the market constituents the Internet equates to the Web; for some of them it equates to a platform to support their applications; for very few of them does it equate to a unique perspective into a subset of possible IP relationships. As I said, this definition is closest to the reality today, but not even everybody on this knowledgeable mailing list feels happy with buying such a service, no less so the end-users at large. Do people in Spain complain that they can't call numbers starting with +350, and insist on getting money back from their monthly bill? Or do they accept that their government has an ongoing dispute with the UK over whether Gibraltar is in fact part of Spain? Good counter-example. Instead of trying to compare how this example of political dispute and the resulting customer satisfaction or frustration is similar to the Cogent-Level(3) situation, I'll simply acknowledge that my analogy, like most, is imperfect. I still hold to my fundamental point, however. The market has evolved to expect more than Internet as an research experiment/hobbyist toy, and now expects the Internet to be a component of their critical infrastructure. Service providers that don't understand this, in addition to having unsatisfied customers, may perhaps incur outside intervention. Would that ultimately be so bad for end-users? Cheers, -Benson --- Benson Schliesser (email) mailto:[EMAIL PROTECTED] I barely understand my own thoughts, much worse those of my betters. Thus, the opinions expressed herein are not necessarily those of my employer. Ponder them at your own risk.
Re: Cogent/Level 3 depeering
On Thu, 06 Oct 2005 22:54:37 PDT, JC Dill said: I also believe that Cogent has a valid argument that Level 3's behavior is anti-competitive in a market where the tier 1 networks *collectively* have a 100% complete monopoly on the business of offering transit-free backbone internet services. As such, L3's behavior might fall into anti-trust territory Please enumerate the tier 1 networks who comprise this collective monopoly. Seriously. Somehow, although civil lawsuits do occasionally name John Does when the actual name is expected to be revealed during pre-trial discovery (usually when the action is known, but the person isn't, as in John Doe, the upper manager in Sales who authorized the tortable activity), I don't see much hope for a lawsuit claiming abuse of a monopoly when you can't name who is a member up front pgpLAd5o1NkS1.pgp Description: PGP signature
RE: Peering vs SFI (was Re: Cogent/Level 3 depeering)
Paul Vixie wrote: [EMAIL PROTECTED] (Schliesser, Benson) writes: Would you care to speculate on which party receives the greater benefit: the sender of bytes, or the receiver of bytes? If both the sender and receiver are being billed for the traffic by their respective (different) service providers (all other issues being equal) is one provider in a better position than the other? If it's still common for one to be billed only for highest of in vs. out then there's no way to compare the benefits since there's always a shadow direction and it won't be symmetric among flow endpoints. Thank you, Paul. I'd be interested in your feedback on these thoughts of mine below. I do believe it is typical, perhaps with some variance but usually amounting to the same thing, that end-users are billed for the highest of in vs. out traffic, roughly the capacity they are provisioned. Thus if I may, I'll build on this to make a more concrete statement: each party in a peering relationship receives equal value for traffic exchanged. (traffic volume at the SFI translates into revenue from end-users) Things aren't so simple in reality, though: you have to look at the element left out of my statement above, the cost of traffic exchanged. If one peer terminates more traffic than it originates, and the originating peer is performing hot-potato routing, then the terminating peer typically has a higher cost burden as it has to transport the traffic the greater distance. However the opposite holds true if the originating peer is performing cold-potato routing. Thus, such things exist as traffic in/out ratios between peers. But this is a blunt tool which seems to help enforce the exclusivity of the Tier-1 club, and actually acts as a barrier to competition. That is, anybody with a different traffic pattern (i.e., because of a different business model) will be excluded from the club despite the fact that they bring equal value in the form of traffic volume to the relationship. And club-outsiders are subject to increased relative operating costs (cost of revenue) compared to club-insiders. So what is the solution? Warm-potato routing seems possible technically, providing an approximation of cost-burden fairness. Is the benefit worth the complexity to manage in practice? And clearly, I'm not advocating endless open peering--the revenue element of the equation (customers) must exist. So what is the best way to determine the criteria by which a network is determined to be a peer? Cheers, -Benson --- Benson Schliesser (email) mailto:[EMAIL PROTECTED] I barely understand my own thoughts, much worse those of my betters. Thus, the opinions expressed herein are not necessarily those of my employer. Ponder them at your own risk.
Re: Cogent/Level 3 depeering
On Fri, Oct 07, 2005 at 02:53:02AM -0600, Lewis Butler wrote: On 05 Oct 2005, at 13:44 , Charles Gucker wrote: Oh man, I have to jump in here for a moment. Not that I agree with what happened, but to refute your claim that Cogent can get L3 elsewhere, it goes both ways. L3 can also get Cogent connectivity elsewhere. This is a big game of chicken, it will be interesting to see who backs down first. Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? Simply put, yes. Longer answer, Level(3) would have to kiss and make up with Cogent before the sessions would be coordinated to be turned up. There would certainly have to be a renewed level of communication between these two networks to come up with this result. I mean, Cogent can pay someone to route to L3 or L3 can fix what they did on their side, they have no need to go anywhere but their own routers, right? Well, there are three options here. - Both networks kiss and make up, ending up turning up the pre-existing peering session, or possibly additional peering sessions. - Cogent obtains transit from another provider to Level(3). - Level(3) obtains transit from another provider to Cogent. Business decisions do not always make sense, stubbornness can very easily get in the way of a proper decison[1]. charles [1] As outlined in this thread, one person's proper decision may not be another person's.
Re: Cogent/Level 3 Contracts (was: Cogent/Level 3 depeering)
Stephen J. Wilcox wrote: On Fri, 7 Oct 2005, William Allen Simpson wrote: Rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, Mr Wilcox. the contract talks of on-net traffic, off-net traffic and excused outages excused outages includes that of third party network providers off-net traffic has a 99% SLA excluding excused outages. Again, rather than speculation, it would be helpful to refer to the actual contracts. Please post the relevant sections, not your summary of an index of definitions, Mr Wilcox. For instance, I rather doubt that the contract language defines a decision of L(3) to terminate connectivity to a third party as an excused outage. But we won't know without the contract. Enlighten us. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
Re: Cogent/Level 3 depeering
On Oct 7, 2005, at 12:56 PM, Charles Gucker wrote: Simply put, yes. Longer answer, Level(3) would have to kiss and make up with Cogent before the sessions would be coordinated to be turned up. There would certainly have to be a renewed level of communication between these two networks to come up with this result. I seriously doubt L3 would have to do anything but revert to the last known good configuration. If Cogent has done anything to those BGP configurations, they're not only being silly, but they are being disingenuous, since they said they left the configs in place. Of course, that only gets the bits flowing, it doesn't solve the underlying issue. -- TTFN, patrick
Re: Cogent/Level 3 depeering
On Oct 7, 2005, at 11:31 AM, Daniel Golding wrote: Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) Which is fine to say today... but it does mean troubles ahead for IPv6 adoption.
Cogent move without renumbering (was: Cogent/Level 3 depeering)
Patrick W. Gilmore wrote: On Oct 6, 2005, at 8:32 PM, Niels Bakker wrote: * [EMAIL PROTECTED] (Deepak Jain) [Fri 07 Oct 2005, 02:29 CEST]: I think Cogent's offer of providing free transit to all single homed Level3 customers is particularly clever and being underpublicized. Yes, and now that we know enough publically available details, it's time as a community to assist in moving all NANOG ISPs from Level3 to Cogent without renumbering. I guess a significant part of the single-homed networks behind Level (3) would be in PA space owned by them, and thus will find the initial step towards multihoming very hard to take (renumbering into PI or their own PA space). Renumber why? If they have a /24, all they need is an AS a BGP capable router. If they don't have a /24 or larger, then they will either need to renumber, or NAT, or some other fun magic. But the upper bound on the difficult of such exercises is exactly equal to changing providers. It's unlikely that they have an AS when not already multihomed. If everybody with a /24 got an AS, we'd run out quickly. Bad policy. However, we should assist everybody without an AS and at least /24 to move to Cogent without renumbering. That means the blocks should be reassigned. That requires registry assistance. To avoid routing table explosion, we probably need to identify adjacent blocks and encourage them to move to Cogent, too. -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
Re: Cogent/Level 3 depeering
At 01:37 PM 10/7/2005, you wrote: On Fri, 7 Oct 2005, Daniel Golding wrote: Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) Or single-home to a tier-2 -- or tier-1.5, or whatever you want to call it in marketing newspeak -- that provides multihoming of their own networks, and get a netblock from their space. Often, that can be more cost effective (even these depeering situations notwithstanding) than single-homing to a tier-1. Until your local loop to that sole provider dies, or you want to play one off another price-wise and don't want to renumber. Multihoming is backhoe insurance, backbone insurance, and portability insurance.
Re: Cogent/Level 3 depeering
[EMAIL PROTECTED] (Daniel Golding) writes: Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) so, CIDR was a bad idea, and we should push forward with one AS per end-site and a global routing table of 500 million entries? -- Paul Vixie
Re: Cogent/Level 3 depeering
[EMAIL PROTECTED] (Charles Gucker) writes: Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? that's what this press release says: http://www.cogentco.com/htdocs/press.php?func=detailperson_id=62 disclaimer-- my employer has friendly relations with both Level(3) and Cogent. -- Paul Vixie
Re: Peering vs SFI (was Re: Cogent/Level 3 depeering)
[EMAIL PROTECTED] (Schliesser, Benson) writes: If it's still common for one to be billed only for highest of in vs. out then there's no way to compare the benefits since there's always a shadow direction and it won't be symmetric among flow endpoints. Thank you, Paul. I'd be interested in your feedback on these thoughts of mine below. I do believe it is typical, perhaps with some variance but usually amounting to the same thing, that end-users are billed for the highest of in vs. out traffic, roughly the capacity they are provisioned. Thus if I may, I'll build on this to make a more concrete statement: each party in a peering relationship receives equal value for traffic exchanged. (traffic volume at the SFI translates into revenue from end-users) value is subjective. that's what's so funny about depeering announcements and counterannouncements, where somebody always says words to effect of we're as big as they are, so there's no reason we should be paying them. that MAY be true. but it's not debatable. either the former peer will see it that way, or they won't. value is subjective, not democratic. so, i disagree that the parties receive equal value in the general case you cite. maybe they do, maybe they don't. that's for them to decide. each of them, that is, to decide. Things aren't so simple in reality, though: you have to look at the element left out of my statement above, the cost of traffic exchanged. If one peer terminates more traffic than it originates, and the originating peer is performing hot-potato routing, then the terminating peer typically has a higher cost burden as it has to transport the traffic the greater distance. However the opposite holds true if the originating peer is performing cold-potato routing. that summary is probably going to match the facts most of the time. but i know that cold-potato is really cheap for some people and really expensive for others, and that the circumstances that really will govern the costs and benefits of traffic exchange go way beyond how hot the potatoes are. Thus, such things exist as traffic in/out ratios between peers. But this is a blunt tool which seems to help enforce the exclusivity of the Tier-1 club, and actually acts as a barrier to competition. it's only a barrier to competition if it restricts customer choice or increases customer price. in other words, person X doesn't get to accuse a company of restricting competition because they're making it hard for person X to enter the business. person X would have to show that their own choices, as a customer, had been constrained or their costs had been increased in order to claim barrier to competition. otherwise it's just whinage. the game is, you enter the field, you invest in some infrastructure and build some channels, you pay what you have to pay to get access to the network that existed before you, you charge what the market will bear, you make up the difference (if there is one) with cash, and eventually you get big enough to have enough negotiating power that your peering/transit costs go down to where you have some margin. if you run out of cash, you go bankrupt and try again. if you win, then you become part of the backdrop against which new competitors enter the field. if you charge so much for access that new competitors need way more cash to get started, then you also increase what the market will bear to the point where you're very nearly helping new competitors as much as you're hurting them. the only way to win long-term is by staying efficient, never slacking, and being frugal, especially regarding debt. this leads to constant churn, even chaos and carnage, but ultimately it's better for the customers, in my opinion, than regulated monopoly would be. the thing that's irritating me at the moment is that network transit isn't matching other technology curves. as with the workstation on my desk, i do not want to pay less for my new machine than i paid two years ago, in fact i'm willing to pay the same, even adjusting for inflation. what i want is to get a lot more for that price now than i got two years ago. in transit pricing, that would mean we'd all be paying the same total every month that we paid when prices were $1000/Mbit/month, but we'd have 20X the bandwidth available. instead of fighting over a stagnant market we'd be competing to see who could get to the next order of magnitude faster (all without invalidating our physical plant faster than we could depreciate it.) but i digress. That is, anybody with a different traffic pattern (i.e., because of a different business model) will be excluded from the club despite the fact that they bring equal value in the form of traffic volume to the relationship. the value they bring is only equal if the folks they're bringing it to say it's equal. see above. And club-outsiders are subject to increased relative operating costs (cost of revenue) compared to
Re: Cogent/Level 3 depeering
On 07 Oct 2005 19:00:46 +, Paul Vixie [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Charles Gucker) writes: Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? First off, that's not my quote. ;-) Second, it would appear routes are once again beng exchanged between Level(3) and Cogent. BGP routing table entry for 209.244.0.0/14, version 103309841 Paths: (1 available, best #1, table Default-IP-Routing-Table) Not advertised to any peer 174 3356, (aggregated by 3356 4.68.0.12) 66.28.1.1 from 66.28.1.1 (66.28.1.1) Origin IGP, metric 1000, localpref 100, valid, external, atomic-aggregate, best Community: 174:21000 16631:1000 From an outside view, it seems like Level(3) caved in to customer demand, but what the true outcome is, nobody will know [publically]. charles that's what this press release says: http://www.cogentco.com/htdocs/press.php?func=detailperson_id=62 disclaimer-- my employer has friendly relations with both Level(3) and Cogent. -- Paul Vixie
Re: Cogent/Level 3 depeering
Yeah, we just noticed the same.. BGP routing table entry for 38.0.0.0/8, version 23735501 Paths: (3 available, best #1, table Default-IP-Routing-Table) Flag: 0x220 Advertised to peer-groups: core Advertised to non peer-group peers: 64.39.2.107 212.100.225.49 3356 174, (received used) 195.50.112.205 from 195.50.112.205 (4.68.0.240) Origin IGP, metric 0, localpref 100, valid, external, best Community: 3356:3 3356:86 3356:575 3356:666 3356:2010 Charles Gucker wrote: On 07 Oct 2005 19:00:46 +, Paul Vixie [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] (Charles Gucker) writes: Ok, as I understand it, Level3 can get Cogent connectivity back simply be restoring the peering that they suspended, right? First off, that's not my quote. ;-) Second, it would appear routes are once again beng exchanged between Level(3) and Cogent. BGP routing table entry for 209.244.0.0/14, version 103309841 Paths: (1 available, best #1, table Default-IP-Routing-Table) Not advertised to any peer 174 3356, (aggregated by 3356 4.68.0.12) 66.28.1.1 from 66.28.1.1 (66.28.1.1) Origin IGP, metric 1000, localpref 100, valid, external, atomic-aggregate, best Community: 174:21000 16631:1000 From an outside view, it seems like Level(3) caved in to customer demand, but what the true outcome is, nobody will know [publically]. charles that's what this press release says: http://www.cogentco.com/htdocs/press.php?func=detailperson_id=62 disclaimer-- my employer has friendly relations with both Level(3) and Cogent. -- Paul Vixie -- -- Tom Sands Chief Network Engineer Rackspace Managed Hosting (210)447-4065 --
Re: Cogent move without renumbering (was: Cogent/Level 3 depeering)
* [EMAIL PROTECTED] (Deepak Jain) [Fri 07 Oct 2005, 02:29 CEST]: I think Cogent's offer of providing free transit to all single homed Level3 customers is particularly clever and being underpublicized. For educational purposes, could someone elaborate on how this would work? If you're a Level3 customer with Level3 PA space (assumed, since you're already assumed to be single-homed, and therefore very unlikely to need PI or BGP) and move to a Cogent circuit with Cogent PA space, then you'd be able to once again reach Cogent's view of the 'net, but then lose Level3's view of the 'net. If, on the other hand, you move to a Cogent circuit, but keep your Level3 PA space, wouldn't that at least require Cogent to announce all of these recircuited customers' Level3 blocks? This could stop working if Level3 filters those announcements, again resulting in non-reachability for existing Level3 downstreams? Or, on the other hand, is Cogent's offer not exclusive of maintaining the customer's existing Level3 circuit as well, in which case the customer will probably incur more pain with juggling two circuits while not speaking BGP in the first place? Or, is there another hand? Thanks. -- Henry Yen Aegis Information Systems, Inc. Senior Systems Programmer Hicksville, New York
Re: Cogent/Level 3 depeering
On Oct 7, 2005, at 11:54 AM, Paul Vixie wrote: [EMAIL PROTECTED] (Daniel Golding) writes: Take-away: Do not single home. I'm shocked folks aren't figuring this out. If you are a webhoster or enterprise and your business model can not support multiple Internet pipes, than you have a suboptimal business model (to put it lightly) so, CIDR was a bad idea, and we should push forward with one AS per end-site and a global routing table of 500 million entries? Paul, I think that's unnecessarily one dimensional. The needs of business to be connected in a reliable fashion are above and beyond being for or against CIDR. Rather, they are the requirements for the routing architecture that the Internet has yet to fulfill. Single homing is bad simply from a reliability standpoint, and the only true technological impediment to everyone multi-homing is cost and the routing architecture. Consider the ability of the average consumer to make use of WiFi to provide mutual backup connectivity to his neighbors with alternate last mile providers. As the cost goes to zero, everyone will want to multi-home. Regards, Tony
RE: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, Hannigan, Martin wrote: The dialup case results in a very large number of users of a large number of ISPs being single-homed to one or the other of these outfits. Keep that in mind too when you next sign a contract for wholesale dialup service. Dialup costs are $5 a month or less wholesale. What do you expect? That reminds me. If you remember the whole thing started with that L3 complains that Cogent is trying to steal its customers. I kind of checked and it appears Cogent is after dialup/dsl/cable ISPs who as you can guess have absolutely opposite traffic ratio to typical hosting provider that uses cogent. Obviously this extra traffic does not cost Cogent anything (even if its not peering but transit) and allows it to level its in/out ratio. Now going back to it L3 considers that by offering them connectivity at almost no cost Cogent is dumping - but L3 did the same to get those customers under their contracts some years ago (also in order to even its ratio) and besides that I've heard several times from smaller ISPs (see discussion on isp-bandwidth year or two ago) that they are willing to provide transit for dialup dsl ISPs at no charge (and I think I know couple cases where that is true) so they would have better ratio for peering. Now Cogent is also offering free transit for single-homed L3 customers to spite L3 after depeering - majority of such single-homed transit customers are in fact these dsl/dialup ISPs Cogent is after which is why they were willing to make this offer ... Now with 0 transit cost and 0 equipment cost (mostly old dialup equipment loans for which have by now been paid for) its no wonder dialup providers are able to offer it at $5/mo if somebody else takes care of the customer support billing ... -- William Leibzon Elan Networks [EMAIL PROTECTED]
RE: Cogent/Level 3 depeering
Now Cogent is also offering free transit for single-homed L3 customers to spite L3 after depeering - majority of such single-homed transit customers are in fact these dsl/dialup ISPs Cogent is after which is why they were willing to make this offer ... Didn't the free peering offer happen _yesterday_ as a result of the disengagement? It's a tactic. Tommorrow, Level(3) could come out with the same. It's not sustainable by either. Nothing is free. We all know this. Now with 0 transit cost and 0 equipment cost (mostly old dialup equipment loans for which have by now been paid for) You mean amortization? Yes, it's about that. They deployed most of the dial gear in 98, 99. I'm sure augmentations happened after that. Anyhow. What you don't understand is the architecture sans TDM switching, ala SS7 bypass. That's what makes the $5 nut a reality. its no wonder dialup providers are able to offer it at $5/mo if somebody else takes care of the customer support billing ... That's what the other $1 to $10 dollars the retailers are charging is for, William. -M
Re: Cogent/Level 3 depeering
On Wed, 05 Oct 2005 19:27:24 PDT, David Schwartz said: Level 3 cut of Cogent's connectivity. Until and unless they give some reason that makes sense, they are no longer making the effort and are not part of the internet. If I had a garden, things would grow *so* wonderfully next year if I spread this stuff on it. So are you saying that if *your* AS was peered at a dozen places, and you dropped *one* because it wasn't cost-effective, that you wouldn't be part of the Internet, even though you still had 11 peers going full blast? By the same logic, Cogent isn't part of the Internet *EITHER*, because they're not bending over backwards to buy transit to get the L3 routes accessible again. For that matter, AS1312 isn't part of the Internet either, because we're only connected at 2 major points at the moment, and we're not making much of an effort to get connectivity to places that for one reason or another don't see a routing announcement for us, or we don't see their announcement. And I'm sure that with 180K routes, there's gotta be at least a dozen that we can't actually talk to... But oddly enough, I *seem* to be on the Internet. What's wrong with this picture? pgpRNtVzzO75i.pgp Description: PGP signature
Re: Cogent/Level 3 depeering
On Wed, Oct 05, 2005 at 11:15:58PM -0700, william(at)elan.net wrote: That reminds me. If you remember the whole thing started with that L3 complains that Cogent is trying to steal its customers. I kind of checked and it appears Cogent is after dialup/dsl/cable ISPs who as you can guess have absolutely opposite traffic ratio to typical hosting provider that uses cogent. Obviously this extra traffic does not cost Cogent anything (even if its not peering but transit) and allows it to level its in/out ratio. Inbound traffic doesn't cost them anything? That old adage only applies to end user transit purchasers who have doing extra outbound and thus have free inbound under the higher of in or out billing. For folks operating an actual network, the bits use the same resources as traffic in the opposite direction, and thus cost the same. The only reason Cogent gives out free or absurdly underpriced inbound transit is ratios. Now going back to it L3 considers that by offering them connectivity at almost no cost Cogent is dumping - but L3 did the same to get those customers under their contracts some years ago (also in order to even its ratio) and besides that I've heard several times from smaller ISPs (see discussion on isp-bandwidth year or two ago) that they are willing to provide transit for dialup dsl ISPs at no charge (and I think I know couple cases where that is true) so they would have better ratio for peering. Now Cogent is also offering free transit for single-homed L3 customers to spite L3 after depeering - majority of such single-homed transit customers are in fact these dsl/dialup ISPs Cogent is after which is why they were willing to make this offer ... I know folks who are willing to give away all manner of things, inbound and outbound, for free or low cost, because they have excess capacity that they're already paying for and nothing better to do with it. If you're desperate and you're willing to sacrifice long term marketing for short term cash it can be a cute technique, but to quote Vijay, it does not scale. Besides, if anyone is depeering Cogent now because of their disruptive pricing in the market, they're a couple years late. Speculate all you like, but I suspect there is more to it than that. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Cogent/Level 3 depeering
Is being a tier-1 now a good or bad sales argument when selling internet access ? -- Tony Sarendal - [EMAIL PROTECTED] IP/Unix -= The scorpion replied, I couldn't help it, it's my nature =-
Fw: Nuclear survivability (was: Cogent/Level 3 depeering)
While I realize that the nuke survivable thing is probably an old wives tale, it seems ridiculous that the Internet can't adjust by routing any packets that used to go directly from Cogent to Level 3 though some 3rd (and) 4th (and) 5th set of providers that are connected in some fashion to both... It's not a myth. If the Internet were running RIP instead of BGP that is likely what would happen. Of course, there are a few downsides to running RIP on the open Internet as well... Today's Internet is a few generations beyond what Paul Baran originally conceived and the policy and politics of business does tend to gum up the works a bit now that there is no serious threat of global nuclear war. --Michael Dillon
Fw: Peering vs SFI (was Re: Cogent/Level 3 depeering)
Time to quote Geoff Huston one more time. A true peer relationship is based on the supposition that either party can terminate the interconnection relationship and that the other party does not consider such an action a competitively hostile act. If one party has a high reliance on the interconnection arrangement and the other does not, then the most stable business outcome is that this reliance is expressed in terms of a service contract with the other party, and a provider/client relationship is established Those people less versed in the art of peering may not understand why the peer who has been disconnected does not consider the action to be competitively hostile. Simply put, in a true peer relationship, the party who terminates the interconnection is shooting themselves in the foot and inflicting as much commercial pain on themselves as they are inflicting on their peer. The reason that it is not competitively hostile is because it does not increase the ability of either peer to compete. Rather, it damages both of them equally because true peers are equals to begin with. As vijay points out, this whole issue is not really about true peering because such equality between peering partners is rare. It's really about the business case for settlement free interconnect and that is rather more complex than merely the choice between free traffic exchange and paid transit. --Michael Dillon P.S. would the Internet be worse off if all traffic exchange was paid for and there was no settlement free interconnect at all? I.e. paid peering, paid full transit and paid partial transit on the menu?
Re: Who is a Tier 1? (was: Cogent/Level 3 depeering)
Cogent and L3 had _no_ interconnectivity besides the direct peering relationship. L3 knew it, Cogent knew it. L3 made a decision to sever that direct relationship, and bifurcation ensued. This was not only not a surprised, it was expected. Whether Cogent is a tier one or not is irrelevant to the decision, and the resulting effects. I suspect that the consumer definition of Tier 1 is something like provides good connectivity to the entire Internet. By this definition, L3 and Cogent are not currently tier 1 providers. The consumers that I had in mind are the VP finance types who approve the spend on Internet services. A few years ago you could probably bamboozle them about your secret sauce containing transit free, peering, x exchange points and so on. Today I suspect they are less susceptible to that kind of story and more likely to rely on the experience of existing customers. And if the existing customers of L3 and Cogent are experiencing agony, what kind of marketing story does that tell? --Michael Dillon
Re: Cogent/Level 3 depeering
Depeering never makes sense to me. Customers of both companies are expecting their vendor to connect them to the customers of the other company. These customers are each paying their respective vendor for this service. Why should one vendor pay the other for this traffic that is mutually beneficial to them while the other pays nothing? Why does the amount of bandwidth or the direction it travels make any difference? The customers are PAYING for the bandwidth. If each vendor pays their own costs to a peering point then they should be passing that cost onto their respective customers as part of the *customer's* bandwidth bill. Perhaps someone wants to make this argument before a judge in order to set a legal precedent for mandatory peering with settlements? --Michael Dillon
Re: Banks and VCs (was: Cogent/Level 3 depeering)
All the while, Cogent undercuts the market of every other carrier who isn't as efficient as they are, leading to massive losses, bankruptcy filing after bankruptcy filing, out of court reorganizations and purchases for foreign companies, etc. Banks and Venture Capitalists love this. They think in terms of markets, not in terms of companies. Banks and VCs both make more money when a market is not overcrowded with competitors. Obviously, if you can spend a bit of money to fund a company which will kill off all the weaker competitors, and make that money back in a strong company who benefits from larger market share, then it makes sense in the big picture. I don't have any evidence that this is what is happening in this case, however this kind of thing is done to stabilise markets. To the winner go the spoils. All the network assets are not destroyed. For the most part they are aggregated into the networks of companies who buy the bankrupt assets. Sometimes a network operator will buy and integrate a functioning network. Other times, the corporate customer base will buy the boxes and become more likely to buy bandwidth from surviving operators. To force either one to peer by law or regulation would be even more disruptive to the industry as a whole, which may even lead to a complete collapse of the existing peering system as we know it. I think that's too harsh. It would certainly lead to change but that is not the same as collapse. If companies were forced by law to peer, then they would have to do so under a settlement system. Since IP network revenue flow is not based on the end-point usage (per minute call charges) there is no need to engage in massive data collection and analysis as the voice business does. Something based simply on netflow data from peering interconnects combined with the the average bandwidth price on each companies top ten customer contracts would add very little cost to computing the settlement amount. I don't advocate one way or another. But I do expect to see things change when there is instability. --Michael Dillon
RE: VoIP outage (was: Cogent/Level 3 depeering)
1000 users, 15 hours, isn't all that much when you think about it - At some point in the near future, an split such as this is almost assured ofhaving FCC attention due to the consequential consumer business impact. If I understand the way existing VoIP service work, this depeering situation means that VoIP customers, trying to call from one network to another, will not be able to connect because the VoIP provider does not relay voice packets. So the VoIP provider will see that both the calling and called party are on the net, however the flow of voice packets will fail. Has anyone asked Vonage and others what they think of this? It is almost certain to be affecting some of their customers. --Michael Dillon
Re: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, tony sarendal wrote: Is being a tier-1 now a good or bad sales argument when selling internet access ? Personally I think it's good strategy to multihome with one tier-1 and one not so tier-1. The ones further down the foodchain are more likely to be peering whores and therefore provide better connectivty to others like them. It's more likely someone skimps on connections they pay per meg for than peering links, therefore it's in my expereience more likely to be uncongested on peering links than transit links. So my answer to your question is it depends. Using Tier-1:s as your only uplinks means everybody else is paying per meg to send traffic to you, is that what you want? -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Cogent/Level 3 depeering
--- Patrick W. Gilmore [EMAIL PROTECTED] wrote: It is strange that people have to be reminded no network has the right to use any other network's resources without permission. Most people realize this in one direction. For instance, the tier ones love to point out Cogent has no right to peer with Level 3. Absolutely correct. What some people seem to forget is that Level 3 has no right to force Cogent to buy transit to get to Level 3. This is where you lost me: if there is no obligation for an SFI between them, then each player absolutely can force the other to buy transit to reach them. The way it plays out is this: whichever player's customers are more upset about the inability to reach the other will force that player to blink and either buy transit or make some other arrangement. The term peering is useful to describe SFI, because there is an implied equivalence between the players: i.e. it would hurt them both equally to partition. As was said by someone earlier, if it is more valuable to one party than the other, the business relationship is skewed, and ripe for a conversion to a settlement-based interconnection. P.S. Does anyone else get that Baby Bell feeling whenever someone talks about being a Tier One? heh. I'm certain we're about to see the Nth iteration of the who's a Tier One Provider discussion, and I'll repeat: there are two contexts for tier one - marketing and routing. In marketing, everyone with a big, national network is a tier-one. In routing, definitions differ, and whatever definition is used, it's a smaller set than the marketing bunch... David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com __ Yahoo! for Good Donate to the Hurricane Katrina relief effort. http://store.yahoo.com/redcross-donate3/
Re: Cogent/Level 3 depeering
On 10/6/05 1:41 AM, Patrick W. Gilmore [EMAIL PROTECTED] wrote: On Oct 5, 2005, at 4:13 PM, Daniel Golding wrote: They can. Cogent has transit and is preventing traffic from traversing its transit connection to reach Level(3). Level(3) does not have transit - they are in a condition of settlement free interconnection (SFI). The ball is in Cogent's court. This is not the first time or the second that they have chosen to partition. Cogent does purchase transit from Verio to Sprint, AOL, and other locations (but not to Level 3). Perhaps Dan would like to explain why that is relevant to the discussion at hand? Or why that puts the ball in Cogent's court? Since you demanded it - Cogent buys transit. Regardless of what their route filters are currently set to, or what communities or arrangements they have with Verio, its transit. They purchase bandwidth to access other networks. Although I have not seen their transit contract, its not a stretch to say that they can use these connections to reach L3. I realize they may claim otherwise, but I have personal experience with them lying about their transit arrangements. And no, not some call center rep or NOC guy, either. Try a Cogent executive. And no, L3's SFI status does not mean it's Cogent's fault. There is no fault here. This is a business arrangement for all concerned. Cogent can make a configuration change to use their transit to reach Level(3). Level(3) has depeered them. I don't think anyone is right or wrong. Generally, when one plays the peering game and loses, one eats it. Cogent however, is putting up a fight first. I don't blame them - its what I would do. However, they must face the music with their customers. It is strange that people have to be reminded no network has the right to use any other network's resources without permission. Most people realize this in one direction. For instance, the tier ones love to point out Cogent has no right to peer with Level 3. Absolutely correct. What some people seem to forget is that Level 3 has no right to force Cogent to buy transit to get to Level 3. Sure. Cogent is free to offer a partial routing table and take their chances with their customers. If Level 3 doesn't mind not being able to pass packets to Cogent, that's fine. If they do mind, they need to figure out a way to solve the problem - with Cogent. The inverse is true as well. As RAS said, it takes two to tango. This problem will be solved soon (in human time - days, weeks at most). One of the networks may go out of business, but that solves the problem because there would no longer be locations on the Internet someone couldn't reach. I suspect it will be solved by less drastic means. Usually these situations resolve in 2 - 10 days. At least, that's been the pattern. My prediction is that Cogent will fold, because they have in the past. Of course, I can't speak to Level(3)'s intestinal fortitude. - Dan
Re: Cogent/Level 3 depeering
On 10/6/05 6:43 AM, tony sarendal [EMAIL PROTECTED] wrote: Is being a tier-1 now a good or bad sales argument when selling internet access ? Its a great sales argument. That's why everyone claims to be one. It just sounds SO good. And its not like the Peering Police are going to enforce it. What does it mean in real life? Nothing. Nada. An organization's SFI status is a particularly poor criteria for choosing a transit provider. There are so many better factors to use - support, packet loss, price, latency, availability, provisioning speed - you name it, its a better criteria than SFI status. - Dan -- Tony Sarendal - [EMAIL PROTECTED] IP/Unix -= The scorpion replied, I couldn't help it, it's my nature =-
Re: Cogent/Level 3 depeering
On Thu, 6 Oct 2005, tony sarendal wrote: Is being a tier-1 now a good or bad sales argument when selling internet access ? its the same as it always was, its a marketing positive. but thats because the market is dumb. if you wish to make your purchasing decision on 'tier-1' status thats up to you, but i'll be looking at performance, price, strategy, service level and what type of supplier i want for a company like mine. cogent is cheap and you get what you pay for. level3 is mid-price, but they really dont care much about their customers (or thats what i found). perhaps you want better customer service or to deal with a smaller company to gain their attention and respect. choose your supplier based on your own criteria, not someone elses or on who has the most marketing points. Steve
Re: Cogent/Level 3 depeering
On Oct 6, 2005, at 9:11 AM, Daniel Golding wrote: Cogent does purchase transit from Verio to Sprint, AOL, and other locations (but not to Level 3). Perhaps Dan would like to explain why that is relevant to the discussion at hand? Or why that puts the ball in Cogent's court? Since you demanded it - Cogent buys transit. Regardless of what their route filters are currently set to, or what communities or arrangements they have with Verio, its transit. They purchase bandwidth to access other networks. Although I have not seen their transit contract, its not a stretch to say that they can use these connections to reach L3. I realize they may claim otherwise, but I have personal experience with them lying about their transit arrangements. And no, not some call center rep or NOC guy, either. Try a Cogent executive. I think you are confused. If Cogent pays Verio to receive (for instance) only 1239 prefixes, and to propagate 174 prefixes only to 1239, then Cogent cannot make a configuration change to fix things. It would require a contractual change. But even if they could, why does this put the onus only on Cogent? Cogent has just as much right to not spend money to reach L3 as L3 has to not spend money to reach Cogent. Perhaps we are miscommunicating. I am not saying Cogent should not buy transit to reach L3. It is a business decision, not a technical argument. I am saying your idea of Cogent buys transit, therefore the ball is in Cogent's court is Just Plain Wrong. The ball is in _both_ of their courts. It is strange that people have to be reminded no network has the right to use any other network's resources without permission. Most people realize this in one direction. For instance, the tier ones love to point out Cogent has no right to peer with Level 3. Absolutely correct. What some people seem to forget is that Level 3 has no right to force Cogent to buy transit to get to Level 3. Sure. Cogent is free to offer a partial routing table and take their chances with their customers. If you think the inverse of the above is also true, we agree. However, you posts have absolutely at least implied (and I would argue outright claim) that L3 should not be expected to do anything because they are in the SFI club, and Cogent should do something because they buy transit. Perhaps we do agree more than I thought. Did I misunderstand your comments about SFI and balls and courts and stuff? Do you think this situation is bilateral, or does one side have more responsibility to ensure interconnectivity than the other? -- TTFN, patrick
Re: Cogent/Level 3 depeering
On 06/10/05, Stephen J. Wilcox [EMAIL PROTECTED] wrote: On Thu, 6 Oct 2005, tony sarendal wrote: Is being a tier-1 now a good or bad sales argument when selling internet access ? its the same as it always was, its a marketing positive. but thats because the market is dumb. if you wish to make your purchasing decision on 'tier-1' status thats up to you, but i'll be looking at performance, price, strategy, service level and what type of supplier i want for a company like mine. cogent is cheap and you get what you pay for. level3 is mid-price, but they really dont care much about their customers (or thats what i found). perhaps you want better customer service or to deal with a smaller company to gain their attention and respect. I didn't mean for this to sound so much like a question, but I belive I posted before my first cup of coffee. This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. Now, is being a tier-1 now a good or bad sales argument when selling internet access ? /Tony going for more coffee
Re: Cogent/Level 3 depeering
Is being a tier-1 now a good or bad sales argument when selling internet access ? Its a great sales argument. That's why everyone claims to be one. It just sounds SO good. And its not like the Peering Police are going to enforce it. What does it mean in real life? Nothing. Nada. An organization's SFI status is a particularly poor criteria for choosing a transit provider. There are so many better factors to use - support, packet loss, price, latency, availability, provisioning speed - you name it, its a better criteria than SFI status. packet loss and latency to *where*? before replying, consider that most of a leaf's traffic is either to/from another leaf of a tier-1 to which they're (possibly indirectly) downstream, or to/from the tree of a tier-1 which peers with the tier-1 to which they're attached. if tier-n, where n 1, is buying transit from tier-1s, which they have to do, then the price game seems to be pretty determined unless one likes to run at a loss or is cross- subsidizing from some other product line. all your bases are belong to us. :-) randy
Re: Cogent/Level 3 depeering
On Oct 6, 2005, at 10:19 AM, tony sarendal wrote: This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. s/every time it happens/every time it happens to YOUR upstream People on Sprint, ATT, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity. Now, is being a tier-1 now a good or bad sales argument when selling internet access ? It's still a good argument, because Marketing != Reality. :) -- TTFN, patrick
Re: Cogent/Level 3 depeering
On 06/10/05, Patrick W. Gilmore [EMAIL PROTECTED] wrote: On Oct 6, 2005, at 10:19 AM, tony sarendal wrote: This is not the first and certainly not the last time we see this kind of event happen. Purchasing a single-homed service from a Tier-1 provider will guarantee that you are affected by this every time it happens. s/every time it happens/every time it happens to YOUR upstream People on Sprint, ATT, GLBX, MCI, etc. were unaffected. Only people who single-home to L3 or Cogent have disconnectivity. Now, is being a tier-1 now a good or bad sales argument when selling internet access ? It's still a good argument, because Marketing != Reality. :) Patrick, it happens to every PA customer who buys his service from one of the Tier-1 providers active in the de-peering. If a PA customer buys his service from a non-tier1 this will most likely not happen, unless that provider has bought transit in a very unwise way. The entire point is that it's not always good to be too close to tier-1 space. PS. sorry about the double-post Patrick.
Press Re: Cogent/Level 3 depeering
Finally, some press taking notice: http://www.techworld.com/opsys/news/index.cfm?NewsID=4531 -- William Allen Simpson Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32
Re: Press Re: Cogent/Level 3 depeering
William Allen Simpson wrote: Finally, some press taking notice: http://www.techworld.com/opsys/news/index.cfm?NewsID=4531 More at: http://news.com.com/Network+feud+leads+to+Net+blackout/2100-1038_3-5889592.html http://www.broadbandreports.com/shownews/68174 http://www.hostingtech.com/?m=showid=964 http://searchnetworking.techtarget.com/newsItem/0,289139,sid7_gci1132045,00.html and of course the obligatory slashdot thread: http://ask.slashdot.org/askslashdot/05/10/05/2247207.shtml?tid=95tid=187tid=4 jc
Re: Cogent/Level 3 depeering
On Thu, Oct 06, 2005 at 01:33:38PM +0200, Mikael Abrahamsson wrote: It's more likely someone skimps on connections they pay per meg for than peering links, therefore it's in my expereience more likely to be uncongested on peering links than transit links. Sometimes yes, sometimes. no. With a transit provider I can call a sales rep and have a new circuit installed in 30 days or I start getting SLA credits. If said provider is doing something wrong, I can vote with my wallet and take my business to someone else who will do better. With a peer, even a friendly one, you are at the mercy of the cashflow, capacity, goodwill, and traffic engineering clue of another network that is essentially out of your control. Some folks are really good at peering, and some folks are really really bad at peering. Ask anyone who does enough peering and they will have a list of network about whom they will say if we didn't send them X amount of traffic, we would shut their non-responsive prefix-leaking non-upgrading frequent-outage asses off in an instant. Just because a network is big and important doesn't mean that they are taking proper steps to manage the traffic and ensure reliable peering, or even that there is anyone manning the helm at all. And then there is ATT... But that is an issue for another day. :) In my experience, since there is no revenue associated with the peering port on the other side, even very big networks who depend on reliable peering for their business manage to sit on necessary upgrades to peers for months or even years longer than they would if it was a customer port. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Cogent/Level 3 depeering
On Oct 6, 2005, at 2:47 AM, Richard A Steenbergen wrote: Inbound traffic doesn't cost them anything? That old adage only applies to end user transit purchasers who have doing extra outbound and thus have free inbound under the higher of in or out billing. For folks operating an actual network, the bits use the same resources as traffic in the opposite direction, and thus cost the same. The only reason Cogent gives out free or absurdly underpriced inbound transit is ratios. You are mistaken. If I sent 100 Gbps outbound and 20 inbound, I can sell 40-60 Gbps of additional inbound for FAR, FAR less than 40-60 Gbps of additional outbound. Zero cost? Probably not. Trivial cost? Possibly, depends on network. I know folks who are willing to give away all manner of things, inbound and outbound, for free or low cost, because they have excess capacity that they're already paying for and nothing better to do with it. If you're desperate and you're willing to sacrifice long term marketing for short term cash it can be a cute technique, but to quote Vijay, it does not scale. Besides, if anyone is depeering Cogent now because of their disruptive pricing in the market, they're a couple years late. Speculate all you like, but I suspect there is more to it than that. It doesn't have to scale. I'm perfectly willing to sell $100K worth of services for $1K worth of cost, knowing I cannot sell $101K because it does not scale. And anyone who isn't is probably not doing good business. But I do agree with you on the couple years late thing. Putting Cogent out of business will _not_ make prices go up. (And I'm not even sure this will put them out of biz.) In fact, Cogent is not the lowest cost provider any more - at least not for bit pushers. -- TTFN, patrick
Re: Fw: Peering vs SFI (was Re: Cogent/Level 3 depeering)
[EMAIL PROTECTED] wrote: P.S. would the Internet be worse off if all traffic exchange was paid for and there was no settlement free interconnect at all? I.e. paid peering, paid full transit and paid partial transit on the menu? This assumes that one party wants to receive the bits more than the other party wants to send them, or visa versa. When users on network A request data stored on servers on network B, which network should bear the brunt of the cost of this transit? Why? Why shouldn't each network (no matter their size) *equally* bear the costs for transmitting the data between their respective customers that their customers both want transmitted? Settlements are an artifact of long distance phone service where each call was metered and the bill was paid by just one party. In that environment it made sense for the network that was paid for the LD call to pay the other network a part of the fee collected for the call. But the internet traffic doesn't work that way. Users pay a fee for their unlimited connection to their ISP. Content providers pay for the bandwidth their servers require to send the content to users. Why not have each user's network bear their own costs of sending/receiving data between other networks. It's data their own customers are PAYING THEM TO TRANSMIT. The problem is that they each want to push some of their own costs off on the other party whenever possible. Only when they are both roughly the same size can they not get away with bullying the other party into paying more than their fair share and thus we have settlement-free peering between the large Tier 1 networks. In every case I've seen when peering was cut off, it was always a case of a big bully trying to force a smaller party (smaller network) to pay more than their fair share of the costs of the total transmission. It doesn't matter if the smaller network is mostly content or mostly eyeballs or which direction most of the bits flow - all that really matters is that the bigger network is big enough to force the issue and make the smaller network pay for transit (if not paying them, paying *someone*) and ultimately bearing an unfair proportion of the total costs of transmitting the data between their two networks, between their two customers. Note: I'm all for eyeball-providers to require network providers who are mostly content providers to cold-potato route their bandwidth consuming data to the eyeball-provider's connection point closest to the user requesting the data, rather than hot-potato handing it off at the connection point closest to the content provider and leaving the eyeball provider with the burden of backhauling the data to their user requesting the data. However, there's a HUGE difference between requiring cold-potato routing for content, and cutting off peering entirely and forcing the content provider to pay TRANSIT (to someone) to send the data to the eyeball provider as is apparently the current situation with L3 and Cogent. jc
Re: Cogent/Level 3 depeering
On Thu, Oct 06, 2005 at 01:59:01PM -0400, Patrick W. Gilmore wrote: You are mistaken. If I sent 100 Gbps outbound and 20 inbound, I can sell 40-60 Gbps of additional inbound for FAR, FAR less than 40-60 Gbps of additional outbound. Zero cost? Probably not. Trivial cost? Possibly, depends on network. Patrick, I keep telling you, you are not an ISP. :) Yes clearly there is SOME reduction in equipment cost at the edge, you need to buy fewer peering and transit ports if there is available capacity on a full duplex circuit in the opposite direction. You may also see some savings on the customer edge where you are utilizing the extra capacity in the opposite direction on trunk ports out of your aggregation layer. Unfortunately in the core traffic is traffic, and you usually don't see such an obvious but I have this extra capacity in the other direction pattern. The opex cost of hauling the bits that other folks hot potato onto you is going to quickly negate the capex cost of the equipment. I know you don't deal with this, since as we've already established you are not an ISP, but the cost of longhaul circuits (even very large and well negotiated ones between major cities on major routes) is huge. The cost per meg to get a bit from one side of the US to the other is roughly equal to or above what people are selling transit for per meg these days, and in many cases that doesn't take into account non-perfect utilization and the need for backup capacity on diverse paths. There is nothing trivial about this cost for an actual network, and this completely different from using a rule of 95th percentile billing to squeeze some extra service out of someone else's network for free. Of course you could always make the argument that since circuit costs are usually fixed, you could sell at any price and still make more money than nothing as long as you have extra capacity. This may make you very popular in the industry for a short time, but eventually you will hit a brick wall where you can't afford to buy more capacity on the revenues you are generating. A visit to your local bankruptcy court usually follows quickly. It doesn't have to scale. I'm perfectly willing to sell $100K worth of services for $1K worth of cost, knowing I cannot sell $101K because it does not scale. Which is why there are a few small networks who don't have extensive circuits and who happen to have some extra inbound capacity available on their transit pipes are selling it for cheap. The concept of it does not scale explains why networks are still paying for their bandwidth, even their inbound bandwidth. On the original subject of Cogent, the cost of selling inbound bandwidth is not significantly cheaper than the cost of selling outbound, infact it may actually be more expensive depending on how you crunch the numbers for the fiber and DWDM longhaul capacity. But I do agree with you on the couple years late thing. Putting Cogent out of business will _not_ make prices go up. (And I'm not even sure this will put them out of biz.) In fact, Cogent is not the lowest cost provider any more - at least not for bit pushers. Lots of people out there are emulating Cogent's business model but on a smaller scale in order to deliver a low price/meg number. They're often cutting corners that even Cogent doesn't cut though, and their model only works because a) they're dumping traffic onto peers and transits, and b) they have found transit providers who are as desperate for business at any price as they are. -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Cogent/Level 3 depeering
On Oct 6, 2005, at 2:57 PM, Richard A Steenbergen wrote: On Thu, Oct 06, 2005 at 01:59:01PM -0400, Patrick W. Gilmore wrote: You are mistaken. If I sent 100 Gbps outbound and 20 inbound, I can sell 40-60 Gbps of additional inbound for FAR, FAR less than 40-60 Gbps of additional outbound. Zero cost? Probably not. Trivial cost? Possibly, depends on network. Patrick, I keep telling you, you are not an ISP. :) Ha, ha. Yes clearly there is SOME reduction in equipment cost at the edge, you need to buy fewer peering and transit ports if there is available capacity on a full duplex circuit in the opposite direction. You may also see some savings on the customer edge where you are utilizing the extra capacity in the opposite direction on trunk ports out of your aggregation layer. Unfortunately in the core traffic is traffic, and you usually don't see such an obvious but I have this extra capacity in the other direction pattern. The opex cost of hauling the bits that other folks hot potato onto you is going to quickly negate the capex cost of the equipment. I know you don't deal with this, since as we've already established you are not an ISP, but the cost of longhaul circuits (even very large and well negotiated ones between major cities on major routes) is huge. The cost per meg to get a bit from one side of the US to the other is roughly equal to or above what people are selling transit for per meg these days, and in many cases that doesn't take into account non-perfect utilization and the need for backup capacity on diverse paths. There is nothing trivial about this cost for an actual network, and this completely different from using a rule of 95th percentile billing to squeeze some extra service out of someone else's network for free. Please note the Possibly, depends on network comment. There are ABSOLUTELY networks where their backbone circuits are empty but their tail circuits to the peering locations are used in one direction. There are networks which have cities / POPs / regions pushing or pulling more than the opposite. There are lots lots of various configurations where you can plop down a sink or a source and know that they will be utilizing unused resources. Doing so, and selling it at a discount, is simply good business. Sorry if your network isn't like that, but that doesn't make it so for everyone. Oh, and I'd argue you ain't an ISP either. :) -- TTFN, patrick
RE: Cogent/Level 3 depeering
On Wed, 05 Oct 2005 19:27:24 PDT, David Schwartz said: Level 3 cut of Cogent's connectivity. Until and unless they give some reason that makes sense, they are no longer making the effort and are not part of the internet. If I had a garden, things would grow *so* wonderfully next year if I spread this stuff on it. So are you saying that if *your* AS was peered at a dozen places, and you dropped *one* because it wasn't cost-effective, that you wouldn't be part of the Internet, even though you still had 11 peers going full blast? Being part of the Internet is not about communicating with 11 people and not the twelfth. It's about communicating with *anyone* (quite literally) that's willing to make a sufficient effort to communicate with you using the standards and practices that have evolved. By the same logic, Cogent isn't part of the Internet *EITHER*, because they're not bending over backwards to buy transit to get the L3 routes accessible again. Bending over backwards was never required. As I said in the part you cut off when you replied, nobody has to run a line all the way to the server in my basement that isn't connectected to anything at all. What they do have to do is make a reasonable effort to communicate with anyone who is willing to make a similar effort. When you contract for Internet access, you are contracting to reach everyone who wants to reach you. This want is not a mental thing, it's an action of making the effort to connect to people. For that matter, AS1312 isn't part of the Internet either, because we're only connected at 2 major points at the moment, and we're not making much of an effort to get connectivity to places that for one reason or another don't see a routing announcement for us, or we don't see their announcement. And I'm sure that with 180K routes, there's gotta be at least a dozen that we can't actually talk to... If they make an effort to talk to you, and you do not make a similar effort to talk to them, then you're not part of the Internet. The Internet is the network that has resulted from this philosophy. It is this philosophy that makes it the Internet. But oddly enough, I *seem* to be on the Internet. What's wrong with this picture? What's wrong is that you are misrepresenting yourself and your connection philosophy. You are, through your providers and peers making that effort. Buying Internet access from someone who purports to provide it is one way of trying to connect with anyone who tries to connect with you. DS
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
On Thu, 6 Oct 2005 11:54:34 +0100 [EMAIL PROTECTED] wrote: While I realize that the nuke survivable thing is probably an old wives tale, it seems ridiculous that the Internet can't adjust by [...] It's not a myth. If the Internet were running RIP instead of BGP For the Internet, I believe it was indeed a myth. I wasn't there, but according to someone who was: http://www.postel.org/pipermail/end2end-interest/2004-April/003940.html John
Contracts (was: Re: Cogent/Level 3 depeering)
There is another point here. For anyone signing contracts where the buyer has significant bargaining power with the seller, you can specifically stipulate that connectivity to the seller's network is not-good-enough to save them from paying an SLA event or indeed breaching the contract. (What is good enough is left as an exercise to the reader). Sellers may wish to push that risk onto the buyer's, but if history is any judge, buyer's are remiss in accepting that liability and risk without a significant financial incentive. (such as a huge discount over prevailing rates). Deepak
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
on Thu, Oct 06, 2005 at 03:25:54PM -0500, John Kristoff wrote: On Thu, 6 Oct 2005 11:54:34 +0100 [EMAIL PROTECTED] wrote: While I realize that the nuke survivable thing is probably an old wives tale, it seems ridiculous that the Internet can't adjust by [...] It's not a myth. If the Internet were running RIP instead of BGP For the Internet, I believe it was indeed a myth. I wasn't there, but according to someone who was: http://www.postel.org/pipermail/end2end-interest/2004-April/003940.html I believe the mental-mythical sequence went something like: - some people (Paul Baran among them) were interested in ways to build communications networks that could survive lots of damage, and came up with the idea of distributed networks that could route through multiple redundant nodes - the US was in a cold war and nuclear arms race - a nuclear attack could inflict lots of damage to communications networks - the Internet was eventually, to some extent, built as a distributed network with routing through multiple redundant nodes (if nothing else, the protocols that ran it were capable of such) - the Internet was therefore built to survive a nuclear attack QED, HTH, HAND -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com antispam news, solutions for sendmail, exim, postfix: http://enemieslist.com/
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
On Thu, 6 Oct 2005, Steven Champeon wrote: on Thu, Oct 06, 2005 at 03:25:54PM -0500, John Kristoff wrote: On Thu, 6 Oct 2005 11:54:34 +0100 [EMAIL PROTECTED] wrote: While I realize that the nuke survivable thing is probably an old wives tale, it seems ridiculous that the Internet can't adjust by [...] It's not a myth. If the Internet were running RIP instead of BGP For the Internet, I believe it was indeed a myth. I wasn't there, but according to someone who was: http://www.postel.org/pipermail/end2end-interest/2004-April/003940.html I believe the mental-mythical sequence went something like: - some people (Paul Baran among them) were interested in ways to build communications networks that could survive lots of damage, and came up with the idea of distributed networks that could route through multiple redundant nodes Read the paper here: http://www.rand.org/publications/RM/baran.list.html Redundant is probably the wrong word, failure-tolerant is probably more accurate. - the US was in a cold war and nuclear arms race - a nuclear attack could inflict lots of damage to communications networks - the Internet was eventually, to some extent, built as a distributed network with routing through multiple redundant nodes (if nothing else, the protocols that ran it were capable of such) - the Internet was therefore built to survive a nuclear attack Roughly modeled after something designed to continue to route packets following the loss of a few nodes. QED, HTH, HAND -- -- Joel Jaeggli Unix Consulting [EMAIL PROTECTED] GPG Key Fingerprint: 5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2
RE: Peering vs SFI (was Re: Cogent/Level 3 depeering)
Michael Dillon wrote: P.S. would the Internet be worse off if all traffic exchange was paid for and there was no settlement free interconnect at all? I.e. paid peering, paid full transit and paid partial transit on the menu? Would you care to speculate on which party receives the greater benefit: the sender of bytes, or the receiver of bytes? If both the sender and receiver are being billed for the traffic by their respective (different) service providers (all other issues being equal) is one provider in a better position than the other? Cheers, -Benson
Re: Cogent/Level 3 depeering
On Wednesday 05 October 2005 15:52, JC Dill wrote: Matthew Crocker wrote: Ok, I *pay* Cogent for 'Direct Internet Access' which is IP Transit service. I *cannot* get to part of the internet via Cogent right now. [snip] *not* providing complete Internet access, I really don't care who's fault it is. Right now *neither* Cogent nor Level3 are providing complete Direct Internet Access. This is a self-solving problem - why would anyone buy internet access (or renew existing contracts as they expire) from either of these networks when neither of them connect to the complete internet?[1] I would think in NANOG that one would know the simple fact that 'The Complete Internet' is complete and utter fiction, and does not exist. What does exist is a complex, dynamic, even stochastic set of relationships between autonomous networks, who can pick and choose their relationships at whim. We conveniently label this collection of relationships as 'The Internet' and erroneously treat it as an individual, which it is not. Sometimes these individual networks are even antagonistic to each other; boo-hoo. Cry me a river; they've done what they've done and most SLA's for transit don't cover traffic outside the transit network. Take it up with your upstream, who will probably simply say it's not their problem (and it's not, unless your upstream is Cogent or Level3). You cannot reach what doesn't exist, and the Compleat Internet does not exist. All philosophy aside, it does bother me that a simple single depeering can cause such an uproar in a network supposedly immune to nuclear war (even though the Internet was not designed from the start to survive nuclear war; Paul Baran's packet-switching work aside; reference 'Where Wizards Stay Up Late' which quotes Taylor and others on the origins of the ARPAnet portion of the old Internet). I shudder to think of what would happen if there were to be a real problem (I mean, really, one link (out of many thousands) is down and the Sky Is Falling!). What happened to resiliency? 'Hold her steady, steady, Mr. Sulu.' -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu
Re: Peering vs SFI (was Re: Cogent/Level 3 depeering)
[EMAIL PROTECTED] (Schliesser, Benson) writes: Would you care to speculate on which party receives the greater benefit: the sender of bytes, or the receiver of bytes? If both the sender and receiver are being billed for the traffic by their respective (different) service providers (all other issues being equal) is one provider in a better position than the other? If it's still common for one to be billed only for highest of in vs. out then there's no way to compare the benefits since there's always a shadow direction and it won't be symmetric among flow endpoints. -- Paul Vixie
RE: Cogent/Level 3 depeering
I would think in NANOG that one would know the simple fact that 'The Complete Internet' is complete and utter fiction, and does not exist. What does exist is a complex, dynamic, even stochastic set of relationships between autonomous networks, who can pick and choose their relationships at whim. Customers don't want to pay for a stochastic set of relationships, they will pay for the Internet however. It's like paying for a telephone that could only call a subset of the world's telephone users. And the solution (assuming you wanted global reachability) was to buy multiple telephone services from different providers, but even then the reachability that those providers offered would change over time. Would you be happy to rely on telephone for critical business (or other) functions? Call me crazy if you'd like, but I tend to think that peering on the Internet is too important... -Benson --- Benson Schliesser (email) mailto:[EMAIL PROTECTED] I barely understand my own thoughts, much worse those of my betters. Thus, the opinions expressed herein are not necessarily those of my employer. Ponder them at your own risk.
RE: Cogent/Level 3 depeering
Customers don't want to pay for a stochastic set of relationships, they will pay for the Internet however. Perhaps we have lied to the them? The internet has always been a stochastic set of relationships -- some relationships of which are based upon two people getting drunk together at the right place, at the right time. Is anyone going to deny this? Further, the internet has always been a best-effort medium. We, as xSP's, have done our best to make the 'best' in 'best effort' as good as we can, to varying levels of success. The fact that the internet is hugely successful, and mostly reliable, is due to smart people and some level of luck. Not because someone peers with someone else. It wasn't designed this way. It's like paying for a telephone that could only call a subset of the Please, for the love of god, do not make analogies to the phone network. Call me crazy if you'd like, but I tend to think that peering on the Internet is too important... Do you think a thread which has made 100 posts on nanog, with people coming out of the woodwork who I haven't seen in years, is something that anyone things is not important? -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben Net Access Corporation, 800-NET-ME-36, http://www.nac.net
Re: Cogent/Level 3 depeering
In a message written on Thu, Oct 06, 2005 at 06:36:00PM -0400, Lamar Owen wrote: All philosophy aside, it does bother me that a simple single depeering can cause such an uproar in a network supposedly immune to nuclear war (even though the Internet was not designed from the start to survive nuclear war; Paul Baran's packet-switching work aside; reference 'Where Wizards Stay Up Late' which quotes Taylor and others on the origins of the ARPAnet portion of the old Internet). I shudder to think of what would happen if there were to be a real problem (I mean, really, one link (out of many thousands) is down and the Sky Is Falling!). What happened to resiliency? I've seen a lot of comments about the disruption caused by this depeering event, and what would happen if $bad_thing happened. I point you back a few weeks to when the hurricane hit. You need look no further to see people offering up their assistance to those in need. Look back further to 9-11, and people offering networking help to those who's infrastructure was damaged. I have no doubt that if the Level 3 / Cogent issue had been caused by a pre-emptive nuclear strike and the nation was called to arms that virtually every ISP that connects to both would be offering them free transit to get them reconnected. Indeed, I could log into my routers now and fix the Cogent / Level 3 problem with about 3 minutes of typing. It would cost my company thousands of dollars to do so, so I'm not going to do it. As I said before, right now this is a business problem. By the same token, if we were just attacked and Level 3 and Cogent were both, together, asking for help I'd log in and have them working as fast as I could type. I bet others would as well. Level 3 and Cogent are able to fix their own problems in this case, either by making up, or by entering into a business relationship with a third party to fix it. This is also a problem that they, themselves created. That's the difference here. I've got a new set of rules to add to this thread: If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment. You really don't understand what's going on here, and it's not, I repeat, not a technical problem. There is nothing wrong with the technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org pgpYhJeDjKGD9.pgp Description: PGP signature
Re: Cogent/Level 3 depeering
If you don't have enable on a router, and you've never negotiated peering with a transit free ISP then you're not qualified to comment. You really don't understand what's going on here, and it's not, I repeat, not a technical problem. There is nothing wrong with the technology, architecture, or anything else. There is something wrong with the business model of one, or both of these companies. I agree. Though many of the people who meet the second criteria don't even have enable anymore. :) That said, the business relationships that have evolved have certainly overwhelmed -- or rather use a very specific definition of connectivity/reachability, etc. When two transit free networks peer, its often in many locations over many different political regions (time zones, geographies, pick your term). Deliberately and voluntarily taking that down does not change the stability of the underlying architecture. Certainly anyone who runs any network of any size knows very well that the Internet does not survive conscience, deliberate breakage well at all -- nor was it architected to. Put a little knowledge into a border router that is a part of the Internet and watch the chaos you can create. Further, the survivability we talk about also requires that the end nodes, clients, networks, ISPs design for fault-tolerance. This would imply no single-connections. Like all de-peerings, this creates the most hardship for those Enterprise customers (and smaller customers) that either don't have the know how to know they need multiple providers and portable space or the smaller customers that can't afford it [business model or actual finance]. Those of us who are customers of both networks or customers of neither network wouldn't even notice. I think Cogent's offer of providing free transit to all single homed Level3 customers is particularly clever and being underpublicized. I wouldn't be surprised if Cogent is in more buildings than Level3 with a high degree of overlap with the entire Level3 lit network. That could be a very nasty competitor to force into your customers awareness by your own action (or inaction) -- especially if your customer is single homed to you and realizes now that isn't enough of the Internet for them. Deepak
Re: Cogent/Level 3 depeering
* [EMAIL PROTECTED] (Deepak Jain) [Fri 07 Oct 2005, 02:29 CEST]: I think Cogent's offer of providing free transit to all single homed Level3 customers is particularly clever and being underpublicized. I wouldn't be surprised if Cogent is in more buildings than Level3 with a high degree of overlap with the entire Level3 lit network. That could be a very nasty competitor to force into your customers awareness by your own action (or inaction) -- especially if your customer is single homed to you and realizes now that isn't enough of the Internet for them. I guess a significant part of the single-homed networks behind Level(3) would be in PA space owned by them, and thus will find the initial step towards multihoming very hard to take (renumbering into PI or their own PA space). -- Niels. -- Calling religion a drug is an insult to drugs everywhere. Religion is more like the placebo of the masses. -- MeFi user boaz
Re: Cogent/Level 3 depeering
I guess a significant part of the single-homed networks behind Level(3) would be in PA space owned by them, and thus will find the initial step towards multihoming very hard to take (renumbering into PI or their own PA space). Its absolutely a high bar. It is no higher than changing providers which I would probably advocate to anyone who asked my opinion who was single homed. However the to-whom question looms larger and larger. The list of transit-free providers that have not forcibly depeered another network is growing short indeed. If Cogent were looking for an opportunity as a solution provider, they could provide PBR route-maps and the following suggestions: If you are an enterprise customer, many services like your main DNS servers, web server, etc. could gain IPs in both spaces and you could set up a proxy that can tell one space from another and reach both spaces. This would solve many of the access provider and webhosting provider problems out there. The ones that would stay broken are specialized applications and websites that are highly sensitive to proxies, etc. But as as a community, I think NANOGers would agree... something that smells like connectivity is still better than none. Deepak
Re: Nuclear survivability (was: Cogent/Level 3 depeering)
Probably the most authoritative statement out there is at http://www.interesting-people.org/archives/interesting-people/23/msg00081.html I quote: So the motivation for Paul's work was to provide a minimal but highly survivable one-way communications arrangement to get out the go-code; it was NOT motivated by a requirement for a survivable command-control system that could support the forces fully in both peace and in war. That's from Willis Ware, who was in the management structure at RAND at the time. But Baran's own attitude is a bit different. Here's quote from Abbate's Inventing the Internet: on page 1 of the introduction to his 1960 paper describing a survivable communications system Baran explicitly characterized his proposed network as a tool for recovering from?rather than forestalling?a nuclear war: The cloud-of-doom attitude that nuclear war spells the end of the earth is slowly lifting from the minds of the many?. It follows that we should?do all those things necessary to permit the survivors of the holocaust to shuck their ashes and reconstruct the economy swiftly. The cited paper is Reliable Digital Communications Systems Using Unreliable Network Repeater Nodes. Report P-1995, Rand Corporation; I haven't been able to find it online. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb
Re: Cogent/Level 3 depeering
On 6-Oct-2005, at 19:38, Schliesser, Benson wrote: Customers don't want to pay for a stochastic set of relationships, they will pay for the Internet however. What is Internet? Let's channel Seth Breidbart briefly and call it the largest equivalence class in the reflexive transitive symmetric closure of the relationship can be reached by an IP packet from. It should be clear that the nature and extent of this network depends very much on the perspective of the connected device from which is it measured. It's like paying for a telephone that could only call a subset of the world's telephone users. ... which is precisely what every telephone service you can buy in the world gives you, to varying degrees. Do people in Spain complain that they can't call numbers starting with +350, and insist on getting money back from their monthly bill? Or do they accept that their government has an ongoing dispute with the UK over whether Gibraltar is in fact part of Spain? Joe
RE: Cogent/Level 3 depeering
Benson Schliesser wrote:Michael Dillon wrote: P.S. would the Internet be worse off if all traffic . exchange was paid for and there was no settlement free interconnect at all? I.e. paid peering, paid full transit and paid partial transit on the menu?Would you care to speculate on which party receives the greater benefit:the sender of bytes, or the receiver of bytes? If both the sender and receiver are being billed for the traffic bytheir respective (different) service providers (all other issues beingequal) is one provider in a better position than the other? Cheers,-Benson Having enable on a router, yet not having experience with peering in any capacity I was wondering if this analogy holds water. Please excuse the simple model, as I want to understand what other factors may be involved (aside from contractual nuances) Provider A has host/service/user traffic that we will call Blue Bricks that need to be moved outside their network.Provider B has host/service/user traffic that we will call Red Bricks that need to be moved outside their network.. Both providers decide to meet at the corner and exchange 1 brick each on a regular basis let's say for 1000 cycles both providers meet and exchange blocks successfully. for the next 200 cycles Provider A brings his expected blue brick to the corner, yet provider B brings two red bricks.While that was not expected .. it is acceptable in the short term. then as time goes by Provider B begins to bring additional blocks, yet seems not to notice the standard 1 block that provider A is bringing. While fairly simple, this model explains that the disproportion of blocks, or traffic as it were, could be a cause of distress. While I hear talk of Price compression and lining pockets respectively from those who have chosen their position, based on what I've read here and in other places, depeering is a non aggressive yet detrimental way to assert the concerns of a peering provider who feels that the relationship has become inequitable. I can see how the costs of arranging peering and maintaining it can be quite sizable on both sides, but what other factors could cause this type of depeering. Perhaps my view is over simplified, but I don't see this as a black and white bad guy scenario. As previous posts have (whether accurately or not) stated, Cogent was notified in advance of Level 3's intentions and both companies had to know that they were shooting themselves in the foot by playing this ever frustrating game of chicken. I welcome flames/education as this can't be as much of a dichotomy as it seems to be ST
Re: Cogent/Level 3 depeering
On Oct 7, 2005, at 1:17 AM, Silver Tiger wrote: Provider A has host/service/user traffic that we will call Blue Bricks that need to be moved outside their network. Provider B has host/service/user traffic that we will call Red Bricks that need to be moved outside their network.. Both providers decide to meet at the corner and exchange 1 brick each on a regular basis let's say for 1000 cycles both providers meet and exchange blocks successfully. for the next 200 cycles Provider A brings his expected blue brick to the corner, yet provider B brings two red bricks. While that was not expected .. it is acceptable in the short term. then as time goes by Provider B begins to bring additional blocks, yet seems not to notice the standard 1 block that provider A is bringing. While fairly simple, this model explains that the disproportion of blocks, or traffic as it were, could be a cause of distress. It does not. You have forgotten that provider A's customers are asking to get those blue bricks. Is he supposed to stop providing this customers the desired bricks just 'cause he doesn't have enough bricks to get Provider B? You also forgot that Providers A B aren't meeting on a street corner. They're meeting on opposite ends of the continent, or even the planet. Provider A might be carrying those blue bricks a LOT further than Provider B is. Now we have a problem 'cause weight times distance equals backache. You also forgot that Providers A B have to pay cab fare to get to those geographically dispersed corners. One might have to take the cab a lot longer than the other, incurring more time money. You also forgot ... well, about 14 other things. -- TTFN, patrick
Re: Cogent/Level 3 depeering
Alex Rubenstein wrote: Further, the internet has always been a best-effort medium. Can someone please explain how Level 3 is making a best effort to connect their customers to Cogent's customers? Various people have stated that uneven data flows (e.g. from mostly-content networks to mostly-eyeball networks) is a good reason to not peer. I'd love to know how it improves Level 3's network to have data from Cogent arrive over some *other* connection rather than directly from a peering connection. Do they suddenly, magically, no longer have backhaul that mostly-content data across their own backbone to their users who have requested it if it should come in from one of their *other* peers who (in normal peering fashion) hot-potato hands it off to them at the first opportunity, rather than coming in directly from Cogent? I don't think so. So why break off peering??? AFAICT there's only one reason to break off peering, and it's to force Cogent to pay (anyone) to transit the data. Why does L3 care if Cogent sends the data for free via peering, or pays someone ELSE to transit the data? I think this is about a big bully trying to force a smaller player off of the big guys' playing field (tier 1 peering). From where I sit it looks like an anti-competitive move that is not a best effort to serve their customers but a specific effort to put another (smaller) competitor out of business (of being a transit-free or mostly transit-free backbone) by forcing them to pay (someone), forcing their costs up. Level 3 must know they are no longer putting for a best effort for their own customers to connect them to the internet (as their customers see it, the complete internet that their customers have come to expect). I Am Not A Lawyer. (duh?) IMHO all L3 customers have a valid argument that Level 3 is in default of any service contract that calls for best effort or similar on L3's part. I also believe that Cogent has a valid argument that Level 3's behavior is anti-competitive in a market where the tier 1 networks *collectively* have a 100% complete monopoly on the business of offering transit-free backbone internet services. As such, L3's behavior might fall into anti-trust territory - because if Cogent caves in over this and buys transit for the traffic destined for L3 then what's to stop the rest of the tier 1 guys from following suit and forcing Cogent to buy transit to get to *all* tier 1 networks? Then who will they (TINT) force out next? What's to stop a big government (like the US) from stepping in and attempting to regulate peering agreements, using the argument that internet access is too important to allow individual networks to bully other networks out of the market - at the expense of customers - and ultimately resulting in less competition and higher rates? Is this type of regulation good for the internet? OTOH is market consolidation good for the internet? I don't like this slippery slope, I don't like it one little bit. jc
Cogent/Level 3 depeering
A couple weeks later than expected, but as of Oct 5 02:51AM EDT it looks like 3356 and 174 are no longer reachable. lg.level3.net: Show Level 3 (Washington, DC) BGP routes for 38.9.51.20 No matching routes found for 38.9.51.20. www.cogentco.com looking glass: Tracing the route to www.Level3.com (209.245.19.42) 1 f29.ba01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 4 msec 0 msec 2 * * * 3 * * * I guess the earlier reports of (3)'s lack of testicular fortitude may have been exagerated after all. :) -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Cogent/Level 3 depeering
On Wed, 5 Oct 2005, Richard A Steenbergen wrote: A couple weeks later than expected, but as of Oct 5 02:51AM EDT it looks like 3356 and 174 are no longer reachable. lg.level3.net: Show Level 3 (Washington, DC) BGP routes for 38.9.51.20 No matching routes found for 38.9.51.20. www.cogentco.com looking glass: Tracing the route to www.Level3.com (209.245.19.42) 1 f29.ba01.b005944-0.dca01.atlas.cogentco.com (66.250.56.189) 4 msec 4 msec 0 msec 2 * * * 3 * * * I guess the earlier reports of (3)'s lack of testicular fortitude may have been exagerated after all. :) It's sure causing a few headaches here. (from level3 looking glass) Show Level 3 (London, England) BGP routes for 38.9.51.20 No matching routes found for 38.9.51.20 As of 16:22 BST Level3 still seems to have no routes for cogent's space. thats about 5 hours now. Vince -- Richard A Steenbergen [EMAIL PROTECTED] http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
