RE: Outside plant protection, fiber cuts, interwebz down oh noes!
You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I agree 100 percent The clipboard makes it official... --Original Message-- From: Jamie Bowden To: Andy Ringsmuth Cc: nanog@nanog.org Subject: RE: Outside plant protection, fiber cuts, interwebz down oh noes! Sent: Apr 13, 2009 9:07 AM You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy Sent on the Now Network� from my Sprint® BlackBerry
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I know it's fun to have these sort of discussions.. however, here in Toronto anyway all of the splicers, construction people and other contractors all know each other enough to be able to spot somebody thats not auposed to be there. The city inspectors are cruising all day looking for health and safety violations, traffic inspectors are looking for issues, and thecop Maffia is making sure you have a pay duty cop. Unless you were incredibly lucky, a rogue crew at work In a chamber would be caught very quickly. On 13-Apr-09, at 9:07 AM, Jamie Bowden ja...@photon.com wrote: You forgot the clip board. Without the clip board, no one will believe it. J -Original Message- From: Andy Ringsmuth [mailto:andyr...@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
*SNIP* located ten feet down a manhole on Monterey Highway, north of Blossom Hill Road in San Jose. Authorities also found two other locations where fiber optic cables were similarly cut -- near Hayes Avenue and Cottle Road in San Jose *SNIP* Just for clarification, these locations are one in the same. And as an update, the splicing operations are done at this location. (I live 3 blocks away The multitudes of crews that were at the site were gone when I drove by at 3:30PM PST today. The splice trailers were still there around 9:30AM PST working on cable. Bobby Glover Director of Information Services South Valley Internet (AS4307)
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Not to turn this into an ethical typ discussion but this arguement would have to assume you could sue the telco not the 'vandal' due to a loss of life if it occured, and that, that dollar amt would be greater then 'securing' all cables. The cost to fix all pintos' gas tanks was only $11 per car unit and it was gambled, though they lost it was cheeper then the lawsuits, I'm betting the while fewer units, its order of magnatitudes more then 11$ per unit to 'secure' access points with a lot less certain negative lawsuit outcomes. Sent from my BlackBerry device on the Rogers Wireless Network -Original Message- From: Ravi Pina r...@cow.org Date: Fri, 10 Apr 2009 01:51:16 To: JC Dilljcdill.li...@gmail.com Cc: nanog@nanog.orgnanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote: Ravi Pina wrote: That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make vault access non-trivial and yet make the access as easy as it needs to be for routine maintenance and repair? Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing. My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to secure the vaults and deal with the costs and extended outage delays when access it secured and it takes longer to get into a vault to fix things. I wasn't thinking Exodus/CW/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? -r
Fwd: Outside plant protection, fiber cuts, interwebz down oh noes!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I've really got ask if this thread has run it's course. Given the nature of earlier discussions of off-topic issues, I think we've pretty much jumped the shark with people's personal anecdotes of how to disable fiber connectivity. - - ferg - -- Forwarded message -- From: Ravi Pina r...@cow.org Date: Thu, Apr 9, 2009 at 10:51 PM Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! To: JC Dill jcdill.li...@gmail.com Cc: nanog@nanog.org nanog@nanog.org On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote: Ravi Pina wrote: That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make vault access non-trivial and yet make the access as easy as it needs to be for routine maintenance and repair? Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing. My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to secure the vaults and deal with the costs and extended outage delays when access it secured and it takes longer to get into a vault to fix things. I wasn't thinking Exodus/CW/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? - -r -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFJ3uJ/q1pz9mNUZTMRAoRhAJ9m7GTv719RlXUrR6vuGigwpuhJSwCg+sc5 KLrSxYR/cRu1IJOjjM4Go0c= =x059 -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Ravi Pina wrote: Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? Remember the exploding Ford Pinto? http://www.calbaptist.edu/dskubik/pinto.htm
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
deles...@gmail.com wrote: Not to turn this into an ethical typ discussion but this arguement would have to assume you could sue the telco not the 'vandal' due to a loss of life if it occured, and that, that dollar amt would be greater then 'securing' all cables. Internet lawyering is a different mailing list... joel The cost to fix all pintos' gas tanks was only $11 per car unit and it was gambled, though they lost it was cheeper then the lawsuits, I'm betting the while fewer units, its order of magnatitudes more then 11$ per unit to 'secure' access points with a lot less certain negative lawsuit outcomes. Sent from my BlackBerry device on the Rogers Wireless Network -Original Message- From: Ravi Pina r...@cow.org Date: Fri, 10 Apr 2009 01:51:16 To: JC Dilljcdill.li...@gmail.com Cc: nanog@nanog.orgnanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote: Ravi Pina wrote: That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make vault access non-trivial and yet make the access as easy as it needs to be for routine maintenance and repair? Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing. My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to secure the vaults and deal with the costs and extended outage delays when access it secured and it takes longer to get into a vault to fix things. I wasn't thinking Exodus/CW/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? -r
Vandalism Likely In Big South Bay Phone Outage [Was: Re: Outside plant protection, fiber cuts, interwebz down oh noes!]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 More on this: [snip] SAN JOSE (CBS 5 / KCBS / AP / BCN) -- Vandals severed multiple fiber optic cables on Thursday, leaving thousands of people in Santa Clara, Santa Cruz and San Benito counties without cell phone, Internet and landline service, police said. Crews were making repairs, but estimated that it would likely be sometime Thursday night before all service was restored. The outage which began about 2 a.m. affected both landlines and cellular phones for various ATT, Verizon, Sprint and T-Mobile customers. San Jose Police Sgt. Ronnie Lopez, when asked whether the severed fiber optic cables were the work of vandals, said, that's a pretty good premise. No arrests had yet been made, but police joined repair crews at the scene of the first four severed cables -- located ten feet down a manhole on Monterey Highway, north of Blossom Hill Road in San Jose. Authorities also found two other locations where fiber optic cables were similarly cut -- near Hayes Avenue and Cottle Road in San Jose, and in the 1700 block of Old County Road in San Carlos. Officials said the incidents were related and the police indicated all three locations were being treated as crime scenes. The FBI said while the sabotage acts were criminal, there was no evidence of terrorrism. [snip] More: http://cbs5.com/local/phone.internet.outage.2.980578.html - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.5.3 (Build 5003) wj4DBQFJ3qPYq1pz9mNUZTMRAiMlAJ4ztuWURVpURUAbhfzweS/h/nyVCQCVFr+s a+KKWjnze77eqjkrbEB0kg== =+hA8 -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Fri, 10 Apr 2009 01:51:16 EDT, Ravi Pina said: Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? The alarm that goes off saying the lid got opened is only 2 minutes before the big red alarm that says you just lost 5 OC-768s. So the link is *still* going to drop even as you're on the 911 call to try to explain to them where your manhole is, the cops *still* won't catch anybody (the perps may be gone before you hang up on the 911 call), and you're taking 2 minutes off a 10-hour outage. A lot of expense for not a lot of improvement. pgpcngmmZlZAP.pgp Description: PGP signature
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Its all risk and cost. You possibly couldn't have spent enough to stop this event. The outside plant wasn't at fault, highly motivated and informed individuals were. Pretty much a non issue, IMHO. Best, Martin On 4/9/09, Charles Wyble char...@thewybles.com wrote: Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? -- Martin Hannigan mar...@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. And, yes, you can get lockable manhole covers. They aren't cheap. McGuard make a popular one. (Yes, yes...why would I possibly know any of this.I'm a fire marshal in a small town as a part time gig, so I have to deal with this kind of thing on a reasonably regular basis) Daryl
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Charles Wyble wrote: So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? Cost, both in implementing (what are likely to be easily-circumvented) physical protection mechanisms and the cost of dealing with those when on-site doing installation and maintenance. 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? An alarm did go off, the moment the fiber was cut. In the old days, the alarm was gas pressure reduction on the coax followed by loss of signal... now it is loss of the optical carrier. It turns out that the absolute minimum in false alarms is to ignore things bumping into the manhole or falling into the vault and to alarm immediately if the fiber is tampered with, which is exactly what happened. A little semi-automated OTDR and you could tell which manhole it is without driving down to the CO, too. 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? I see individuals raising manhole covers and going down to do maintenance on their own all the time. Glass is cheap enough that the right solution to this problem is route diversity. An alarm goes off when one path is cut, but you have another path that is still running. Now it takes twice as many people to do the cutting. And if you really care, you can back that path up with other technology like microwave radio. But it all comes down to cost. ADSL and POTS subscribers in Santa Cruz County are willing to pay ATT money for service that doesn't have sufficient route diversity along Monterey Highway. As long as it is more profitable to run the network that way, that's how it will be run. And people who care, *can* back this up. My home ADSL was down but my 90 Mbps home microwave link was running fine, and my VoIP was unaffected as well. My bank couldn't process transactions (Frame Relay was down) but the gas station next door could (VSAT was up). A few years ago it was the other way around when Galaxy 4 went belly-up. Either one of those *could* pay a few extra dollars a month and have both... and if that becomes financially worthwhile, maybe they will. But they can't expect their race-to-the-cheapest telco or ISP to do it for them without specific contractual agreements to that effect, and frankly a 14-hour outage just isn't enough lost business to pay for it. (If it was, I'd have a lot easier time signing people up as customers of my south SF bay area/north monterey bay area wireless ISP) Matthew Kaufman
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Fri, Apr 10, 2009 at 2:02 AM, deles...@gmail.com wrote: Not to turn this into an ethical typ discussion but this Maybe it's an ethical issue, with an ethical solution. Random news article from google: Workers are seeking to preserve the health care benefit packages, said Libby Sayre, area director for District 9 of the union, which covers California, Nevada and Hawaii. Union leaders say members' health care costs would more than triple under ATT's current proposal. Sayre said ATT posted a $12.9 billion profit last year and added there are few indications the company will be hit hard by the recession. She added its chief executive officer, Randall Stephenson, earns more than $10 million a year. The article goes further to explain that there are 90,000 workers affected. The arithmetic here is pretty easy. So the operational plan is to try harder not to screw over your employees, could be. -Randy Fischer
RE: Outside plant protection, fiber cuts, interwebz down oh noes!
Your right about having the right tools whats a manhole hook cost $50 -carlos -Original Message- From: Daryl G. Jurbala [mailto:da...@introspect.net] Sent: Friday, April 10, 2009 10:37 AM To: Charles Wyble Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. And, yes, you can get lockable manhole covers. They aren't cheap. McGuard make a popular one. (Yes, yes...why would I possibly know any of this.I'm a fire marshal in a small town as a part time gig, so I have to deal with this kind of thing on a reasonably regular basis) Daryl
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I've had pretty good luck when necessary using a large screwdriver, a claw hammer, or a small crow-bar. I know others who claim it can be done with a spade, pick-axe, etc. although I have not tested those implements. Owen On Apr 10, 2009, at 12:05 PM, Carlos Alcantar wrote: Your right about having the right tools whats a manhole hook cost $50 -carlos -Original Message- From: Daryl G. Jurbala [mailto:da...@introspect.net] Sent: Friday, April 10, 2009 10:37 AM To: Charles Wyble Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote: 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. And, yes, you can get lockable manhole covers. They aren't cheap. McGuard make a popular one. (Yes, yes...why would I possibly know any of this.I'm a fire marshal in a small town as a part time gig, so I have to deal with this kind of thing on a reasonably regular basis) Daryl
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
- Original Message - From: Ravi Pina r...@cow.org To: Charles Wyble char...@thewybles.com Cc: nanog@nanog.org Sent: Thursday, April 09, 2009 5:41 PM Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? I can comment on this, I live three blocks from the scene of the cut. The manholes themselves sit along railroad tracks and an overpass. At 2am, it's a very dark area, and there is very little traffic at that time. It would be an ideal area to perform this type of vandalism. On a side note, when I was passing the area this morning at around 10am PDT, there were two fiber-trailers working in two separate manholes. My company (AS4307) fell off the map from about 2am until roughly 10:42pm when one of our upstreams (AS20115) finally came back. Our primary (AS174) came back about 11:30pm. Of course during the majority of the outage, we also had no land-lines or cell-phones, so were effectively isolated. Bobby Glover Director of Information Services South Valley Internet
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On a side note, when I was passing the area this morning at around 10am PDT, there were two fiber-trailers working in two separate manholes. This is probably the result of having to splice in a new section of fiber, since it would probably have been difficult to splice the ends of the cut section back together.
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 10, 2009, at 12:05 PM, Carlos Alcantar wrote: Your right about having the right tools whats a manhole hook cost $50 Less than half that. http://www.toolup.com/condux/08023000.html -- Jay Hennigan - CCIE #7880 - Network Engineering - j...@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 10, 2009, at 12:05 PM, Carlos Alcantar wrote: Your right about having the right tools whats a manhole hook cost $50 Less than half that. http://www.toolup.com/condux/08023000.html And maybe even less than half *that*. You don't actually need the tool in many cases. A good bit of rebar (trivially found at many construction sites), a prybar, a heavy screwdriver, threaded rod, the trick is just to get the thing out of its rim. Specialized tools are for those who are doing it every day. I suspect that the person responsible did not go out and buy a hook, so this may be pointless anyways. :-) ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Outside plant protection, fiber cuts, interwebz down oh noes!
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Thu, Apr 9, 2009 at 18:04, Charles Wyble char...@thewybles.com wrote: Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. I'll pipe in with this: No amount of money can deter a determined entity. If there is a will, there is a way, etc. Want to protect your outside plant, then make it resilient network-wise. There use to be a time when dual paths was acceptable, I (personally) think that quad paths should be the norm. -Jim P.
RE: Outside plant protection, fiber cuts, interwebz down oh noes!
Hold on. Who says this sabotage? These incidents happen all the time without sabotage being involved. A ship sank off the coast of Pakistan and took out both international cables serving the country ... We had the undersea earthquake that seven seven cables in the Taiwan straits. The truth is that physical diversity is an ideal, not a reality. I have seen lots of accidents that took multiple operators and seriously disrupted in a given locality. The only difference here is that in the Heart of Geek Territory. Hence the Natives are restless ... Roderick S. Beck Director of European Sales Hibernia Atlantic -Original Message- From: Charles Wyble [mailto:char...@thewybles.com] Sent: Thu 4/9/2009 11:04 PM To: nanog@nanog.org Subject: Outside plant protection, fiber cuts, interwebz down oh noes! Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
I didn't say it was sabatoage... It would appear that this was a deliberate act I tried to be very careful to say that it appears to have been sabatoage, but that it's not confirmed. Also this isn't the middle of the ocean, but cable underground. That usually doesn't get cut unless it's by a back hoe. And speaking of unions construction crews charge lots of money to work in the middle of the night, so it's usually avoided. :) Rod Beck wrote: Hold on. Who says this sabotage? These incidents happen all the time without sabotage being involved. A ship sank off the coast of Pakistan and took out both international cables serving the country ... We had the undersea earthquake that seven seven cables in the Taiwan straits. The truth is that physical diversity is an ideal, not a reality. I have seen lots of accidents that took multiple operators and seriously disrupted in a given locality. The only difference here is that in the Heart of Geek Territory. Hence the Natives are restless ... Roderick S. Beck Director of European Sales Hibernia Atlantic -Original Message- From: Charles Wyble [mailto:char...@thewybles.com] Sent: Thu 4/9/2009 11:04 PM To: nanog@nanog.org Subject: Outside plant protection, fiber cuts, interwebz down oh noes! Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Thu, Apr 09, 2009 at 03:04:16PM -0700, Charles Wyble wrote: Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? I think we'd only be speculating with no actual data surrounding the vaults the bundles traversed. That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I would also love it if bacon was healthy for me and didn't make my cholesterol 280. The bay area is also particularly unique in the sense that there aren't many available paths to run fiber. There are mountains on one side and the bay on the other. Your available diverse paths are the left and right side of the tracks, and as a coworker pointed out the left has been full since 1996. -r
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote: Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? This was supposedly an inside job, and I even heard the cabinets were locked. How do you stop an employee with the key from opening a lock? (See #2.) 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? Possibly, and yes. 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Probably, and who knows? How much did this cost the telcos involved? Probably nearly nothing. How much would it cost them to do what you suggest in #2? Probably 1,000,000 times nearly nothing, _at_least_. Guess what the telcos involved will choose? Hell, you would too in their place. -- TTFN, patrick
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Rod Beck wrote: Hold on. Who says this sabotage? By the time the second plane hit WTC, intent was apparent. I think in this case intent is also apparent based on proximity (and the previously mentioned reward ATT has posted for the capture of vandals). Mike
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Thu, Apr 9, 2009 at 4:55 PM, Rod Beck rod.b...@hiberniaatlantic.com wrote: Hold on. Who says this sabotage? the hacksaw that was taken to two manholes within two hours of each other? I'd love to see the RFO explaining an accident like that.
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Charles Wyble wrote: So allow me to think out loud for a minute 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? Some people do lock their vaults/pits/manholes. But, to be honest, I'm not sure it helps a lot. How many passersby would stop someone appearing to be in a phone company/telco high-vis vest using bolt cutters - telling them the lock had seized? (I can also think of quite a few options which don't require opening a lid, but here's not the place to discuss!) 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? Alarms mean power. Adding power to hundreds of km of a route to every pit/manhole would cost a lot - it's underground and often quite wet. Better to provide diverse route protection for the same cost - then you protect against accidental external aggression. Maybe you could do something neat with fibre and some of the active monitoring stuff to detect pit openning passively, but you'd want it to be pretty good and reliable. Lots of false alarms lead to NOCs not caring. 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Obscurity and that most people are blissfully unaware of manholes and other street furniture. Locking is certainly possible but I'm not convinced it adds a LOT (see above). Accidental external aggression is far more likely. Backhoe fade and equipment failure is a bigger problem than vandalism. MMC
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
Ravi Pina wrote: That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make vault access non-trivial and yet make the access as easy as it needs to be for routine maintenance and repair? My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to secure the vaults and deal with the costs and extended outage delays when access it secured and it takes longer to get into a vault to fix things. jc
Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote: Ravi Pina wrote: That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make vault access non-trivial and yet make the access as easy as it needs to be for routine maintenance and repair? Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing. My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to secure the vaults and deal with the costs and extended outage delays when access it secured and it takes longer to get into a vault to fix things. I wasn't thinking Exodus/CW/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? -r
