Re: Outside plant protection, fiber cuts, interwebz down oh noes!
*SNIP* located ten feet down a manhole on Monterey Highway, north of Blossom Hill Road in San Jose. Authorities also found two other locations where fiber optic cables were similarly cut -- near Hayes Avenue and Cottle Road in San Jose *SNIP* Just for clarification, these locations are one in the same. And as an update, the splicing operations are done at this location. (I live 3 blocks away The multitudes of crews that were at the site were gone when I drove by at 3:30PM PST today. The splice trailers were still there around 9:30AM PST working on cable. Bobby Glover Director of Information Services South Valley Internet (AS4307)
Re: Fiber cut in SF area
Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Just my 2¢
Re: Fiber cut in SF area
Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least itâs a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Cute, but a lot of people seem to be wondering this, so a better answer is deserved. The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk There are all sorts of strategies for attacking ATM's, and being susceptible to a sledgehammer, crowbar, or truck smashing into the unit shouldn't be hard to understand. Most data centers have security that is designed to keep honest people out of places that they shouldn't be. Think that security guard at the front will stop someone from running off with something valuable? Maybe. Have you considered following the emergency fire exits instead? Running out the loading dock? Etc? Physical security is extremely difficult, and defending against a determined, knowledgeable, and appropriately resourced attacker out to get *you* is a losing battle, every time. Think about a door. You can close your bathroom door and set the privacy lock, but any adult with a solid shoulder can break that door, or with a pin (or flathead or whatever your particular knob uses) can stick it in and trigger the unlock. Your front door is more solid, but if it's wood, and not reinforced, I'll give my steel-toed boots better than even odds against it. What? You have a commercial hollow steel door? Ok, that beats all of that, let me go get my big crowbar, a little bending will let me win. Something more solid? Ram it with a truck. You got a freakin' bank vault door? Explosives, torches, etc. Fort Knox? Bring a large enough army, you'll still get in. Notice a pattern? For any given level of protection, countermeasures are available. Your house is best secured by making changes that make it appear ordinary and non-attractive. That means that a burglar is going to look at your house, say nah, and move on to your neighbor's house, where your neighbor left the garage open. But if I were a burglar and I really wanted in your house? There's not that much you could really do to stop me. It's just a matter of how well prepared I am, how well I plan. So. Now. Fiber. Here's the thing, now. First off, there usually isn't a financial motivation to attack fiber optic infrastructure. ATM's get some protection because without locks, criminals would just open them and take the cash. Having locks doesn't stop that, it just makes it harder. However, the financial incentive for attacking a fiber line is low. Glass is cheap. We see attacks against copper because copper is valuable, and yet we cannot realistically guard the zillions of miles of copper that is all around. Next. Repair crews need to be able to access the manholes. This is a multifaceted problem. First off, since there are so many manholes to protect, and there are so many crews who might potentially need to access them, you're probably stuck with a standardized key approach if you want to lock them. While this offers some protection against the average person gaining unauthorized access, it does nothing to prevent inside job attacks (and I'll note that this looks suspiciously like an inside job of some sort). Further, any locking mechanism can make it more difficult to gain access when you really need access; some manholes are not opened for years or even decades at a time. What happens when the locks are rusted shut? Is the mechanism weak enough that it can be forced open, or is it tolerable to have to wait extra hours while a crew finds a way to open it? Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. Are you going to protect the concrete too? If not, what prevents me from simply breaking away the concrete around the manhole cover rim (admittedly a lot of work) and just discarding the whole thing? Wait. I just want to *break* the cable? Screw all that. Get me a backhoe. I'll just eyeball the direction I think the cable's going, and start digging until I snag something. Start to see the problems? I'm not saying that security is a bad thing, just a tricky thing. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing
Re: BGP FlowSpec support on provider networks
On Apr 11, 2009, at 12:54 AM, Christopher Morrow wrote: On Fri, Apr 10, 2009 at 6:38 PM, John Payne j...@sackheads.org wrote: On Apr 10, 2009, at 4:27 PM, Fouant, Stefan stefan.fou...@neustar.biz wrote: Hi folks, I am trying to compile data on which providers are currently supporting BGP Flowspec at their edge, if there are any at all. The few providers I've reached out to have indicated they do not support this and have no intention of supporting this any time in the near future. I'm also curious why something so useful as to have the ability to advertise flow specification information in NLRI and distribute filtering information is taking so long to gain a foothold in the industry... Can you name 3 major vendors who support it? I suspect more providers would juniper... and when they dropped the IPR stuff other vendors basically walked away :( Causing consultations with lawyers by others involved with the draft. Life is interesting. IPR, Politics and techie communication skills. The path to failure. - Jared
Re: Fiber cut in SF area
Once upon a time, Jo¢ jbfixu...@gmail.com said: Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno We've had several occasions here where somebody has stolen a backhoe or front-end loader from a construction site, driven to the nearest ATM, and loaded the whole ATM into a (usually stolen) truck. Also, what is the density of outdoor ATMs? I'm in a suburban area, and there may be one every mile or two. How large is the fiber plant? Miles and miles of continuous fiber, every inch of which is equally important. A lot of it here is even on poles, not buried. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: Fiber cut in SF area
* Joe Greco: The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk Heh. Once you install ATMs into solid walls, the attacks get a tad more interesting. In some places of the world, gas detectors are almost mandatory because criminals pump gas into the machine, ignite it, and hope that the explosion blows a hole into the machine without damaging the money (which seems to work fairly well if you use the right gas at the right concentration).
Re: Fiber cut in SF area
On Sat, Apr 11, 2009 at 11:10 AM, Florian Weimer f...@deneb.enyo.de wrote: * Joe Greco: The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk Heh. Once you install ATMs into solid walls, the attacks get a tad more interesting. In some places of the world, gas detectors are almost mandatory because criminals pump gas into the machine, ignite it, and hope that the explosion blows a hole into the machine without damaging the money (which seems to work fairly well if you use the right gas at the right concentration). also, there is the fact that some very large percentage of ATM machines were installed with the same admin passwd setup. I recall ~1.5 yrs ago some news about this, and that essentially banks send out the ATM machines with a stock passwd (sometimes the default which is documented in easily google-able documents) per bank (BoFA uses passwd123, Citi uses passwd456 ) I'm not sure that the manholes == atm discussion is valid, but in the end the same thing is prone to happen to the manholes, there isn't going to be a unique key per manhole, at best it'll be 1/region or 1/manhole-owner. In the end that key is compromised as soon as the decision is made :( Also keep in mind that keyed locks don't really provide much protection, since anyone can order lockpicks over the interwebs these days, even to states where ownership is apparently illegal :( -Chris
Re: Fiber cut in SF area
The best protecion is good engineering taking advantage of technologies and architecures available since long time ago at any of the different network layers. Why network operators/carriers don't do it ?, it's another issue and most of the time is a question of bottom line numbers for which there are no engineering solutions. My .02
Re: SIP - perhaps botnet? anyone else seeing this?
On Fri, 10 Apr 2009 10:20:35 + (GMT) Leland E. Vandervort lel...@taranta.discpro.org wrote: On Fri, 10 Apr 2009, Roland Dobbins wrote: IANAL, but I suggest you check again with your legal department - I doubt this is actually the case (your jurisdiction may vary, but in most Western nations, you can grab packets for diagnostic/ troubleshooting/forensics purposes). Already did check... we can't grab packets except in response to judicial order or specific abuse case with a valid ID of the end-user, or of course for general technical diagnostics -- if for diagnostics, we cannot use such collected data in the context of only a suspicion of abuse at all as it would constitute an infringement on the individual's privacy. So in short, we can do it REACTIVELY in response to a complaint.. but if we do it PROACTIVELY, then it cannot be used and is of educational value only (with caveats surrounding confidentiality, non-disclosure, and destruction,, etc.) You can if it the volume is interfering with your own service, I believe (though IANAL, either) -- see this text from http://www4.law.cornell.edu/uscode/18/2511.html It shall not be unlawful under this chapter for an operator of a switchboard, or an officer, employee, or agent of a provider of wire or electronic communication service, whose facilities are used in the transmission of a wire or electronic communication, to intercept, disclose, or use that communication in the normal course of his employment while engaged in any activity which is a necessary incident to the rendition of his service or to the protection of the rights or property of the provider of that service, except that a provider of wire communication service to the public shall not utilize service observing or random monitoring except for mechanical or service quality control checks. Note carefully that the second part applies to a provider of wire communication service, which is a phone company, not an ISP -- ISPs are providers of electronic communication service. (Just to make life fun -- if you're a VoIP *provider*, you probably fall under both sections, but if you're just carrying VoIP traffic I don't think you are). --Steve Bellovin, http://www.cs.columbia.edu/~smb
[OT] Re: Fiber cut in SF area
On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. You can cut the concrete, too, for that matter, with oxyacetylene, as long as you wear certain protective gear. We have a few vault covers here that are concrete covering the largest vaults we have. You need more than a manhole hook to get one of those covers up. The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. One way of making a site more resistant to 'inside job' issues is with SCIF- like controls (see http://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility ) and using combination locks such as the Sargent and Greenleaf 8077AD for control, and the SG 833 superpadlock for security (see http://www.sargentandgreenleaf.com/PL-833.php ). The tech would have the 833's key, and the area supervisor the combination. The 8077AD's combination is very easily changed in the field, and could be changed frequently. The key to this method's success is that the keyholder to the 833 cannot have the combination, and the holder of the combination cannot have an 833 key. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost
Re: [OT] Re: Fiber cut in SF area
You can cut the concrete, too, for that matter, with oxyacetylene, as long as you wear certain protective gear. We have a few vault covers here that are concrete covering the largest vaults we have. You need more than a manhole hook to get one of those covers up. And when you think you have it safely burried someone drives a tunnel boring machine through it - http://www.flickr.com/photos/23919...@n00/3426407496/ brandon
RE: Fiber cut in SF area
Jo? wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it?s a start. That is an option, but it doesn't address the real problem. The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. No one should be surprised that ATT would cut-corners on critical infrastructure. The good news is that this incident will likely result in increased Federal scrutiny if not regulation. We know how spectacularly energy and banking deregulation failed. Is that mistake being repeated with telecommunications? The bad news is that some of the $16M/yr ATT spends lobbying Congress (for things like fighting number portability and getting a free pass on illegal domestic surveillance) will likely be redirected to ask for money to fix the problem they created. This assumes ATT is as badly managed, and the US FCC and DHS are better managed, than has been the case for the last 8 years. Time will tell. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: http://www.jivamedia.com/askdrdawn/askdrdawn.php http://geekspeak.org/ Roger Marquis
Re: Fiber cut in SF area
The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. BBN won the contract to build the first four IMPs. Theory and research about it is older, look at: http://www.lk.cs.ucla.edu/LK/Bib/REPORT/PhD/proposal-01.html But you are right, redundancy is the issue, cost is the factor. Jorge.
Re: [OT] Re: Fiber cut in SF area
On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. Yes, but we were discussing locked covers, which (given the underlying assumptions of this discussion) might be a bit heavier. Further, it would be vaguely suspicious and more noticeable for a road crew or power company truck to be deploying such gear, might draw more attention. The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. More expense. :-) One way of making a site more resistant to 'inside job' issues is with SCIF- like controls (see http://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility ) and using combination locks such as the Sargent and Greenleaf 8077AD for control, and the SG 833 superpadlock for security (see http://www.sargentandgreenleaf.com/PL-833.php ). The tech would have the 833's key, and the area supervisor the combination. The 8077AD's combination is very easily changed in the field, and could be changed frequently. The key to this method's success is that the keyholder to the 833 cannot have the combination, and the holder of the combination cannot have an 833 key. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost Certainly it is *possible* to do it, but given the other variables, does it make *sense*? Consider what I was saying about just going to town with a backhoe. You have a lot to protect. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
RE: BGP FlowSpec support on provider networks
-Original Message- From: Jared Mauch [mailto:ja...@puck.nether.net] Can you name 3 major vendors who support it? I suspect more providers would juniper... and when they dropped the IPR stuff other vendors basically walked away :( Causing consultations with lawyers by others involved with the draft. Life is interesting. IPR, Politics and techie communication skills. The path to failure. I am familiar with the situation with the IPR and I have to say it's a very disappointing turn of events. I've long held Juniper in high regard as a leader in innovation, but in this instance I feel their actions are doing quite the opposite. That aside, it's 2009 and we're still left with a situation where methodologies which have been used for roughly a decade now (i.e. BGP triggered destination-based filtering) is still considered the norm. Now I realize that FlowSpec isn't a panacea, but it certainly meets some of the requirements that many customers have today, and it gives us a lot more flexibility over simply destination based filtering. Whether it's FlowSpec or something else, what's it going to take to get the vendors and the providers to start moving forward on technologies that are way overdue given the current trend of worms, botnets, and other Internet nastiness? Stefan Fouant: NeuStar, Inc. Principal Network Engineer 46000 Center Oak Plaza Sterling, VA 20166 [ T ] +1 571 434 5656 [ M ] +1 202 210 2075 [ E ] stefan.fou...@neustar.biz [ W ] www.neustar.biz
Re: BGP FlowSpec support on provider networks
Now I realize that FlowSpec isn't a panacea, but it certainly meets some of the requirements that many customers have today, and it gives us a lot more flexibility over simply destination based filtering. Whether it's FlowSpec or something else, what's it going to take to get the vendors and the providers to start moving forward on technologies that are way overdue given the current trend of worms, botnets, and other Internet nastiness? Well, pretty clearly it's going to have to be multivendor, and not IPR encumbered. Aside from that, of course, the usual advice is to talk to your SE and vote with your wallet. From our point of view, BGP triggered destination-based filtering is still one of our most important tools. We have thought about FlowSpec but haven't felt the need sufficiently strongly. Due to MA we are now moving to a mixed Cisco/Juniper network - and FlowSpec is no longer all that interesting since Cisco doesn't implement it. Steinar Haug, Nethelp consulting, sth...@nethelp.no
RE: Fiber cut in SF area
On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: Why didn't the man in the street pharmacy have its own backup plans? Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides ATT or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions? Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy. Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc. Readiness works better if everyone does their part, including grandma. Next time it won't be ATT, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or or or . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, Everything fails sometimes. What's your plan? http://www.ready.gov/ personal opinion only
Re: Fiber cut in SF area
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate. The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer. Sorry for the random thoughts... -Mike On Sat, Apr 11, 2009 at 4:11 PM, Sean Donelan s...@donelan.com wrote: On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. For a good man in the street perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: Why didn't the man in the street pharmacy have its own backup plans? Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides ATT or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions? Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy. Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc. Readiness works better if everyone does their part, including grandma. Next time it won't be ATT, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or or or . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, Everything fails sometimes. What's your plan? http://www.ready.gov/ personal opinion only
Re: Fiber cut in SF area
While OT the news reports indicated ATMs were offline and many credit card processing machines were down. This is no big shock because many ATM networks are on frame relay and POS credit card machines use POTS lines. The outage also impacted mobile service too if it hadn't been said. I hope we can put this thread to rest soon. -r On Sat, Apr 11, 2009 at 04:25:26PM -0700, Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate. The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer. Sorry for the random thoughts... -Mike
Re: Fiber cut in SF area
Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. ... Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Re: Fiber cut in SF area
Don't really care so much about the bank's security, especially if it was one that received some the bailout money :) I was more worried about if people could make withdraws from their bank accounts. Deposits they could do as they could enter them in later but withdraws I think would be different. On Sat, Apr 11, 2009 at 5:19 PM, Roy r.engehau...@gmail.com wrote: Mike Lyon wrote: Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. ... Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Re: Fiber cut in SF area
Sean Donelan wrote: Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.
Re: [OT] Re: Fiber cut in SF area
On Sat, Apr 11, 2009 at 2:43 PM, Joe Greco jgr...@ns.sol.net wrote: On Saturday 11 April 2009 08:31:55 Joe Greco wrote: Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes. Yes, but we were discussing locked covers, which (given the underlying assumptions of this discussion) might be a bit heavier. Further, it would be vaguely suspicious and more noticeable for a road crew or power company truck to be deploying such gear, might draw more attention. Cop: 'What are you fellows doing there with the torch? Me: Us? Oh yea some dipstick plugged up our lock here with epoxy, our quick solution cause of the outage is to cut the lock/blah off with a torch, bummer, eh? I hate dipsticks... Cop: Cool, have a good night! :( The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. More expense. :-) and complexity and parts to lose and people to have away during normal outage repairs and ... :( fail. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost Certainly it is *possible* to do it, but given the other variables, does it make *sense*? Consider what I was saying about just going to town with a backhoe. You have a lot to protect. and I also would ask.. what's the cost/risk here? 'We' lost at best ~1day for some folks in the outage, nothing global and nothing earth-shattering... This has happened (this sort of thing) 1 time in how many years? Expending $$ and time and people to go 'put padlocks on manhole covers' seems like spending in the wrong place... (yes, I agree also that simply dropping into a manhole with an axe/hacksaw is pretty simple to do, it's also just about impossible to realisitcally protect against) -Chris
RE: Fiber cut in SF area
I know as far as att/sbc/pacbell a lot of the time they run the ring within the same conduit to at least have hardware protection on the circuit I'm sure it's the same with other providers. -carlos -Original Message- From: Roy [mailto:r.engehau...@gmail.com] Sent: Saturday, April 11, 2009 6:02 PM To: nanog@nanog.org Subject: Re: Fiber cut in SF area Sean Donelan wrote: Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.
Re: Fiber cut in SF area
Jorge Amodio wrote: s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. /DARPA/ARPA/ may be splitting hairs. According to http://www.livinginternet.com/i/ii_roberts.htm DARPA head Charlie Hertzfeld promised IPTO Director Bob Taylor a million dollars to build a distributed communications network. And apologies WRT /BBM/BBN/. Guess it was really has been a while now (given the 4 and 5 figure checks to BBN I signed back in the day). Sean Donelan wrote: On Sat, 11 Apr 2009, Roger Marquis wrote: The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! The net was created to enable communications bttn point A and point B in this exact scenario. Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call. Apparently even some network operators don't yet grasp the significance of this event. Why didn't the man in the street pharmacy have its own backup plans? I assume they, as most of us, believed the government was taking care of the country's critical infrastructure. Interesting how well this illustrates the growing importance of the Internet vis-a-vis other communications channels. Roger Marquis
Re: [OT] Re: Fiber cut in SF area
Christopher Morrow morrowc.li...@gmail.com writes: and I also would ask.. what's the cost/risk here? 'We' lost at best ~1day for some folks in the outage, nothing global and nothing earth-shattering... This has happened (this sort of thing) 1 time in how many years? Expending $$ and time and people to go 'put padlocks on manhole covers' seems like spending in the wrong place... as long as the west's ideological opponents want terror rather than panic, and also to inflict long term losses rather than short term losses, that's true. in this light you can hopefully understand why bollards to protect internet exchanges against truck bombs are not only penny wise pound foolish (since the manholes a half mile away won't be hardened or monitored or even locked) but also completely wrongheaded (since terrorists need publicity which means they need their victims to be fully able to communicate.) -- Paul Vixie
Re: Fiber cut in SF area
An easy way to describe what your saying is Security by obscurity is not security On Apr 11, 2009, at 8:31 AM, Joe Greco wrote: Jo¢ wrote: I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers can be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno The nice thing about the outdoors is how much of it there is. Cute, but a lot of people seem to be wondering this, so a better answer is deserved. The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious. http://www.youtube.com/watch?v=4P8WM8ZZDHk There are all sorts of strategies for attacking ATM's, and being susceptible to a sledgehammer, crowbar, or truck smashing into the unit shouldn't be hard to understand. Most data centers have security that is designed to keep honest people out of places that they shouldn't be. Think that security guard at the front will stop someone from running off with something valuable? Maybe. Have you considered following the emergency fire exits instead? Running out the loading dock? Etc? Physical security is extremely difficult, and defending against a determined, knowledgeable, and appropriately resourced attacker out to get *you* is a losing battle, every time. Think about a door. You can close your bathroom door and set the privacy lock, but any adult with a solid shoulder can break that door, or with a pin (or flathead or whatever your particular knob uses) can stick it in and trigger the unlock. Your front door is more solid, but if it's wood, and not reinforced, I'll give my steel-toed boots better than even odds against it. What? You have a commercial hollow steel door? Ok, that beats all of that, let me go get my big crowbar, a little bending will let me win. Something more solid? Ram it with a truck. You got a freakin' bank vault door? Explosives, torches, etc. Fort Knox? Bring a large enough army, you'll still get in. Notice a pattern? For any given level of protection, countermeasures are available. Your house is best secured by making changes that make it appear ordinary and non-attractive. That means that a burglar is going to look at your house, say nah, and move on to your neighbor's house, where your neighbor left the garage open. But if I were a burglar and I really wanted in your house? There's not that much you could really do to stop me. It's just a matter of how well prepared I am, how well I plan. So. Now. Fiber. Here's the thing, now. First off, there usually isn't a financial motivation to attack fiber optic infrastructure. ATM's get some protection because without locks, criminals would just open them and take the cash. Having locks doesn't stop that, it just makes it harder. However, the financial incentive for attacking a fiber line is low. Glass is cheap. We see attacks against copper because copper is valuable, and yet we cannot realistically guard the zillions of miles of copper that is all around. Next. Repair crews need to be able to access the manholes. This is a multifaceted problem. First off, since there are so many manholes to protect, and there are so many crews who might potentially need to access them, you're probably stuck with a standardized key approach if you want to lock them. While this offers some protection against the average person gaining unauthorized access, it does nothing to prevent inside job attacks (and I'll note that this looks suspiciously like an inside job of some sort). Further, any locking mechanism can make it more difficult to gain access when you really need access; some manholes are not opened for years or even decades at a time. What happens when the locks are rusted shut? Is the mechanism weak enough that it can be forced open, or is it tolerable to have to wait extra hours while a crew finds a way to open it? Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. Are you going to protect the concrete too? If not, what prevents me from simply breaking away the concrete around the manhole cover rim (admittedly a lot of work) and just discarding the whole thing? Wait. I just want to *break* the cable? Screw all that. Get me a backhoe. I'll just eyeball the direction I think the cable's going, and start digging until I snag something. Start to see the problems? I'm not saying that security is a bad thing, just a tricky thing. ... JG -- Joe Greco - sol.net Network Services
Re: Fiber cut in SF area
Roger Marquis wrote: Why didn't the man in the street pharmacy have its own backup plans? I assume they, as most of us, believed the government was taking care of the country's critical infrastructure. Interesting how well this illustrates the growing importance of the Internet vis-a-vis other communications channels. It's also possible that they just planned on being down in such an event. There's two factors here: Not all low frequency risks are worth mitigating (how many of us have generators at home). Humans are bad at planning around rare events. Econimist Nassim Taleb's book The Black Swan (isbn 978-1400063512) ought to be on everyones list for coverage of the subject matter. Fiber cuts are well outside the realm of experience for your average business manager. The normal remediation strategy (for telecommunications outage) in fact worked just fine, call your provider, and or wait for them to fix it. Roger Marquis
Re: [OT] Re: Fiber cut in SF area
On Sat, 11 Apr 2009, Lamar Owen wrote: The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather. I'm starting to wonder what makes more sense -- locking down thousands of miles of underground tunnel with mil-spec expensive locks that ideally keep unauthorized people out, OR simple motion and or video cameras in the tunnels themselves which relay their access back to a central facility, along with a video feed of sorts, to help identify who is there, whether approved or not. With locks, you know they gained access after the fact and that your locking wasn't sufficient enough. With active monitoring of the area where the cables live, you at least know the moment someone goes in, and have some lead time (and maybe a video) to do something to prevent it, or catch them in the act. Unfortunately, that kind of monitoring is also expensive and complex. I wonder what the cost of the outage was, and how much it might cost to monitor it? Would it be worth $2,000 per site per year? A great webcam, with day/night capability, and a cell phone, in a locked box, with a solar panel, on top of a pole, near the site. Sure, if you know it's there, taking it out is easy, but someone will still know something is wrong when it goes dark or the picture changes significantly. Are there some low-cost, highly-effective ways that the tunnels which carry our precious data and communications can at least be monitored remotely? Waiting for someone to cut a cable and then deploying a crew seems reactive, whereas knowing the moment someone goes INTO the tunnel is proactive, whether the person(s) are there to do some normal maintenance or something malicious. Beckman I suppose rats and other rodents could cause such a system to be too annoying to pay attention to. --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Fiber cut in SF area
An easy way to describe what your saying is Security by obscurity is not security Yes and no. From a certain point of view, security is almost always closely tied to obscurity. A cylinder lock is simply a device that operates through principles that are relatively unknown to the average person: they just know that you stick a key in, turn it, and it opens. The security of such a lock is dependent on an attacker not knowing what a pin and tumbler design is, and not having the tools and (trivial) skills needed to defeat it. That is obscurity of one sort. Public key crypto is, pretty much by definition, reliant on the obscurity of private keys in order to make it work. Ouch, eh. And hard to obtain is essentially a parallel as well. Simply making keyblanks hard to obtain is really a form of obscurity. How much security is dependent on that sort of strategy? It can (and does) work well in many cases, but knowing the risks and limits is important. But that's all assuming that you're trying to secure something against a typical attacker. My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: Fiber cut in SF area
Joe Greco wrote: My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against. The Untold Story of the World's Biggest Diamond Heist published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds Which brings me to a new point: if we accept that security by obscurity is not security, then, what (practical thing) IS security? Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of security by obscurity so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of security by obscurity isn't security was originally meant to convey to software vendors don't rely on closed source to hide your bugs and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. Mike
