actual problems in networks
Hello There is some working service about problems in tier's networks ? Like this: http://www.backbone-news.com/ thanks Piotr
Re: Looking for a Tier 1 ISP Mentor for career advice.
On Sun, 20 Nov 2011 21:40:08 EST, Tyler Haske said: I'm looking for a mentor who can help me focus my career so eventually I wind up working at one of the Tier I ISPs as a senior tech. I want to handle the big pipes that hold everyone's data. OK, so I'm not a mentor from a Tier-1, and I don't directly monkey with routers as part of $DAYJOB. But anyhow... :) With great power comes great responsibility. Be prepared for high stress levels. ;) Also, keep in mind that unless you're insanely brilliant, three things will happen before you get experienced enough to be a senior tech at a Tier 1: 1) You will have grey hair (at least some). 2) The half life of technical know-how in this industry is about 5 years. You'll have been through several half-lifes of what you'll know when you escape from college. Develop the skills needed to learn the next 3 or 4 Next Big Things quickly. 3) You'll have learned that handling a big pipe at a Tier 1 isn't all there is to running a network - and in fact, quite often the Really Cool Toys are elsewhere. Sure, they may have the fastest line cards, but they're going to tend to lag on feature sets just because you *don't* want to deploy cutting-edge code if you're a Tier-1. As an example, AS1312 deployed IPv6 over a decade before some of the Tier 1's could even *spell* it (find out why 6bone existed - it's instructive history). I'm sure that MPLS didn't make its first appearance in TIer-1 core nets either. And the list goes on.. (Hint - where did the Tier 1's get the IPv6/MPLS/whatever experienced engineers to guide their deployment? :) pgprsfp20Z7w7.pgp Description: PGP signature
Re: Looking for a Tier 1 ISP Mentor for career advice.
I really appreciate the specific insights offered by Keegan and Valdis. - Linking me places to apply for jobs doesn't help. I'm aware of who is considered Tier I, and how to find their website. - I'm in Kalamazoo Michigan, and I can commute up to 50 miles. I can't move until I finish my Bachelors in Computer Networking. - The job market here is bad. - I do have a home lab. (Cisco equipment) 2 3350s 2 2950s 4 2611XMs. - This isn't merely a technical request. I'd like support in this endeavor, from someone who's 'been there', to tell me things I CAN'T Google. Tyler.
Re: Looking for a Tier 1 ISP Mentor for career advice.
On Mon, Nov 21, 2011 at 09:09:50AM -0500, Tyler Haske wrote: I really appreciate the specific insights offered by Keegan and Valdis. - Linking me places to apply for jobs doesn't help. I'm aware of who is considered Tier I, and how to find their website. - I'm in Kalamazoo Michigan, and I can commute up to 50 miles. I can't move until I finish my Bachelors in Computer Networking. - The job market here is bad. - I do have a home lab. (Cisco equipment) 2 3350s 2 2950s 4 2611XMs. - This isn't merely a technical request. I'd like support in this endeavor, from someone who's 'been there', to tell me things I CAN'T Google. Tyler. Valdis evolked fond memories... (built the 6bone's first node! and was part of the baseline mesh for over a decade, when it was dismantled) wrt your home lab. you are at a disadvantage (except of course for your certifications) in that the cool toys are not yet in vendor code. consider augmenting your kit w/ OSS versions of routing code (I still like zebra) and dig into fundamentals (ISIS BGP interaction, MPLS, esp with the still unstable OAM code - pick ITU/SG15 or IETF flavors -, consider where the market is headed... look into dynamic discovery in HIP networks, true mobility (not the mobile-IP that is current fashion))... if you are still keen, I can put you in touch w/ some good researchers doing dynamic BGP failover and over the Internet rekeying, if you want to collaberate on things. /bill
Re: Looking for a Tier 1 ISP Mentor for career advice.
On Nov 21, 2011, at 9:09 AM, Tyler Haske wrote: I really appreciate the specific insights offered by Keegan and Valdis. - Linking me places to apply for jobs doesn't help. I'm aware of who is considered Tier I, and how to find their website. - I'm in Kalamazoo Michigan, and I can commute up to 50 miles. I can't move until I finish my Bachelors in Computer Networking. - The job market here is bad. - I do have a home lab. (Cisco equipment) 2 3350s 2 2950s 4 2611XMs. - This isn't merely a technical request. I'd like support in this endeavor, from someone who's 'been there', to tell me things I CAN'T Google. The problem is that even talking about commuting to grand rapids (next biggest city compared to kz, excluding bc) there aren't a lot of local places. There is a nice set of WISPs out there on the west side that may be interesting. There's a few interesting things to think about here: 1) The core space has gotten less interesting in recent years IMHO. While there are still cool things to do, there's more interesting ways to think about problems. 2) A multi-talented person is more useful than someone who thinks only about networking or hosts. This also comes with its own perils as you may not fit well in places that place you inside a box. 3) are you at WMU? Any openings there in the IT/Networking group? What about KVCC, or others? There used to be a more robust local community of ISPs out there (e.g.: net-link/corecomm/voyager). You may want to consider talking to the folks at Climax Telephone as well. They were doing some interesting things last I checked. Learn about the difference between purchasing and leasing. Understand the business side of the equation, not just the technical. These skills will bear fruit when you ask for hardware. Hope this helps some. The market does change quickly (but is becoming a bit slower in some ways) so do be prepared for the business constant of change. If you are unable to adapt to change, you will be left behind. - Jared
Re: Looking for a Tier 1 ISP Mentor for career advice.
Although it is outside of your current commuting distance, if you are looking to stay in Michigan, you might look into Merit in Ann Arbor, or one of the major universities. Merit has been around since the NSFNET/MichNet days. On November 21, 2011 at 9:09 AM Tyler Haske tyler.ha...@gmail.com wrote: I really appreciate the specific insights offered by Keegan and Valdis. - Linking me places to apply for jobs doesn't help. I'm aware of who is considered Tier I, and how to find their website. - I'm in Kalamazoo Michigan, and I can commute up to 50 miles. I can't move until I finish my Bachelors in Computer Networking. - The job market here is bad. - I do have a home lab. (Cisco equipment) 2 3350s 2 2950s 4 2611XMs. - This isn't merely a technical request. I'd like support in this endeavor, from someone who's 'been there', to tell me things I CAN'T Google. Tyler.
Dynamic (changing) IPv6 prefix delegation
Hello List, As a pfSense developer I recently ran into a test system that (actually) gets a IPv6 prefix from it's ISP. (Hurrah). What is bewildering to me is that each time the system establishes a new PPPoE session to the ISP they assign a different IPv6 prefix via delegation together with a differing IPv4 address for the WAN. Is this going to be forward for other consumer ISPs in the world? One of the thoughts that came to mind is T-Online in Germany that still disconnects it's (PPPoE) user base every 24 hours for a new random IP. Short of setting really short timers on the RA messages for the LAN I can see a multitude of complications for consumers in the long run. People that configure their NAS, Media Player and Printer on their own network. And using ULA for either is not workable unless they somehow manage to grow DNS skill on the end user. Their NAS probably wants to download from the 'net and access videos from the NAS. The media player wants to be able to access youtube and the laptop needs to (reliably) find it's printer each time. I really hope that ISPs will commit to assigning the same prefix to the same user on each successive connection. Here is to hoping. Kind regards, Seth
First real-world SCADA attack in US
On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: ASA log viewer
On Sun, Nov 20, 2011 at 17:33, Jimmy Hess mysi...@gmail.com wrote: Yes. logging permit-hostdown However, if you don't need to refuse connections when TCP syslog fails, then you don't need 100% of your syslog messages, you should use UDP syslog for performance. TCP just makes sure you will get all syslog messages between time A and time B or none of them. If there are WAN issues, there are many cases where one would prefer SOME syslog messages, with an understanding that the network bottleneck means messages are being lost, rather than few/no syslog messages to help debug the issue -- -JH Except you can't do syslog via TLS with UDP. :-/ -- Duane Toler deto...@gmail.com
Re: Dynamic (changing) IPv6 prefix delegation
Seth Mos seth@dds.nl writes: Hello List, As a pfSense developer I recently ran into a test system that (actually) gets a IPv6 prefix from it's ISP. (Hurrah). What is bewildering to me is that each time the system establishes a new PPPoE session to the ISP they assign a different IPv6 prefix via delegation together with a differing IPv4 address for the WAN. Is this going to be forward for other consumer ISPs in the world? I certainly hope not. But you should be prepared to handle the situation anyway. Even those ISPs providing a stable prefix may have to change it from time to time. Which means that there is always a risk that the prefix changes with a new PPPoE session, even if that doesn't happen every time. And if the prefix does change, then the old prefix will most likely not be routed out the new PPP interface even if the lease hasn't expired yet. You'll probably want to deprecate the old prefix when this happens, signalling to the hosts that they should prefer the new prefix for new sessions. Bjørn
Re: Dynamic (changing) IPv6 prefix delegation
On 21/11/2011 16:33, Bjørn Mork wrote: But you should be prepared to handle the situation anyway. s/be prepared to handle the situation/plan to handle this as default/ Nick
XO Contact
Hello, Can someone from XO contact me offlist please ? Regards,
Re: First real-world SCADA attack in US
I wonder if they are using private IP addresses. -as On 21 Nov 2011, at 13:32, Jay Ashworth wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: First real-world SCADA attack in US
LOL. I see what you did there. -Hammer- I was a normal American nerd -Jack Herer On 11/21/2011 01:17 PM, Arturo Servin wrote: I wonder if they are using private IP addresses. -as On 21 Nov 2011, at 13:32, Jay Ashworth wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: First real-world SCADA attack in US
I checked the SCADA boxes used in our smart building. They are all using 127.0.0.1 Is that a security risk? -- Leigh Porter On 21 Nov 2011, at 19:20, Arturo Servin arturo.ser...@gmail.com wrote: I wonder if they are using private IP addresses. -as On 21 Nov 2011, at 13:32, Jay Ashworth wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __ __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __
Re: First real-world SCADA attack in US
Might I suggest using 127.0.0.2 if you want less spam :P Pretty scary that folks have 1. Their scada gear on public networks, not behind vpns and firewalls. 2. Allow their hardware vendor to keep a list of usernames / passwords. 2b. Obviously don't change these so often. Whens the last time they really called support and refreshed the password with the hw vendor Probably when they installed the gear... Sheesh.. Perhaps the laws people suggest we need to protect ourselves should be added to. If you are the operator of a network and due to complete insanity leave yourself wide open to attack, you are just as guilty as the bad guys... But then again I don't want to goto jail for leaving my car door open and having someone steal my car, so nix that idea. Ryan Pavely Director Research And Development Net Access Corporation http://www.nac.net/ On 11/21/2011 2:48 PM, Leigh Porter wrote: I checked the SCADA boxes used in our smart building. They are all using 127.0.0.1 Is that a security risk?
Re: Dynamic (changing) IPv6 prefix delegation
On Nov 21, 2011, at 7:21 AM, Seth Mos wrote: Hello List, As a pfSense developer I recently ran into a test system that (actually) gets a IPv6 prefix from it's ISP. (Hurrah). What is bewildering to me is that each time the system establishes a new PPPoE session to the ISP they assign a different IPv6 prefix via delegation together with a differing IPv4 address for the WAN. Is this going to be forward for other consumer ISPs in the world? Unfortunately, there are some ISPs that believe this is the right thing to do. Some go so far as to claim that scrambling customer prefixes is a mechanism to help insure customer privacy. One of the thoughts that came to mind is T-Online in Germany that still disconnects it's (PPPoE) user base every 24 hours for a new random IP. Short of setting really short timers on the RA messages for the LAN I can see a multitude of complications for consumers in the long run. Yep... It remains to be seen whether they will persist in this ill-conceived behavior after the support calls start rolling in. People that configure their NAS, Media Player and Printer on their own network. And using ULA for either is not workable unless they somehow manage to grow DNS skill on the end user. Their NAS probably wants to download from the 'net and access videos from the NAS. The media player wants to be able to access youtube and the laptop needs to (reliably) find it's printer each time. I suspect that mDNS/Rendezvous will become much more widespread in the IPv6 household and will become the primary service discovery mechanism. It actually works quite well and is relatively resilient to either frequent renumbering or the ill-advised use of ULA. I really hope that ISPs will commit to assigning the same prefix to the same user on each successive connection. It would be nice, but, I suspect there will always be some fraction of residential ISPs determined not to do the right thing. Look at the number that are refusing to make generous prefix allocations to residential end users and limiting them to /56, /60, or even worse, /64. Owen
Re: Looking for a Tier 1 ISP Mentor for career advice.
On 22/11/11 03:09, Tyler Haske wrote: I really appreciate the specific insights offered by Keegan and Valdis. - Linking me places to apply for jobs doesn't help. I'm aware of who is considered Tier I, and how to find their website. Don't limit yourself to Tier 1's on the outset. A lot of Network Engineers have worked at least a couple of engineering roles before landing the one that best suits them. Companies usually want to hire experience. That experience coming from as many varied places as possible, actually has some value. In my own case, aside from pure bit-pushing I have had retail sales (electronics sector), technical support, sales, pre-sales and design experience as well as the hands-on engineering of supporting infrastructure (datacentre rack environments, electricity and environmental systems exposure, plus Layer 1-4+...) The disadvantage in angling directly to Tier 1 and working your way up within that organisation will be the potential lack of diversity in your experience. The best thing you can do (IMHO) in lieu of moving to a network-hub city for your hunt, is get your foot in the door with a company that has a significant need for input at the network level, that can help you get your start in terms of hands-on exposure to network operations and management. It'll give you some real-world perspective and it'll provide some of the experience that people will be looking for when reviewing your CV. If you have that, are visibly keen, flexible and continue to (visibly) develop your talents as an engineer, you'll never struggle for work. You can pidgeon-hole yourself pretty quickly if you narrow your skill-focus too far. Mark. PS: Accepted i'm not in the US, so YMMV, but nothing i'm saying strikes me as generically unreasonable.
Re: First real-world SCADA attack in US
- Original Message - From: Ryan Pavely para...@nac.net Perhaps the laws people suggest we need to protect ourselves should be added to. If you are the operator of a network and due to complete insanity leave yourself wide open to attack, you are just as guilty as the bad guys... But then again I don't want to goto jail for leaving my car door open and having someone steal my car, so nix that idea. There is a difference, there, Ryan, both in degree of danger, and in duty of care. If you leave your car open, the odds that someone will steal it *and use it to plow into a crowd of people* are pretty low; the odds that someone breaking into a SCADA network mean to cause harm to the unsuspecting public are probably a bit higher. Also, the people running that SCADA network *get paid* to do so in a fashion which does not cause undue risk to the general public be they customers of the utility or not; this is also not true of your stolen car. So I don't think there's all that much danger of making laws to protect the public from attacked SCADA networks not secured in accordance with generally accepted best practices being generalized into you're going to jail if someone steals your car, even if they *do* use it as a weapon. Even as stupid and grandstander as our Congress is. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: First real-world SCADA attack in US
First https://ciip.wordpress.com/2009/06/21/a-list-of-reported-scada-incidents/ On 22/11/11 04:32, Jay Ashworth wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra
Re: First real-world SCADA attack in US
Am 21.11.2011 um 21:22 schrieb Ryan Pavely: But then again I don't want to goto jail for leaving my car door open and having someone steal my car, so nix that idea. Oh, but you are. (Not sure about criminal liability, but definitely civil.) -- Stefan Bethke s...@lassitu.de Fon +49 151 14070811
Re: First real-world SCADA attack in US
- Original Message - From: Mark Foster blak...@blakjak.net First Hey; I don't write em; I just quote em. :-) https://ciip.wordpress.com/2009/06/21/a-list-of-reported-scada-incidents/ The Willows CA is the only one in the first part of that list that was a) an actual attack, b) that actually had results c) in the US, but yeah; I was unsurprised to find out they were wrong in their characterization. Cheers, - jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: Dynamic (changing) IPv6 prefix delegation
On Mon, Nov 21, 2011 at 12:27:55PM -0800, Owen DeLong wrote: Unfortunately, there are some ISPs that believe this is the right thing to do. Some go so far as to claim that scrambling customer prefixes is a mechanism to help insure customer privacy. s/ISPs/governments, privacy people and influential media outlets/ There is significant political pressure (at least over here) to continue that IPv4 habit for IPv6 as well. Best regards, Daniel -- CLUE-RIPE -- Jabber: d...@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
Re: Looking for a Tier 1 ISP Mentor for career advice.
I appreciate the feedback so far. I'd love to have varied experience with a bunch of different companies, but first I'm trying to guarantee my first network engineering job out of college. Currently I'm studying for the CCNP, exam, with plans to do the CCIP also (its what I have the equipment for). Learning IPv6 is a good idea. With regards to a bigger lab I really wish I had more money to throw at equipment. (I'm aware I can emulate virtualize up to a point) I've looked at the career sites for Western, KVCC, Davenport, CTS Telecommunication, Charter Communication and Stryker today, and nothing is posted. How aggressive should I be at trying to work at one of these places? I really don't have a solid plan for getting a job after graduation. Should I sidetrack and learn Active Directory and Exchange for instance? It would make me more marketable, but distract me from my goals. Tyler
Re: Dynamic (changing) IPv6 prefix delegation
On Nov 21, 2011, at 12:47 PM, Daniel Roesen wrote: On Mon, Nov 21, 2011 at 12:27:55PM -0800, Owen DeLong wrote: Unfortunately, there are some ISPs that believe this is the right thing to do. Some go so far as to claim that scrambling customer prefixes is a mechanism to help insure customer privacy. s/ISPs/governments, privacy people and influential media outlets/ There is significant political pressure (at least over here) to continue that IPv4 habit for IPv6 as well. Yes, IMHO, Germany has some of the most misguided privacy laws and habits in human history. In the rest of the world, it is primarily ISPs that are repeating this mantra, but, hopefully reality will eventually set in and correct the situation even in Germany. Owen
Re: First real-world SCADA attack in US
On 21 Nov 2011, at 20:23, Ryan Pavely para...@nac.net wrote: Might I suggest using 127.0.0.2 if you want less spam :P Pretty scary that folks have 1. Their scada gear on public networks, not behind vpns and firewalls. Do people really do that? Just dump a /24 of routable space on a network and use it? Fifteen years ago perhaps, but now, really? Or are these legacy installations with Cisco routers that don't do 'ip classless' and that everybody has forgotten about? 2. Allow their hardware vendor to keep a list of usernames / passwords. Yeah I can believe this. That's if they bothered changing the passwords at all. 2b. Obviously don't change these so often. Whens the last time they really called support and refreshed the password with the hw vendor Probably when they installed the gear... Sheesh.. I am curious now as to what you would find port scanning for port 23 on some space owned by utility companies. Now, I'm not about to do this, but it would be interesting. Does anybody know what really happened here? We're they just using some ancient VHF radio link to an unmanned pumping station that somebody hacked with an old TCM3105 or AM2911 modem chip and a ham radio? -- Leigh __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __
Re: First real-world SCADA attack in US
On 11/21/11 4:09 PM, Leigh Porter wrote: On 21 Nov 2011, at 20:23, Ryan Pavelypara...@nac.net wrote: Might I suggest using 127.0.0.2 if you want less spam :P Pretty scary that folks have 1. Their scada gear on public networks, not behind vpns and firewalls. Do people really do that? Just dump a /24 of routable space on a network and use it? Fifteen years ago perhaps, but now, really? Or are these legacy installations with Cisco routers that don't do 'ip classless' and that everybody has forgotten about? 2. Allow their hardware vendor to keep a list of usernames / passwords. Yeah I can believe this. That's if they bothered changing the passwords at all. 2b. Obviously don't change these so often. Whens the last time they really called support and refreshed the password with the hw vendor Probably when they installed the gear... Sheesh.. I am curious now as to what you would find port scanning for port 23 on some space owned by utility companies. Now, I'm not about to do this, but it would be interesting. Does anybody know what really happened here? We're they just using some ancient VHF radio link to an unmanned pumping station that somebody hacked with an old TCM3105 or AM2911 modem chip and a ham radio? -- Leigh Probably nowhere near that sophisticated. More like somebody owned the PC running Windows 98 being used as an operator interface to the control system. Then they started poking buttons on the pretty screen. Somewhere there is a terrified 12 year old. Please don't think I am saying infrastructure security should not be improved - it really does need help. But I really doubt this was anything truly interesting. -- Mark Radabaugh Amplex m...@amplex.net 419.837.5015
Re: First real-world SCADA attack in US
On 11/21/11 10:32 AM, Jay Ashworth wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention.This is a problem that doesn't really need a bunch of new legislation. It's an education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. Industrial Controls systems are normally only replaced when they are so old that parts can no longer be obtained. PC's started to be widely used as operator interfaces about the time Windows 95 came out. A lot of those Win95 boxes are still running and have been connected to the network over the years. And... if you can destroy a pump by turning it off and on too often then somebody engineered the control and drive system incorrectly. Operators (and processes) do stupid things all the time. As the control systems engineer your supposed to deal with that so that things don't go boom. -- Mark Radabaugh Amplex m...@amplex.net 419.837.5015
Re: First real-world SCADA attack in US
Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention. This is a problem that doesn't really need a bunch of new legislation. It's an education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. Industrial Controls systems are normally only replaced when they are so old that parts can no longer be obtained. PC's started to be widely used as operator interfaces about the time Windows 95 came out. A lot of those Win95 boxes are still running and have been connected to the network over the years. And... if you can destroy a pump by turning it off and on too often then somebody engineered the control and drive system incorrectly. Operators (and processes) do stupid things all the time. As the control systems engineer your supposed to deal with that so that things don't go boom. -- Mark Radabaugh Amplex m...@amplex.net 419.837.5015 === There are still industrial control machines out there running MS-DOS. As you said not replaced until you can't get parts anymore. Chuck
Re: First real-world SCADA attack in US
On 11/21/11 4:38 PM, Charles Mills wrote: Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention.This is a problem that doesn't really need a bunch of new legislation. It's an education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. Industrial Controls systems are normally only replaced when they are so old that parts can no longer be obtained. PC's started to be widely used as operator interfaces about the time Windows 95 came out. A lot of those Win95 boxes are still running and have been connected to the network over the years. And... if you can destroy a pump by turning it off and on too often then somebody engineered the control and drive system incorrectly. Operators (and processes) do stupid things all the time. As the control systems engineer your supposed to deal with that so that things don't go boom. -- Mark Radabaugh Amplex m...@amplex.net mailto:m...@amplex.net 419.837.5015 tel:419.837.5015 === There are still industrial control machines out there running MS-DOS. As you said not replaced until you can't get parts anymore. Chuck Oh yeah just not too many of those MS-DOS machines have TCP stacks :-) I still get calls to work on machines I designed in 1999. It's a real pain finding a computer that can run the programming software. A lot of the software was written for 386 or slower machines and used timing loops to control the RS-232 ports. Modern processors really screw that software up. -- Mark Radabaugh Amplex m...@amplex.net 419.837.5015
Re: First real-world SCADA attack in US
On Mon, Nov 21, 2011 at 10:32 AM, Jay Ashworth j...@baylink.com wrote: On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra I can say from experience working on one rural sewage treatment plant that IT security is not even in their consciousness. I have also seen major medical software companies that have the same admin password on all install sites and don't see a problem with it. Trying to explain the consequence of this is almost impossible. It's very very scary.
RE: First real-world SCADA attack in US
Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention. +1 Just for context, let me tell everyone about an operational characteristic of one such system (Sold by a Fortune 10 (almost Fortune 5 ;) company for not a small amt. of $) that might be surprising; the hostname of the server system cannot be longer than eight characters. The software gets so many things so very very wrong I wonder how it is there are not more exploits! ~JasonG
Re: First real-world SCADA attack in US
On Mon, Nov 21, 2011 at 4:51 PM, Jason Gurtz jasongu...@npumail.com wrote: Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention. +1 Just for context, let me tell everyone about an operational characteristic of one such system (Sold by a Fortune 10 (almost Fortune 5 ;) company for not a small amt. of $) that might be surprising; the hostname of the server system cannot be longer than eight characters. The software gets so many things so very very wrong I wonder how it is there are not more exploits! siemens, honeywell... essentially all of the large named folks have just horrendous security postures when it comes to any facilities/scada-type systems. they all believe that their systems are deployed on stand-alone networks, and that in the worst case there is a firewall/vpn between their 'management' site and the actually deployed system(s). You think your SCADA network is secure, what about your management company's network? What about actual AAA for any of the changes made? Can you patch the servers/software on-demand? or must you wait for the vendor to supply you with the patch set? folks running scada systems (this includes alarm systems for buildings, or access systems! HVAC in larger complexes, etc) really, really ought to start with RFC requirements that include strong security measures, before outfitting a building you'll be in for 'years'. -chris
Re: Looking for a Tier 1 ISP Mentor for career advice.
2011/11/21 valdis.kletni...@vt.edu On Sun, 20 Nov 2011 21:40:08 EST, Tyler Haske said: I'm looking for a mentor who can help me focus my career so eventually I wind up working at one of the Tier I ISPs as a senior tech. I want to handle the big pipes that hold everyone's data. OK, so I'm not a mentor from a Tier-1, and I don't directly monkey with routers as part of $DAYJOB. But anyhow... :) With great power comes great responsibility. Be prepared for high stress levels. ;) Also, keep in mind that unless you're insanely brilliant, three things will happen before you get experienced enough to be a senior tech at a Tier 1: 1) You will have grey hair (at least some). Not at all required.. Although you may grow a few belt loops and maybe ruin a marriage or two trying to get there early. Also, don't forget to read, cert guides, config guides, websites, RFC's. Grey hair and wisdom aren't mutually inclusive. 3) You'll have learned that handling a big pipe at a Tier 1 isn't all there is to running a network - and in fact, quite often the Really Cool Toys are elsewhere. Sure, they may have the fastest line cards, but they're going to tend to lag on feature sets just because you *don't* want to deploy cutting-edge code if you're a Tier-1. Totally agree. I touch alot of routers some of them close to what Tier-1 would use. I also have a few friends that work in large ISP's. I'd say their ultimate goal is to touch a little as possible which is usually as unglamorous as it sounds. Also, alot of things are scripted so much of what you touch may not be as fun. As an example, AS1312 deployed IPv6 over a decade before some of the Tier 1's could even *spell* it (find out why 6bone existed - it's instructive history). I'm sure that MPLS didn't make its first appearance in TIer-1 core nets either. And the list goes on.. (Hint - where did the Tier 1's get the IPv6/MPLS/whatever experienced engineers to guide their deployment? :) Also, how many junior and mid-level guys leave a Tier I for a network where they can touch things and then come back as experts. Also, the intermediate job tends to pay for certs and training which is a plus.
RE: Dynamic (changing) IPv6 prefix delegation
Look at the number that are refusing to make generous prefix allocations to residential end users and limiting them to /56, /60, or even worse, /64. Owen, What does Joe Sixpack do at home with a /48 that he cannot do with a /56 or a /60? Nathan
Re: First real-world SCADA attack in US
On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security That URL says: The Nov. 8 incident was described in a one-page report from the Illinois Statewide Terrorism and Intelligence Center, according to Joe Weiss, a prominent expert on protecting infrastructure from cyber attacks. Joe Weiss gave a good talk at Stanford last Oct 12. http://www.stanford.edu/class/ee380/ My quick summary: The whole SCADA industry isn't tuned into network security issues. It's not part of their culture. -- Several years ago, Idaho National Labs ran an experiment. They blew up a diesel generator by remote control. Aurora is the buzzword. The abstract page for his talk has a link to a CNN video. It only has a few seconds of the generator. Here is a longer version on YouTube: http://www.youtube.com/watch?v=fJyWngDco3g -- These are my opinions, not necessarily my employer's. I hate spam.
Re: First real-world SCADA attack in US
If NSA had no signals information prior to the attack, this should be a wake up call for the industry. Andrew From: Jay Ashworth j...@baylink.com To: NANOG nanog@nanog.org Sent: Monday, November 21, 2011 3:32 PM Subject: First real-world SCADA attack in US On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: Looking for a Tier 1 ISP Mentor for career advice.
--- tyler.ha...@gmail.com wrote: From: Tyler Haske tyler.ha...@gmail.com I'd love to have varied experience with a bunch of different companies, but first I'm trying to guarantee my first network engineering job out of college. --- You've already taken the first step. That step being you becoming more motivated than many of the other soon-to-be-graduates around you. This motivation will carry you a long way in your career. Who knows, you may be applying to someone here on this list one day... scott
Re: Dynamic (changing) IPv6 prefix delegation
On Mon, Nov 21, 2011 at 22:18, Nathan Eisenberg nat...@atlasnetworks.us wrote: Look at the number that are refusing to make generous prefix allocations to residential end users and limiting them to /56, /60, or even worse, /64. Owen, What does Joe Sixpack do at home with a /48 that he cannot do with a /56 or a /60? Flexibility. With dhcpv6 prefix delegation, you are going to want devices to be able to request (at least) /60s for further delegation (and better yet /56s to allow them to delegate /60s with further delegation when needed). While Joe may not have as complex of an environment as his neighbor Sue, should we target the common Joe, or the advanced Sue? As I suspect Owen will say, there is no reason *not* to give out /48s (ipv6 space is huge), and this is good opportunity to enable the residential user to not have to work around artificial limits in the future. Gary
Re: Looking for a Tier 1 ISP Mentor for career advice.
On Mon, Nov 21, 2011 at 02:32:53PM -0800, Scott Weeks wrote: --- tyler.ha...@gmail.com wrote: From: Tyler Haske tyler.ha...@gmail.com I'd love to have varied experience with a bunch of different companies, but first I'm trying to guarantee my first network engineering job out of college. --- You've already taken the first step. That step being you becoming more motivated than many of the other soon-to-be-graduates around you. This motivation will carry you a long way in your career. Who knows, you may be applying to someone here on ===-- replying this list one day... line-wrapped that for you scott... gift bows are USD2.00 extra. scott /bill
RE: Dynamic (changing) IPv6 prefix delegation
What does Joe Sixpack do at home with a /48 that he cannot do with a /56 or a /60? Flexibility. With dhcpv6 prefix delegation, you are going to want devices to be able to request (at least) /60s for further delegation (and better yet /56s to allow them to delegate /60s with further delegation when needed). While Joe may not have as complex of an environment as his neighbor Sue, should we target the common Joe, or the advanced Sue? As I suspect Owen will say, there is no reason *not* to give out /48s (ipv6 space is huge), and this is good opportunity to enable the residential user to not have to work around artificial limits in the future. Gary Prefix delegation for what? What does Sue do at home that requires 2 levels of prefix delegation inside the house? Does Sue really need to be able to have 65536 subnets instead of 256 in her home? Nathan
Re: Dynamic (changing) IPv6 prefix delegation
Sent from my iPhone On Nov 21, 2011, at 14:18, Nathan Eisenberg nat...@atlasnetworks.us wrote: Look at the number that are refusing to make generous prefix allocations to residential end users and limiting them to /56, /60, or even worse, /64. Owen, What does Joe Sixpack do at home with a /48 that he cannot do with a /56 or a /60? Nathan First, the better question is what advantage is there in building such limiting present day limitations into the future? Second, the answer is facilitate a broad range of automated hierarchical topologies allowing for both breadth and depth of prefix distribution among partitions within the home environment. I admit we have not even begun to scratch the surface of how, where, or why these topologies may evolve, but I can see that due to the tendency for software to be developed to the lowest common denominator, if we make said denominator too low, we will forever blockade the opportunities for such innovations to see the light of day. Owen
RE: actual problems in networks
Yes, the outages listserv, on a good day: http://www.outages.org/index.php/Main_Page#Outages_Mailing_Lists -Original Message- From: Piotr [mailto:piotr.1...@interia.pl] Sent: Monday, November 21, 2011 7:25 AM To: nanog@nanog.org Subject: actual problems in networks Hello There is some working service about problems in tier's networks ? Like this: http://www.backbone-news.com/ thanks Piotr
Re: First real-world SCADA attack in US
On Nov 21, 2011, at 4:30 PM, Mark Radabaugh wrote: Probably nowhere near that sophisticated. More like somebody owned the PC running Windows 98 being used as an operator interface to the control system. Then they started poking buttons on the pretty screen. Somewhere there is a terrified 12 year old. Please don't think I am saying infrastructure security should not be improved - it really does need help. But I really doubt this was anything truly interesting. That's precisely the problem: it does appear to have been an easy attack. (My thoughts are at https://www.cs.columbia.edu/~smb/blog/2011-11/2011-11-18.html) --Steve Bellovin, https://www.cs.columbia.edu/~smb
RE: First real-world SCADA attack in US
Subject: First real-world SCADA attack in US On an Illinois water utility: http://www.msnbc.msn.com/id/45359594/ns/technology_and_science-security that which does not kill us makes us stronger --Friedrich Nietzsche
Re: First real-world SCADA attack in US
On Mon, Nov 21, 2011 at 3:35 PM, Mark Radabaugh m...@amplex.net wrote: On 11/21/11 10:32 AM, Jay Ashworth wrote: education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. The existing normal methods used by much of the IT industry fail way too often, and therefore, some measure of regulation is in order, when the matter is about critical public infrastructure -- it's simply not in the public interest to let agencies fail or use slipshod/ half measure techniques that are commonly practiced by some of the IT industry. They should be required to engage in practices that can be proven to mitigate risks to a know controllable quantity. The weakness of typical IT security is probably OK, when the only danger of compromise is that an intruder might get some sensitive information, or IT might need to go to the tapes. That just won't do, when the result of compromise is, industrial equipment is forced outside of safe parameters, resulting in deaths, or a city's water supply is shut down, resulting in deaths. Hard perimeter and mushy interior with OS updates just to address known issues, and malware scanners to try and catch things just won't do. ...an OS patch introduces a serious crash bug is also a type of security issue. Patching doesn't necessarily improve security; it only helps with issues you know about, and might introduce issues you don't know about. Enumerating badness is simply not reliable, and patch patch patch is simply an example of that -- when security really matters, don't attach it to a network, especially not one that might eventually be internet connected -- indirect or not. Connection to a management LAN that has any PC on it that is or was ever internet connected counts as an internet connection. Industrial Controls systems are normally only replaced when they are so old that parts can no longer be obtained. PC's started to be widely used as operator interfaces about the time Windows 95 came out. A lot of those Win95 boxes are still running and have been connected to the network over the years. The Windows 95 part is fine. The connected to the network part is not fine. -- -JH
Re: First real-world SCADA attack in US
- Original Message - From: Jimmy Hess mysi...@gmail.com On Mon, Nov 21, 2011 at 3:35 PM, Mark Radabaugh m...@amplex.net wrote: On 11/21/11 10:32 AM, Jay Ashworth wrote: education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. Careful with the attribution; you're quoting Mark, not me. The weakness of typical IT security is probably OK, when the only danger of compromise is that an intruder might get some sensitive information, or IT might need to go to the tapes. That just won't do, when the result of compromise is, industrial equipment is forced outside of safe parameters, resulting in deaths, or a city's water supply is shut down, resulting in deaths. (72 character hard wrap... please.) Hard perimeter and mushy interior with OS updates just to address known issues, and malware scanners to try and catch things just won't do. Precisely. THe case in point example these days is traffic light controllers. I know from traffic light controllers; when I was a kid, that was my dad's beat for the City of Boston. Being a geeky kid, I drilled the guys in the signal shop, the few times I got to go there (Saturdays, and such). The old design for traffic signal controllers was that the relays that drove each signal/group were electrically interlocked: the relay that made N/S able to engage it's greens *got its power from* the relay that made E/W red; if there wasn't a red there, you *couldn't* make the other direction green. These days, I'm not sure that's still true: I can *see* the signal change propagate across a row of 5 LED signals from one end to the other. Since I don't think the speed of electricity is slow enough to do that (it's probably on the order of 5ms light to light), I have to assume that it's processor delay as the processor runs a display list to turn on output transistors that drive the LED light heads. That implies to me that it is *physically* possible to get opposing greens (which we refer to, in technical terms as traffic fatalities) out of the controller box... in exactly the same way that it didn't used to be. That's unsettling enough that I'm going to go hunt down a signal mechanic and ask. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: First real-world SCADA attack in US
On Mon, Nov 21, 2011 at 11:16:14PM -0500, Jay Ashworth wrote: That implies to me that it is *physically* possible to get opposing greens (which we refer to, in technical terms as traffic fatalities) out of the controller box... in exactly the same way that it didn't used to be. Not necessarily. Microwave ovens have an interlock system that has 3 sequentially timed microswitches. The first two cut power to the oven, and the third one shorts out the power supply in case the previous two failed, blowing a fuse. The switches are operated by 2 fingers placed on the door so that if the door is bent enough to not seal properly, the switches will be activated in the wrong order causing the shorting switch to operate. This can also happen if you slam the door closed too hard. This is all nice in theory, in practice the microswitches are so flimsy nowadays that I'd not be too surprised if the shorting switch did not succeed in blowing a fuse - and the other two will easily weld together even in normal use (I have seen this happen. Swap the switches and fuse and the oven works again.) The traffic lights can also have some kind of fault-detection logic that sees they are in an illegal state and latches them into a fault mode. IMHO this is stupid extra complexity when relays are obviously 100% correct and reliable for this function, but it seems to be all the rage nowadays to use some kind of proven correct software system for safety critical logic. It is so much sexier than mechanical or electro-mechanical interlocks. Anybody who has seen what kind of bizarre malfunctions failed electrolytics cause in consumer electronics will probably not feel very comfortable trusting traffic lights whose safety relies on software that is proven correct. OTOH, the risk is astronomically small compared to someone just running the red lights. Jussi Peltola
Re: First real-world SCADA attack in US
On Tue, Nov 22, 2011 at 8:35 AM, Mark Radabaugh m...@amplex.net wrote: Having worked on plenty of industrial and other control systems I can safely say security on the systems is generally very poor. The vulnerabilities have existed for years but are just now getting attention. This is a problem that doesn't really need a bunch of new legislation. It's an education / resource issue. The existing methods that have been used for years with reasonable success in the IT industry can 'fix' this problem. I agree, it is mostly education and resources issue . But the environment of control networks is slightly different from IT industry, IMHO. 1) control network people have been living in a kind of isolation for too long and haven't realized that their networks are connected to Big Bad Internet (or at least intranet..) now so the threat model has changed completely. 2) There aren't many published cases of successful (or even unsuccessful) attacks on control networks. As a result, the risk of an attack is considered to have large potential loss and but *very* low probability of occurring and high cost of countermeasures = ignoring.. 3) Interconnections between control networks and normal LANs are a kind of grey area (especially taking into account that both types of networks are run by different teams of engineers). It is very hard to get any technical/security requirements etc - usually none of them exist. And as the whole system as as secure as the weakest element the result is easily predictable. 4) any changes in control network are to be done in much more conservative way. all those apply the patch..oh, damn, it crashed..rollback' doesn't work there. In addition (from my experience which might not be statistically reliable) the testing/lab resources are usually much more limited for control networks; 5) as the life cycle of hwsw is much longer than in IT industry, it is very hard to meet the security requirements w/o significant changes to existing control network (inc. procedures/policies) - but see #4 above.. So there is a gap - those control networks are 10 (20?) years behind internet in terms of security. This gap can be filled but not immediately. The good news that such stories as the one we are discussing could help scary the decision makers..oops, sorry, I was going to say 'raise the level of security awareness' -- SY, Jen Linkova aka Furry
Re: First real-world SCADA attack in US
On Tue, 22 Nov 2011 07:11:43 +0200, Jussi Peltola said: Anybody who has seen what kind of bizarre malfunctions failed electrolytics cause in consumer electronics will probably not feel very comfortable trusting traffic lights whose safety relies on software that is proven correct. Beware of bugs in the above code; I have only proved it correct, not tried it. -- Donald Knuth :) pgpEBPFBJhtki.pgp Description: PGP signature
