Re: CISCO 0-day exploits
Large operators have very little to gain from calling out the equipment suppliers. In my personal experience large operators are already getting custom code builds based on their exact requirements, which include disabling many of the “standard” features they don’t use. Sent from my iPhone > On Feb 11, 2020, at 9:48 AM, Ahmed Borno wrote: > > > Being realistic, as you mentioned, these vendors do not have the right > incentive. > > Thats one thing that operators can do and maybe it should be a recurring > theme at NANOG, calling out vendors to put some sanity and logic into how > iACLs and CoPP are handled. They can do a lot if they cared to spend some $ > on being creative, maybe a BoF for this specific topic. > > Creativity in the form of ways to avoid the fragile stacks and L3 packet of > death, They can even separate the Mgmt plane from the Control plane if they > are serious about it, they can enforce iACL on Mgmt interfaces, they can have > logic to validate packets before they are processed, and to be fair, this > needs to happen in the existing install base too, not only the new ones. I am > trying to say that if they cant hire skilled programmers then they should > show innovation around the most vulnerable part of their code...the trusting > nature of protocols. > > P.S: How many junior network engineers care to turn on authentication on L2 > segments. > > ~A > >> On Tue, Feb 11, 2020 at 6:24 AM Saku Ytti wrote: >> On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: >> >> > Sorry for the sad tone, i just wish network operators would find a way to >> > challenge these vendors and call their less than optimal quality. >> >> It's hard, TINA. We can talk about white label, but in the end of the >> day, that box is just as proprietary as rest of them, because you >> can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs >> were not supported, because vendors thought the specs were their >> secret sauce. >> When some vendor finally releases full specs on github including P4 >> compiler target for their chip and will sell chip on their web for 1 >> unit at x USD, we may start to see some real progress, we can start >> building open source NOS with data-planes. >> >> Maybe INTC could start the revolution with Tofino. Ship PCI cards with >> Tofino and few 100GE ports (local switching support) and open it up >> entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like >> they have lot to lose, considering terrible market performance. >> >> >> -- >> ++ytti
Re: CISCO 0-day exploits
Being realistic, as you mentioned, these vendors do not have the right incentive. Thats one thing that operators can do and maybe it should be a recurring theme at NANOG, calling out vendors to put some sanity and logic into how iACLs and CoPP are handled. They can do a lot if they cared to spend some $ on being creative, maybe a BoF for this specific topic. Creativity in the form of ways to avoid the fragile stacks and L3 packet of death, They can even separate the Mgmt plane from the Control plane if they are serious about it, they can enforce iACL on Mgmt interfaces, they can have logic to validate packets before they are processed, and to be fair, this needs to happen in the existing install base too, not only the new ones. I am trying to say that if they cant hire skilled programmers then they should show innovation around the most vulnerable part of their code...the trusting nature of protocols. P.S: How many junior network engineers care to turn on authentication on L2 segments. ~A On Tue, Feb 11, 2020 at 6:24 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: > > > Sorry for the sad tone, i just wish network operators would find a way > to challenge these vendors and call their less than optimal quality. > > It's hard, TINA. We can talk about white label, but in the end of the > day, that box is just as proprietary as rest of them, because you > can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs > were not supported, because vendors thought the specs were their > secret sauce. > When some vendor finally releases full specs on github including P4 > compiler target for their chip and will sell chip on their web for 1 > unit at x USD, we may start to see some real progress, we can start > building open source NOS with data-planes. > > Maybe INTC could start the revolution with Tofino. Ship PCI cards with > Tofino and few 100GE ports (local switching support) and open it up > entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like > they have lot to lose, considering terrible market performance. > > > -- > ++ytti >
Re: CISCO 0-day exploits
On Tue, 11 Feb 2020 at 16:09, Ahmed Borno wrote: > Sorry for the sad tone, i just wish network operators would find a way to > challenge these vendors and call their less than optimal quality. It's hard, TINA. We can talk about white label, but in the end of the day, that box is just as proprietary as rest of them, because you can't buy BRCM and make it open. It's like 90s of Linux, GPUs and NICs were not supported, because vendors thought the specs were their secret sauce. When some vendor finally releases full specs on github including P4 compiler target for their chip and will sell chip on their web for 1 unit at x USD, we may start to see some real progress, we can start building open source NOS with data-planes. Maybe INTC could start the revolution with Tofino. Ship PCI cards with Tofino and few 100GE ports (local switching support) and open it up entirely. Maybe JNPR could ship Trio PCI cards, why not, it's not like they have lot to lose, considering terrible market performance. -- ++ytti
Re: CISCO 0-day exploits
I remember my conversation with an executive one day, where I was enlightened on corporate greed. I asked, why is there no investment in quality code, and I was schooled. The exec said, one dollar spent on fixing bugs, returns zero dollars but one dollar spent on nee features brings in 3 dollars ;) These vendors tried different DSL throughout the years and it is way too difficult for them to execute on that shift, too many things at stake, i mean their business, not the end customer...of course. They are not even being 'creative', they can easily pull some tricks like the one you mentioned (compiler built sanity) in their system configuration logic, like you cant turn on an interface without an iACL applied...but then why would they :) Sorry for the sad tone, i just wish network operators would find a way to challenge these vendors and call their less than optimal quality. ~A On Tue, Feb 11, 2020 at 2:05 AM Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > > > So yeah iACLs, CoPP and all sorts of basic precautions are needed, but > I'm thinking something more needs to be done, specially if these ancient > code stacks are being imported into new age 'IoT' devices, multiplying the > attack vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. > > I also think the way we develop needs to be fundamentally rethought, > we need to stop believing I am the person who can code working C, it's > the other people who are incompetent. At some point we should look, > are the tools we using the right tools? Can we move complexity from > human to computers at compile time to create more guarantees of > correctness? MSFT claims 70% of their bugs are memory safety, issue > which could be solved almost perfectly programmatically by making the > compiler and language smarter, not the person more resistant to > mistakes. > I think ANET at least for some part essentially writes their own DSL > which compiles to C++, I think solution like this for any large > long-lived project probably quickly pays dividends in quality, because > you can address lot of the systematic errors during the compilation > time and in DSL design. > > > -- > ++ytti >
Re: CISCO 0-day exploits
On 2/11/2020 2:04 AM, Saku Ytti wrote: > On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > >> So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm >> thinking something more needs to be done, specially if these ancient code >> stacks are being imported into new age 'IoT' devices, multiplying the attack >> vector by a factor of too many. > > I can't see situation getting better. Why should vendor invest in high > quality code, certainly the cultural shift will cost something, it's > not 0 cost and what is the upside? If IOS and JunOS realistically were > significantly less buggy many of us would stop buying support, because > we either know how to configure these or can get help faster free from > the community, we largely need the support because the software > quality is so bad _everyone_ finds new bugs all the time and we don't > have the source code to fix it as a community. > So I suspect significantly better quality software would at least > initially cost more to produce and it would reduce revenue in loss of > support. Yeah, things need to get better, and soon. At least, some things... Was I too subtle just now? -- Harlan Stenn http://networktimefoundation.org - be a member!
Re: CISCO 0-day exploits
On Tue, 11 Feb 2020 at 09:09, Ahmed Borno wrote: > So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm > thinking something more needs to be done, specially if these ancient code > stacks are being imported into new age 'IoT' devices, multiplying the attack > vector by a factor of too many. I can't see situation getting better. Why should vendor invest in high quality code, certainly the cultural shift will cost something, it's not 0 cost and what is the upside? If IOS and JunOS realistically were significantly less buggy many of us would stop buying support, because we either know how to configure these or can get help faster free from the community, we largely need the support because the software quality is so bad _everyone_ finds new bugs all the time and we don't have the source code to fix it as a community. So I suspect significantly better quality software would at least initially cost more to produce and it would reduce revenue in loss of support. I also think the way we develop needs to be fundamentally rethought, we need to stop believing I am the person who can code working C, it's the other people who are incompetent. At some point we should look, are the tools we using the right tools? Can we move complexity from human to computers at compile time to create more guarantees of correctness? MSFT claims 70% of their bugs are memory safety, issue which could be solved almost perfectly programmatically by making the compiler and language smarter, not the person more resistant to mistakes. I think ANET at least for some part essentially writes their own DSL which compiles to C++, I think solution like this for any large long-lived project probably quickly pays dividends in quality, because you can address lot of the systematic errors during the compilation time and in DSL design. -- ++ytti
Re: CISCO 0-day exploits
Disclaimer, I do not work for any vendor right now, and I don't sell any product that might benefit from scaring anyone, so this is just some whining for a real issue that someone needs to do something about. I've worked for the CDP vendor for a long time, and I do concur to what Saku is saying...the L3 packet of death [threat] is very real, and I'd like to take this opportunity (the buzz around CDPwn) to say a thing or two about these 'soft, mushy and vulnerable' code stacks we have running all over the world, under our feet, waiting for someone with the right incentive to take advantage of. IMHO, "Skilled" software developers, and in parallel...'software exploiters/reverse engineers' haven't been paying attention to these 'infrastructure' boxes (for now), maybe it is because they always had other pieces of the technology stack to work with, and these other tech. stacks were much more rewarding to spend time on (I'm quite sure Node.js / Kubernetes for example...will have a lot more vulnerability researchers looking at them than CDP/LLDP/SNMPetc code). < and That is a serious sustainability issue on our hands, the risk here is very high; when it comes to infrastructure security of nations, especially in a world where miscreants are no longer script kiddies but actual nation sponsored soldiers...Even MBS is doing it in person. Because the moment some miscreants from some oppressive regime decides to do damage, and not necessarily remote code execution as many might think, but more on the 'L3 packet of death' kind of situation that Saku mentioned earlier, these miscreants have a lot to play with, and the attacks vector is huge, it is green and it is ready for the ripe. In my life time, I've looked at so many 'DDTS' descriptions, and I saw nothing but an unwritten disclaimer of : 'I can be easily used for DDoS'...and that is the case even if *SIRT did their brief analysis of these bugs, so again, if some miscreants found it in themselves to look at bugs with the right 'optics', we are going to be in an interesting situation. Luckily, we haven't seen a CDPwn/STPwn/BGPwn/NTPwn/*.*Pwn...etc worm/ransomware yet, but we also don't have reason to think its not possible, and to make matters worse, the code these babies are running is ancient (in every possible way), many of the libraries used to develop that code is glibc_ishy like in nature, and to make matters a bit more interesting, patching those babies is not easy, and the nature of their software architecture makes them even much more fragile than any piece of cheap IP camera out there on the internet or on enterprise networks. So yeah iACLs, CoPP and all sorts of basic precautions are needed, but I'm thinking something more needs to be done, specially if these ancient code stacks are being imported into new age 'IoT' devices, multiplying the attack vector by a factor of too many. ~A On Mon, Feb 10, 2020 at 5:42 AM Saku Ytti wrote: > On Mon, 10 Feb 2020 at 13:52, Jean | ddostest.me via NANOG > wrote: > > > I really thought that more Cisco devices were deployed among NANOG. > > > > I guess that these devices are not used anymore or maybe that I > > understood wrong the severity of this CVE. > > Network devices are incredibly fragile and mostly work because no one > is motivated to bring the infrastructure down. Getting any arbitrary > vendor down if you have access to it on L2 is usually so easy you > accidentally find ways to do it. > There are various L3 packet of deaths where existing infra can be > crashed with single packet, almost everyone has no or ridiculously > broken iACL and control-plane protection, yet business does not seem > to suffer from it. > > Probably lower availability if you do upgrade your devices just > because there is a known issue, due to new production affecting > issues. > > -- > ++ytti >
Re: CISCO 0-day exploits
On 10/02/2020 18:13, Scott Weeks wrote: > Just because you use cisco devices doesn't mean you have to use > their proprietary protocols, such as EIGRP or CDP. OSPF or LLDP > work just fine and interoperate with other vendors... :) The CDPwn vulnerability covers similar vulnerabilities in LLDP, and does indeed demonstrate that network segmentation (i.e. "dude it's just L2") is not the last word in mitigating against said vulnerabilities. You ought to all be far more concerned, IMO. -- Tom
Re: CISCO 0-day exploits
> > I really thought that more Cisco devices were deployed among NANOG. > > I guess that these devices are not used anymore or maybe that I > understood wrong the severity of this CVE. A proper network design helps to mitigate flaws like this. If you have CDP off, which many people do, then this exploit is not that big of a deal to you. If your devices are on a management network then it’s not that big of a deal. Just because a certain vendor has vulnerabilities exposed doesn’t it’s an all hand on deck scenario. Many of the folks on NANOG have a good grasp of network design. Sure, some don’t. But for the most part they do. Justin Wilson li...@mtin.net — https://j2sw.com - All things jsw (AS209109) https://blog.j2sw.com - Podcast and Blog
Re: CISCO 0-day exploits
--- nanog@nanog.org wrote: From: "Jean | ddostest.me via NANOG" > https://www.armis.com/cdpwn/ > > What's the impact on your network? Everything is under control? --- I really thought that more Cisco devices were deployed among NANOG. I guess that these devices are not used anymore or maybe that I understood wrong the severity of this CVE. --- Just because you use cisco devices doesn't mean you have to use their proprietary protocols, such as EIGRP or CDP. OSPF or LLDP work just fine and interoperate with other vendors... :) scott
Re: CISCO 0-day exploits
On 10/02/2020 13:40, Saku Ytti wrote: > There are various L3 packet of deaths where existing infra can be > crashed with single packet, almost everyone has no or ridiculously > broken iACL and control-plane protection, yet business does not seem > to suffer from it. The cynic in me would suggest that we haven't had a World War in a while; business is far too good. -- Tom
Re: CISCO 0-day exploits
I remember a Cisco device with an ACL that was leaking. It was a 20 lines ACL with few lines to drop some packets based on UDP ports. When under heavy stress, nearly line rate, we would see some of these packets going through the ACL. I said to my peers that the ACL was leaking. They didn't believe me so I showed them the netflows. We were very surprised to see that. We thought that drop means drop. On 2020-02-10 08:40, Saku Ytti wrote: On Mon, 10 Feb 2020 at 13:52, Jean | ddostest.me via NANOG wrote: I really thought that more Cisco devices were deployed among NANOG. I guess that these devices are not used anymore or maybe that I understood wrong the severity of this CVE. Network devices are incredibly fragile and mostly work because no one is motivated to bring the infrastructure down. Getting any arbitrary vendor down if you have access to it on L2 is usually so easy you accidentally find ways to do it. There are various L3 packet of deaths where existing infra can be crashed with single packet, almost everyone has no or ridiculously broken iACL and control-plane protection, yet business does not seem to suffer from it. Probably lower availability if you do upgrade your devices just because there is a known issue, due to new production affecting issues.
Re: CISCO 0-day exploits
On Mon, 10 Feb 2020 at 13:52, Jean | ddostest.me via NANOG wrote: > I really thought that more Cisco devices were deployed among NANOG. > > I guess that these devices are not used anymore or maybe that I > understood wrong the severity of this CVE. Network devices are incredibly fragile and mostly work because no one is motivated to bring the infrastructure down. Getting any arbitrary vendor down if you have access to it on L2 is usually so easy you accidentally find ways to do it. There are various L3 packet of deaths where existing infra can be crashed with single packet, almost everyone has no or ridiculously broken iACL and control-plane protection, yet business does not seem to suffer from it. Probably lower availability if you do upgrade your devices just because there is a known issue, due to new production affecting issues. -- ++ytti
Re: CISCO 0-day exploits
On Monday, 10 February, 2020 11:50, "Jean | ddostest.me via NANOG" said: > I really thought that more Cisco devices were deployed among NANOG. > > I guess that these devices are not used anymore or maybe that I > understood wrong the severity of this CVE. The phones / cameras side of it seems very much like an Enterprise problem. I'm not sure what the split is here of people operating Enterprise networks vs Service Provider, but I'd expect a skew towards the latter. There is some SP kit on the vulnerable list too, but in my experience, CDP there is used to validate L2 topologies amongst SP kit only, and disabled on customer-facing ports. So maybe a "we *do* have CDP turned off everywhere we don't need it, right?" sanity-check, but not necessarily a rush to patch. I'd have expected greater consternation had this hit vanilla-IOS/XE boxes that are likely to be in managed CPE roles, such as ISR and ASR1K. There I can see the potential for CDP to be enabled customer-facing, either for diagnostics with the customer, or for the voice / data VLAN stuff outlined in the article. Regards, Tim.
Re: CISCO 0-day exploits
I really thought that more Cisco devices were deployed among NANOG. I guess that these devices are not used anymore or maybe that I understood wrong the severity of this CVE. Happy NANOG #78 Cheers Jean On 2020-02-07 09:21, Jean | ddostest.me via NANOG wrote: CDPwn: 5 new zero-day Cisco exploits https://www.armis.com/cdpwn/ What's the impact on your network? Everything is under control? Jean
CISCO 0-day exploits
CDPwn: 5 new zero-day Cisco exploits https://www.armis.com/cdpwn/ What's the impact on your network? Everything is under control? Jean
