Re: [c-nsp] many 2960-X rebooting today

2018-03-16 Thread Bryan Holloway 

We ran into this on 3750Xs back in July.

Sometimes we saw this:

%PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) 
Exception (0x2000)!


c.f.: https://lists.gt.net/cisco/nsp/197344

There are links to Cisco's "response" on the matter ...


On 3/16/18 2:27 PM, Nick Cutting wrote:

I'm reasonably certain it was exploited - the last MSG is related to the bug.

"Stack for process SMI IBC server process running low"


-Original Message-
From: Brandon Applegate [mailto:bran...@burn.net]
Sent: Friday, March 16, 2018 2:28 PM
To: Nick Cutting <ncutt...@edgetg.com>
Cc: cisco-nsp mailing list <cisco-nsp@puck.nether.net>
Subject: Re: [c-nsp] many 2960-X rebooting today

This message originated from outside your organization.



On Mar 16, 2018, at 2:08 PM, Nick Cutting <ncutt...@edgetg.com> wrote:

Thanks we have disabled this now - It is in our new build script, these were 
rolled out a few months ago.

I guess there is no way of seeing if this exploit was executed, perhaps in the 
crashdump somewhere?


I’m struggling to remember.  I want to say you will see a %SYS-5-CONFIG - 
Configured from XXX by YYY message.

The questions become:

-   Are you syslogging out to a server that would have caught this ?
-   Is there any IP in there of where it was originated from ?
- If so - other than an abuse report to the respective ISP and blocking 
the IP - what can be done ?

I guess the other thing I’d add - is if there’s any weak crypto (type 7, or 
even a weak type 5 etc.) passwords or keys in your config, you might want to 
change these.  In other words, assume they have a copy of your config and act 
accordingly.

PS: This is all assuming it was an exploit like this in the first place.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
0641 D285 A36F 533A 73E5  2541 4920 533C C616 703A "For thousands of years men 
dreamed of pacts with demons.
Only now are such things possible."

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] many 2960-X rebooting today

2018-03-16 Thread Nick Cutting 
I'm reasonably certain it was exploited - the last MSG is related to the bug.

"Stack for process SMI IBC server process running low"


-Original Message-
From: Brandon Applegate [mailto:bran...@burn.net] 
Sent: Friday, March 16, 2018 2:28 PM
To: Nick Cutting <ncutt...@edgetg.com>
Cc: cisco-nsp mailing list <cisco-nsp@puck.nether.net>
Subject: Re: [c-nsp] many 2960-X rebooting today

This message originated from outside your organization.


> On Mar 16, 2018, at 2:08 PM, Nick Cutting <ncutt...@edgetg.com> wrote:
> 
> Thanks we have disabled this now - It is in our new build script, these were 
> rolled out a few months ago.
> 
> I guess there is no way of seeing if this exploit was executed, perhaps in 
> the crashdump somewhere?

I’m struggling to remember.  I want to say you will see a %SYS-5-CONFIG - 
Configured from XXX by YYY message.

The questions become:

-   Are you syslogging out to a server that would have caught this ?
-   Is there any IP in there of where it was originated from ?
- If so - other than an abuse report to the respective ISP and blocking 
the IP - what can be done ?

I guess the other thing I’d add - is if there’s any weak crypto (type 7, or 
even a weak type 5 etc.) passwords or keys in your config, you might want to 
change these.  In other words, assume they have a copy of your config and act 
accordingly.

PS: This is all assuming it was an exploit like this in the first place.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
0641 D285 A36F 533A 73E5  2541 4920 533C C616 703A "For thousands of years men 
dreamed of pacts with demons.
Only now are such things possible."

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] many 2960-X rebooting today

2018-03-16 Thread Brandon Applegate 

> On Mar 16, 2018, at 2:08 PM, Nick Cutting  wrote:
> 
> Thanks we have disabled this now - It is in our new build script, these were 
> rolled out a few months ago.
> 
> I guess there is no way of seeing if this exploit was executed, perhaps in 
> the crashdump somewhere?

I’m struggling to remember.  I want to say you will see a %SYS-5-CONFIG - 
Configured from XXX by YYY message.

The questions become:

-   Are you syslogging out to a server that would have caught this ?
-   Is there any IP in there of where it was originated from ?
- If so - other than an abuse report to the respective ISP and blocking 
the IP - what can be done ?

I guess the other thing I’d add - is if there’s any weak crypto (type 7, or 
even a weak type 5 etc.) passwords or keys in your config, you might want to 
change these.  In other words, assume they have a copy of your config and act 
accordingly.

PS: This is all assuming it was an exploit like this in the first place.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
0641 D285 A36F 533A 73E5  2541 4920 533C C616 703A
"For thousands of years men dreamed of pacts with demons.
Only now are such things possible."



signature.asc
Description: Message signed with OpenPGP
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] many 2960-X rebooting today

2018-03-16 Thread Nick Cutting 
Thanks we have disabled this now - It is in our new build script, these were 
rolled out a few months ago.

I guess there is no way of seeing if this exploit was executed, perhaps in the 
crashdump somewhere?

-Original Message-
From: cisco-nsp [mailto:cisco-nsp-boun...@puck.nether.net] On Behalf Of Brandon 
Applegate
Sent: Friday, March 16, 2018 1:19 PM
To: cisco-nsp mailing list <cisco-nsp@puck.nether.net>
Subject: Re: [c-nsp] many 2960-X rebooting today

This message originates from outside of your organisation.



> On Mar 16, 2018, at 12:49 PM, Nick Cutting <ncutt...@edgetg.com> wrote:
> 
> Anyone seen a number of internet facing 2960-X switches restart today?
> 
> We have had 3 different clients, 6 different switches all reboot today.
> 
> No uptime in common, no code version in common.
> 
> One of them has WS-C2960X-24TS-L - Version 15.2(2)E6
> 
> The only thing they do have in common is that they have internet IP addresses 
> for MGT - with SSH allowed, locked down to certain public IP's.
> 
> Just wondering if this may be the execution of an exploit by a baddie.
> 
> Nick

I haven’t - but the first thing that popped into my head was:

https://github.com/Sab0tag3d/SIET

You might want to scan/nmap your switches.  I know some folks that got hit with 
this last year.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
0641 D285 A36F 533A 73E5  2541 4920 533C C616 703A "For thousands of years men 
dreamed of pacts with demons.
Only now are such things possible."

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] many 2960-X rebooting today

2018-03-16 Thread Brandon Applegate 


> On Mar 16, 2018, at 12:49 PM, Nick Cutting  wrote:
> 
> Anyone seen a number of internet facing 2960-X switches restart today?
> 
> We have had 3 different clients, 6 different switches all reboot today.
> 
> No uptime in common, no code version in common.
> 
> One of them has WS-C2960X-24TS-L - Version 15.2(2)E6
> 
> The only thing they do have in common is that they have internet IP addresses 
> for MGT - with SSH allowed, locked down to certain public IP's.
> 
> Just wondering if this may be the execution of an exploit by a baddie.
> 
> Nick

I haven’t - but the first thing that popped into my head was:

https://github.com/Sab0tag3d/SIET

You might want to scan/nmap your switches.  I know some folks that got hit with 
this last year.

--
Brandon Applegate - CCIE 10273
PGP Key fingerprint:
0641 D285 A36F 533A 73E5  2541 4920 533C C616 703A
"For thousands of years men dreamed of pacts with demons.
Only now are such things possible."



signature.asc
Description: Message signed with OpenPGP
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[c-nsp] many 2960-X rebooting today

2018-03-16 Thread Nick Cutting 
Anyone seen a number of internet facing 2960-X switches restart today?

We have had 3 different clients, 6 different switches all reboot today.

No uptime in common, no code version in common.

One of them has WS-C2960X-24TS-L - Version 15.2(2)E6 

The only thing they do have in common is that they have internet IP addresses 
for MGT - with SSH allowed, locked down to certain public IP's.

Just wondering if this may be the execution of an exploit by a baddie.

Nick







___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/