Hi Phil,
Answers below:
1) IOS - s72033-advipservicesk9_wan-mz.122-18.SXF17a.bin
2) HSRP configured between two core 6509's. SVI is VLAN1 (I know don't ask)
trunked between the cores via 10G. Only ports in VLAN1 on one core switch
are impacted and seeing the flooding.
3) Building floor switches connect to both cores (Routed and running EIGRP)
4) Spanning Tree Below:
Core1:
spanning-tree mode pvst
spanning-tree vlan 1-199,336,503-930 priority 16384
Core2:
spanning-tree mode pvst
spanning-tree vlan 1-199,336,503-930 priority 0
5) No rate limiting or CoPP configured. We are seeing drops even when the
CPU is not hitting 100% (most likely due to ASIC oversubscription).
6) Source of traffic is unknown at this stage. Will turn to wireshark
tomorrow.
7) I don't believe there are any L2 loops. If spanning-tree was an issue I
would think the CPU would gradually hit 100% and stay there.
We are seeing output drops on interfaces and oversubscription of ASICs as a
result of this flooding which I think is the main culprit for the brief
connectivity outages. Is there a way similar to CoPP to protect the ASICs to
ensure they are never 100% utilised? Egress shaping on all suspect ports?
Thanks,
Aaron.
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of JC Cockburn
Sent: Wednesday, 14 July 2010 8:03 PM
To: 'Phil Mayers'
Cc: [email protected]
Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
Importance: High
Hi Phil,
I had a problem like this last year on 6500's.
It was related to bug: CSCsk23521
Basically a server in our datacenter used multicast addresses in the range
allocated for BPDU's, and this just killed the SP (100% CPU...).
If you do a "remote command switch sh proc cpu" on the 6500 you can see if
the SP CPU is under fire...
Cheers
JC
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Phil Mayers
Sent: Wednesday, July 14, 2010 1:41 PM
To: [email protected]
Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
On 14/07/10 11:30, Aaron Riemer wrote:
> Hi Group,
>
>
>
> We are having trouble with unicast flooding on a particular VLAN and
> associated ports and as a result brief spikes in CPU usage on one of our
> 6509 core switches.
>
>
>
> ARP and MAC timeouts are set to default and we haven't had problems with
> this in the past. The problem is I believe this is causing brief 100%
spikes
> within the SP or RP and as a result brief connectivity outages.
Which is it? SP or RP?
>
>
>
> We have narrowed down the source of the unicast flooding but we need to
know
> why it is occurring.
Rather more info required I think.
* IOS version
* Config of ports & SVIs in question
* Nature of downstream devices (if any)
* spanning tree config (if any)
* rough idea of the size of the ARP & MAC tables
* Any MLS rate-limit or CoPP config
* Nature of the source of the unicast-flooded traffic
* Any possibility of loops in the network?
> Has anyone experienced this in the past? Could unicast flooding over
> multiple interfaces account for this kind of behaviour?
Anything punted to the CPU at high rate could cause this kind of thing.
That's why MLS limiters and CoPP are important on this platform, even
with all their limitations.
_______________________________________________
cisco-nsp mailing list [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
#sh proc cpu hist
11111 11111 11111
6666888882222233333222223333377777111116666655555222228888
100
90
80
70
60
50
40
30
20 ***** *****
10 ********* ***** ***** ********** ****
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1111121711111661111211111111112111111111111111111111112111
7554702837532995254235247545231644546447462368336334730243
100
90
80 *
70 * **
60 * **
50 * **
40 * **
30 * **
20 *** ** # ** *** * * * ** * ** * * * * ** * * *
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
2222622473339222229222929299738899991112111122212772222223221122222222
3455988052031270011439343642004940359970789816991862061805647914201649
100 *
90 * * * * ** *****
80 * * * * * ** ****** **
70 * * * * * * *** ****** **
60 * * * * * * *** ****** **
50 * * * * * * *** ****** **
40 * ** * * * * *** ****** ** *
30 *********** * * ** ************ ** ** * * ** * *
20 **********************************************************************
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
west-core#sh mod
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
2 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
3 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
4 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
5 2 Supervisor Engine 720 (Active) WS-SUP720-BASE
6 8 CEF720 8 port 10GE with DFC WS-X6708-10GE
7 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX
8 24 CEF720 24 port 1000mb SFP WS-X6724-SFP
9 48 CEF720 48 port 1000mb SFP WS-X6748-SFP
Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
1 0017.5975.cc50 to 0017.5975.cc7f 10.2 7.2(1) 8.5(0.46)RFW Ok
2 000d.29d1.4c20 to 000d.29d1.4c4f 3.0 7.2(1) 8.5(0.46)RFW Ok
3 000d.29da.82d0 to 000d.29da.82ff 3.0 7.2(1) 8.5(0.46)RFW Ok
4 0013.c448.2988 to 0013.c448.29b7 10.1 7.2(1) 8.5(0.46)RFW Ok
5 0013.7f0d.b7f0 to 0013.7f0d.b7f3 3.3 8.1(3) 12.2(18)SXF1 Ok
6 c47d.4f8f.5000 to c47d.4f8f.5007 2.1 12.2(18r)S1 12.2(18)SXF1 Ok
7 0005.7444.7dac to 0005.7444.7ddb 10.2 7.2(1) 8.5(0.46)RFW Ok
8 0016.c81c.a8ec to 0016.c81c.a903 2.3 12.2(14r)S5 12.2(18)SXF1 Ok
9 0015.c6b4.2b34 to 0015.c6b4.2b63 1.5 12.2(14r)S5 12.2(18)SXF1 Ok
Mod Sub-Module Model Serial Hw Status
---- --------------------------- ------------------ ----------- ------- -------
5 Policy Feature Card 3 WS-F6K-PFC3A 2.5 Ok
5 MSFC3 Daughterboard WS-SUP720 2.5 Ok
6 Distributed Forwarding Card WS-F6700-DFC3C 1.4 Ok
8 Centralized Forwarding Card WS-F6700-CFC 2.0 Ok
9 Centralized Forwarding Card WS-F6700-CFC 2.0 Ok
Mod Online Diag Status
---- -------------------
1 Pass
2 Pass
3 Pass
4 Pass
5 Pass
6 Pass
7 Pass
8 Pass
9 Pass
#remote command switch sh proc cpu hist
111111111111111 1111111111 11111
0000011111000009999900000111118888844444888886666699999777
100
90
80
70
60
50
40
30
20
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1111111211111111121111111111111111111111111111121111111122
4657352033485523416675532868632396753884233144304325646433
100
90
80
70
60
50
40
30
20 *** * * *** ****** **** **** ** * ** * **
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
2122222242222222122222322242222423832222222232222222223222222222222222
3903740453452352932200901294367140517955448406766244960505943468485348
100
90 *
80 *
70 *
60 *
50 * * *
40 * * * * *
30 * * * * * * *** ******* * ***** **** ** ** ** *
20 **********************************************************************
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
_______________________________________________
cisco-nsp mailing list [email protected]
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
