Hello List, Just for the record, I will post this in case some guys out there have the same problem some day.
Last friday, one of my core routers, a Cisco 6509 with two SUP720-3BXL modules running s72033-advipservicesk9_wan-mz.122-33.SXH2a, crashed and restarted out of the blue. Crashfile info says the following : *Jun 11 06:43:06.310: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 30 seconds [5/1] Jun 11 06:43:36.310: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 60 seconds [5/1] Jun 11 06:44:06.310: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 90 seconds [5/1] *Jun 11 06:44:25.366: SP: icc_send_request_internal: ipc_send_rpc_blocked failed, result 6 Jun 11 06:44:25.366: SP: -Traceback= 40BC1538 40BC16F8 40BC19E0 40B11AE4 40B120D0 40752F58 40752F44 *Jun 11 06:44:36.310: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 120 seconds [5/1] *Jun 11 06:44:51.366: SP: IPC: Message 43EDD2BC timed out waiting for Ack Jun 11 06:44:51.366: SP: IPC: MSG: ptr: 0x43EDD2BC, flags: 0x20101, retries: 21, seq: 0x2155C10, refcount: 2, retry: 00:00:00, rpc_result = 0x0, data_buffer = 0x503FAF5C, header = 0x8C7A7C8, data = 0x8C7A7E8 || HDR: src: 0x10000, dst: 0x2150010, index: 0, seq: 23568, sz: 80, type: 1, flags: 0x404 hi: 0x6F4F386, lo: 0x8C7A7E8 || DATA: 00 00 00 05 00 00 00 00 00 00 1B 59 00 00 00 01 00 00 00 07 Jun 11 06:44:51.366: SP: IPC: Send failed: IPC msg timeout MSG: ptr: 0x43EDD2BC, flags: 0x20101, retries: 21, seq: 0x2155C10, refcount: 2, retry: 00:00:00, rpc_result = 0x0, data_buffer = 0x503FAF5C, header = 0x8C7A7C8, data = 0x8C7A7E8 || HDR: src: 0x10000, dst: 0x2150010, index: 0, seq: 23568, sz: 80, type: 1, flags: 0x404 hi: 0x6F4F386, lo: 0x8C7A7E8 || DATA: 00 00 00 05 00 00 00 00 00 00 1B 59 00 00 00 01 00 00 00 07 Jun 11 06:44:51.366: SP: -Traceback= 403E6CB0 403EB96C 403EC00C 40405988 40752F58 40752F44 Jun 11 06:44:51.366: %C6K_PROCMIB-SP-3-IPC_TRANSMIT_FAIL: Failed to send process statistics update : error code = timeout -Traceback= 409A39A4 409A39F4 409A3C00 409A3E60 40752F58 40752F44 *Jun 11 06:45:06.310: %CPU_MONITOR-SP-6-NOT_HEARD: CPU_MONITOR messages have not been heard for 150 seconds [5/1] Jun 11 06:45:36.310: %CPU_MONITOR-SP-3-TIMED_OUT: CPU_MONITOR messages have failed, resetting system [5/1] * *%Software-forced reload * 06:45:36 UTC Fri Jun 11 2010: Breakpoint exception, CPU signal 23, PC = 0x41183348 For some reason, RP and SP were not able to communicate using the EOBC. I have googling around and looks like folks out there (among c-nsp too) have already seen this for Cisco 6500 and 7600. In this particular case, Cisco says : CPU_MONITOR-3-TIMED_OUT or CPU_MONITOR-6-NOT_HEARD Problem Problème The switch reports these error messages: CPU_MONITOR-3-TIMED_OUT: CPU monitor messages have failed, resetting system CPU_MONITOR-6-NOT_HEARD: CPU monitor messages have not been heard for [dec] seconds Description Description These messages indicate that CPU monitor messages have not been heard for a significant amount of time. A time-out most probably occurs, which resets the system. [dec] is the number of seconds. The problem possibly occurs because of these reasons: - Badly seated line card or module <=== Not likely - Bad ASIC or bad backplane <=== Not likely - Software bugs <=== Probably - Parity error <=== Don't know - High traffic in the Ethernet out of band channel (EOBC) channel <=== According to the IPC stats, nothing fancy The EOBC channel is a half duplex channel that services many other functions, which includes Simple Network Management Protocol (SNMP) traffic and packets that are destined to the switch. If the EOBC channel is full of messages because of a storm of SNMP traffic, then the channel is subjected to collisions. When this happens, EOBC is possibly not able to carry IPC messages. This makes the switch display the error message. Workaround Contournement Reseat the line card or module. If a maintenance window can be scheduled, reset the switch in order to clear any transient issues. Personally, I'd say I hit a bug with this but I can't seem to find it using cisco web tools. Anyone could point me to the right direction ? Thank you all. Best regards. Y. -- Youssef BENGELLOUN-ZAHR ……………………………………………… Ingénieur Réseaux et Télécoms Technopole de l'Aube en Champagne - BP 601 - 10901 TROYES Cedex 9 Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE Tel +33 (0) 825 000 720 Tel. direct +33 (0) 1 77 35 59 14 Tel. portable +33 (0) 6 22 42 63 80 Email [email protected] ……………………………………………………………………………….....www.720.fr _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
