Try ELAM capture... TAC can help with the ELAM or CPU profiling. -Ozgur
----- Original Message ---- From: Rin <[email protected]> To: [email protected] Sent: Fri, 1 October, 2010 10:34:08 Subject: [c-nsp] High CPU caused by interupt on 7600 router Hi group, I have a 7609 PE router with 2 TenGigabit interfaces uplink to other P routers and Gigabit interfaces downlink to access switch. Recently, I detect the CPU of that router is punted approximately every 35mins and result in below output: ============================================================== PE-Router#sho proc cpu hist 6666666644444555552222211111333331111133333111115555511111 100 90 80 70 60 50 40 30 20 10 ******** ***** ***** 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 CPU% per second (last 60 seconds) 88 6 99 5664764335455448544554465856444344555656540444344454357766 100 90 ** ** 80 ** ** 70 ** * ** 60 ** * ** 50 ** * ** 40 #* * ** 30 #* * #* 20 #* * #* 10 *## #* * ** ** ** ***** *******## * ***** 0....5....1....1....2....2....3....3....4....4....5....5.... 0 5 0 5 0 5 0 5 0 5 CPU% per minute (last 60 minutes) * = maximum CPU% # = average CPU% 989968999998999999999979999988898989999887299889 8888 9999988989884889 7653706465425842251504506532186452964008644208308676970312489352156780 100 * * * ** ** * * ** * 90 **** ***** ********** ***** ************ *** * **** ********** * *** 80 **** ************************************ ***** **** ************ *** 70 ****************************************** ***** **** ************ *** 60 ****************************************** ***** **** ************ *** 50 ****************************************** ***** **** **************** 40 ****************************************** ***** **** **************** 30 ****************************************** ***** **** **************** 20 ************************************************ **** **************** 10 ********************************************************************** 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7. 0 5 0 5 0 5 0 5 0 5 0 5 0 CPU% per hour (last 72 hours) * = maximum CPU% # = average CPU% ============================================================== I can confirm this caused by interrupt by configure ERM feature to detect the interrupt on the router and check CPU right after the interrupt happens. The following output display the logging: ============================================================================ === *Sep 29 13:39:38.830: %SYS-4-CPURESRISING: System is seeing global cpu util 97% at total level more than the configured critical limit 80 % *Sep 29 13:39:38.830: %SYS-4-CPURESRISING: System is seeing global cpu util 97% at interrupt level more than the configured critical limit 70 % *Sep 29 13:39:48.830: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at total level for the configured critical limit 80%, current value 44% *Sep 29 13:39:48.830: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at interrupt level for the configured critical limit 70%, current value 44% *Sep 29 13:42:13.750: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.16.252.251) *Sep 29 14:14:11.122: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.16.252.251) *Sep 29 14:16:58.862: %SYS-4-CPURESRISING: System is seeing global cpu util 90% at total level more than the configured critical limit 80 % *Sep 29 14:16:58.862: %SYS-4-CPURESRISING: System is seeing global cpu util 90% at interrupt level more than the configured critical limit 70 % *Sep 29 14:17:08.862: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at total level for the configured critical limit 80%, current value 70% *Sep 29 14:17:13.862: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at interrupt level for the configured critical limit 70%, current value 37% *Sep 29 14:22:18.046: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (172.16.252.251) *Sep 29 14:56:58.886: %SYS-4-CPURESRISING: System is seeing global cpu util 86% at total level more than the configured critical limit 80 % *Sep 29 14:56:58.886: %SYS-4-CPURESRISING: System is seeing global cpu util 86% at interrupt level more than the configured critical limit 70 % *Sep 29 14:57:08.886: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at total level for the configured critical limit 80%, current value 78% *Sep 29 14:57:13.886: %SYS-6-CPURESFALLING: System is no longer seeing global high cpu at interrupt level for the configured critical limit 70%, current value 44% ============================================================================ === ============================================================================ === PE-Router#sho proc cpu sort 5min CPU utilization for five seconds: 1%/1%; one minute: 3%; five minutes: 6% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 436 1566429721118606284 140 0.00% 0.35% 0.35% 0 Port manager per 7 205461504 12733776 16135 0.00% 0.33% 0.32% 0 Check heaps 433 12936 87590 147 0.15% 0.01% 0.18% 1 Virtual Exec 334 93262848 29028603 3212 0.00% 0.14% 0.14% 0 HIDDEN VLAN Proc 200 40238324 301738465 133 0.00% 0.13% 0.13% 0 IP Input 202 2747572 19557286 140 0.00% 0.02% 0.04% 0 ADJ resolve proc 55 31762204 999074 31791 0.00% 0.03% 0.03% 0 Per-minute Jobs 289 25288024 27364507 924 0.00% 0.03% 0.03% 0 IPC LC Message H 486 11390424 313585288 36 0.00% 0.03% 0.01% 0 TDP Hello 25 20877464 86502304 241 0.00% 0.03% 0.01% 0 IPC Seat Manager 104 11541792 46704022 247 0.00% 0.01% 0.01% 0 esw_vlan_stat_pr 313 16750072 85168825 196 0.00% 0.02% 0.00% 0 CEF: IPv4 proces ============================================================================ === I followed the troubleshooting high CPU Utilization Guide but still cannot detect which kind of traffic is punted to CPU. Can anyone in the list show me the way how to detect which packets sending to CPU on 7609 router? Here are the troubleshooting guide that I have checked: http://www.cisco.com/en/US/products/hw/routers/ps368/prod_tech_notes_list.ht ml#anchor4 http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note0918 6a00804916e0.shtml#utilities Thanks, Rin _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
