I was trouble shooing my network today and found a nasty little bug when some one does 'show isis database' from exec mode on C181X Software (C181X-ADVIPSERVICESK9-M), Version 15.0(1)M1, IOS.
After issuing the command you get the output of it, and some time in the next 30 sec the router crashes. example: LTRKAKHQR01-c1811w#sh isis database IS-IS Level-1 Link State Database: LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL STPBFLGURT1.00-00 0x00005F0A 0x49A3 907 0/0/0 galaxydoor.00-00 0x0000200A 0x2DF0 900 0/0/0 LTRKAKHQR01-c1.00-00* 0x00000953 0x64C1 1099 0/0/0 TAMQFLTART1.00-00 0x00005859 0x1542 908 0/0/0 IS-IS Level-2 Link State Database: LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL STPBFLGURT1.00-00 0x000060A8 0x1149 914 0/0/0 galaxydoor.00-00 0x0000200F 0x645F 912 0/0/0 LTRKAKHQR01-c1.00-00* 0x00000991 0xF41F 916 0/0/0 TAMQFLTART1.00-00 0x00005926 0x83FD 913 0/0/0 LTRKAKHQR01-c1811w#term mon LTRKAKHQR01-c1811w#sh clock 01:44:47.438 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:44:56.418 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:45:01.690 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:45:06.182 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:45:10.146 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:45:12.658 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock 01:45:16.222 UTC Sun Feb 7 2010 LTRKAKHQR01-c1811w#sh clock ______BAM! Lockup at this point______ from the log output: Feb 6 20:45:27 192.168.3.210 103: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:20 UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (0/0),process = Check heaps. Feb 6 20:45:27 192.168.3.210 104: LTRKAKHQR01-c1811w: -Traceback= 0x8007CCB0z 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 20:45:27 192.168.3.210 105: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:22 UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more than (2000)msecs (0/0),process = Check heaps. Feb 6 20:45:27 192.168.3.210 106: LTRKAKHQR01-c1811w: -Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z 0x82050E54z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 20:45:27 192.168.3.210 107: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-3-BADMAGIC: Corrupt block at 86AC28DC (magic 813E0508), -Traceback= 0x82052388z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 20:45:27 192.168.3.210 108: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-MTRACE: mallocfree: addr, pc Feb 6 20:45:27 192.168.3.210 109: LTRKAKHQR01-c1811w: 86297A44,80BAE58C 86297A44,40000294 86CC08A0,80BAE570 86CC08A0,3000021E Feb 6 20:45:27 192.168.3.210 110: LTRKAKHQR01-c1811w: 86DC8180,8154FFC0 866536C8,8154FE24 866536C8,8154FE24 866536C8,8154FE88 Feb 6 20:45:27 192.168.3.210 111: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-MTRACE: mallocfree: addr, pc Feb 6 20:45:27 192.168.3.210 112: LTRKAKHQR01-c1811w: 8666F860,81569A98 866536C8,8154FE88 866536C8,8154FE24 866536C8,8154FE24 Feb 6 20:45:27 192.168.3.210 113: LTRKAKHQR01-c1811w: 866536C8,8154EE70 866536C8,8154EE70 866536C8,8154EE70 866536C8,8154EE70 Feb 6 20:45:27 192.168.3.210 114: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk 86AC28DC, words 6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt 1, -Traceback= 0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 20:45:28 192.168.3.210 115: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-MEMDUMP: 0x86AC28DC: 0x813E0508 0x0 0x0 0x8364CCBC Feb 6 20:45:28 192.168.3.210 116: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-MEMDUMP: 0x86AC28EC: 0x8012DAC4 0x86AC57F0 0x86AB9C1C 0x80001772 Feb 6 20:45:28 192.168.3.210 117: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-MEMDUMP: 0x86AC28FC: 0x1 0x86AC2980 0x15C 0x86D41800 Feb 6 20:45:28 192.168.3.210 118: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24 UTC: %SYS-6-STACKLOW: Stack for process Virtual Exec running low, 12/12000 Feb 6 20:45:40 192.168.3.250 1038: TAMQFLTART1: Feb 7 2010 01:45:39 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down ------- Some times I'll get a crashinfo file, other times I will not. >From a previous crash info: ------------- CMD: 'sh isis database' 21:23:27 UTC Sat Feb 6 2010 validblock_diagnose, code = 2 current memory block, bp = 0x8700E0B0, memorypool type is Processor data check, ptr = 0x8700E0E0 next memory block, bp = 0x87010FC4, memorypool type is Processor data check, ptr = 0x87010FF4 previous memory block, bp = 0x870053DC, memorypool type is Processor data check, ptr = 0x8700540C ========= Dump bp = 0x8700E0B0 ====================== 8700DFB0: 0 8700EAB0 FFFFFFFF 0 0 0 0 0 8700DFD0: 0 0 6347E519 0 8207070C D02688F2 6347E519 85F7C994 8700DFF0: 85F7C994 811AEC4C 8700E010 811AB16C D0D0D0D 245EBB78 D0D0D0D 867CD2F4 8700E010: 8700E040 813B2838 D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 82070710 8700E030: 85F7C994 6347E519 875E3258 875E3214 8700E070 813B2B8C D0D0D0D 48822022 8700E050: 1 8700E2D8 0 0 8700E070 6347E519 8700E400 0 8700E070: 8700E0B0 813B4470 0 0 28822022 6347E519 0 0 8700E090: 0 0 6347E519 85F7C994 0 0 8700E400 0 8700E0B0: 8700E350 813E0508 0 0 8012DAC4 87010FC4 870053F0 80001772 8700E0D0: 1 0 8700E158 872550DC FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 8700E0F0: 0 0 FFFFFFFF FFFFFFFF FFFF FFFFFFFF FFFFFFFF FFFFFFFF 8700E110: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 0 8700E130: 2F 86427028 0 85F7C994 0 6347E519 6347E519 245EBB78 8700E150: 8700E2F0 867CD2F4 8700E1D8 811ABDE0 FFFFFFFF 6347E519 D02688F2 2F 8700E170: 0 0 0 C0 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 8700E190: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF ========= Feb 6 2010 21:24:09 UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs (1/1),process = Check heaps. -Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Dump bp->next = 0x87010FC4 ====================== 87010EC4: 61780000 87010EF0 84228082 73796E74 A4CB80 7002FD0 87010F20 87010E70 87010EE4: 87010EF8 83EB0000 83EB0000 0 83EB0000 0 A4CB80 0 87010F04: 0 867F2054 0 0 86493648 87010FB0 80B77310 FFFFFF 87010F24: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 87010F44: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 2A 1 FFFFFFFF 87010F64: FFFFFFFF 0 0 0 0 0 0 0 87010F84: 0 0 0 0 0 0 0 0 87010FA4: 0 0 0 87010FB8 8012086C 0 80124418 FD0110DF 87010FC4: AB1234CD E40000 15F 873074D0 80B4FCA0 87015E18 8700E0C4 80002712 87010FE4: 1 8200EA4C 166 872550DC 0 0 87307494 0 87011004: 87307494 258 2C7 140018 2C1 0 0 0 87011024: 0 430000 83EC2BBC 41414120 536D616C 6C204368 756E0000 87011B6C 87011044: 87015E14 0 0 87011B70 87011B88 87011BA0 87011BB8 87011BD0 87011064: 87011BE8 87011C00 87011C18 87011C30 87011C48 87011C60 87011C78 87011C90 87011084: 87011CA8 87011CC0 87011CD8 87011CF0 87011D08 87011D20 87011D38 87011D50 870110A4: 87011D68 87011D80 87011D98 87011DB0 87011DC8 87011DE0 87011DF8 87011E10 ========== Dump bp->previous = 0x870053F0 ===================== 870052F0: 0 0 0 0 0 0 0 0 87005310: 0 FD0110DF AB1234CD FFFE0000 0 82FC74AC 81BDE144 87005390 87005330: 870052A8 80000024 1 0 1 850B5B2C 83824BA0 0 87005350: 0 0 1 0 Feb 6 2010 21:24:11 UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more than (2000)msecs (1/1),process = Check heaps. -Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z 0x82050E54z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z 0 0 0 90000 87005370: 1 870051FC 0 0 0 0 0 FD0110DF 87005390: AB1234CD FFFE0000 0 82FC74AC 81BD9178 870053DC 8700532C 8000000E 870053B0: 1 0 1 850B5B2C 1 0 0 0 870053D0: 0 0 FD0110DF AB1234CD 750000 75 83646E94 82C4EED4 870053F0: 8700E0B0 870053A4 4652 0 82C89068 7C 850B1410 DEADBEEF 87005410: 82C89068 0 D0D0D0D 83EC321C 83EC3218 D0D0D0D D0D0D0D D0D0D0D 87005430: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 87005450: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 87005470: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 87005490: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 870054B0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 870054D0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D ============================================ Feb 6 2010 21:24:12 UTC: %SYS-3-BADMAGIC: Corrupt block at 8700E0B0 (magic 8700E350), -Traceback= 0x82052388z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc 873068BC,80BAE58C 873068BC,40000294 859EC9B4,80BAE570 859EC9B4,3000021E 86EAC770,8154FFC0 86EA8998,8154FE24 86EA8998,8154FE24 859EC9B4,81540E3C Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc 859EC9B4,8153B354 859EC9B4,3000021E 86EA8998,8154FE88 86EAB098,81569A98 86EA8998,8154FE88 86EA8998,8154FE24 86EA8998,8154FE24 86EA8998,8154EE70 Feb 6 2010 21:24:12 UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk 8700E0B0, words 6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt 1, -Traceback= 0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0B0: 0x8700E350 0x813E0508 0x0 0x0 Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0C0: 0x8012DAC4 0x87010FC4 0x870053F0 0x80001772 Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0D0: 0x1 0x0 0x8700E158 0x872550DC %Software-forced reload 21:24:12 UTC Sat Feb 6 2010: Unexpected exception to CPU: vector 1500, PC = 0x8011E220, LR = 0x8011E1E4 -Traceback= 0x8011E220z 0x8011E1E4z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z CPU Register Context: MSR = 0x02029220 CR = 0x28000042 CTR = 0x81F26400 XER = 0x00000000 R0 = 0x8011E1E4 R1 = 0x8511CBA8 R2 = 0xFFE97C10 R3 = 0x83FA9978 R4 = 0x82F869BC R5 = 0x00000000 R6 = 0x83970000 R7 = 0x82F60000 R8 = 0x02029220 R9 = 0x83AD0000 R10 = 0x00000000 R11 = 0x00000000 R12 = 0x24000088 R13 = 0xFFE994A8 R14 = 0x820554DC R15 = 0x00000000 R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000 R20 = 0x00000000 R21 = 0x00000000 R22 = 0x83D60000 R23 = 0x83D60000 R24 = 0xAB1234AB R25 = 0xAB1234CD R26 = 0x83D60000 R27 = 0x85FBD91C R28 = 0x00000000 R29 = 0x83647534 R30 = 0x83980000 R31 = 0x00000000 ------ I've tried this on both 1811w's on my network and had the exact same problems. Any one else seen this or know if it's a known bug? I've searched the cisco site and cannot find a reference to this issue. -- Bryan Fields 727-409-1194 - Voice 727-214-2508 - Fax http://bryanfields.net _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
