This message stands out: > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 error detection reg2: ECC > multibit
> -----Original Message----- > From: [email protected] [mailto:juniper-nsp- > [email protected]] On Behalf Of Nilesh Khambal > Sent: Monday, August 17, 2009 10:57 AM > To: Clue Store > Cc: [email protected] > Subject: Re: [j-nsp] m10i Nastiness Friday night > > It looks like CFEB dumped core and restarted. Please open a JTAC case > and let me them figure out what went wrong with CFEB. Please gather all > logs around the time of the problem. Usually following logs should be a > good start. > > - show log messages[.(0-9).gz] (From RE) > - show syslog messages (from CFEB) > - show nvram (from CFEB). > - CFEB coredump file generated under "/var/tmp" > - Any other surrounding information such temperature, memory, CPU > information about RE and CFEB around the time of the problem. > > Given the old version of code you are running on the box, this may be a > known issue fixed in later release such as 8.5 which you are running on > the other box. Let JTAC analyze that. > > Thanks, > Nilesh. > > Clue Store wrote: > > Hi All, > > > > Last friday we had some nastiness on one of our m10i's. As I am not a > > Juniper expert, I was wondering if someone could decipher the log > messages > > and determine if is possibly a CFEB issue, or just a fluke Junos issue > and > > whether I should do anything or let it be and see if it does it again. I > > have another m10i running 8.5, so I am thinking of just upgrading this > box > > to the same as my other, but i'd like to hear what some of you on the > list > > think. > > > > TIA, > > Clue > > > > Hostname: JuniperM10i-HMNDLAMA > > Model: m10i > > JUNOS Base OS boot [8.0R2.8] > > JUNOS Base OS Software Suite [8.0R2.8] > > JUNOS Kernel Software Suite [8.0R2.8] > > JUNOS Packet Forwarding Engine Support (M7i/M10i) [8.0R2.8] > > JUNOS Routing Software Suite [8.0R2.8] > > JUNOS Online Documentation [8.0R2.8] > > > > > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 machine check caused > by > > error on the Processor Bus > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 PCI status register: > > 0x0020, error detect register 1: 0x00, 2: 0x08 > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 error ack count = 0 > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 error address: > 0x0f3827f8 > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 Processor bus error > status > > register: 0x52 > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb transfer type 0b01010, > transfer > > size 2 > > Aug 14 23:38:51 JuniperM10i-HMNDLAMA cfeb mpc106 error detection reg2: > ECC > > multibit > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb ^B > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb last message repeated 6 times > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Context: Interrupt Level (0) > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Registers: > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R00: 0x00000446 R01: > 0x00799450 > > R02: 0x00000000 R03: 0x4f3827fc > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R04: 0x00000552 R05: > 0x00000000 > > R06: 0x007994a0 R07: 0x00000004 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R08: 0x00000548 R09: > 0x0017f48b > > R10: 0x00000002 R11: 0xb0c7d8ec > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R12: 0x28002044 R13: > 0x02420020 > > R14: 0xf1ae2100 R15: 0x82600020 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R16: 0x442104c2 R17: > 0x2248000b > > R18: 0x00670000 R19: 0x00670000 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R20: 0x00670000 R21: > 0x006ce5a0 > > R22: 0x007902d0 R23: 0x00670000 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R24: 0x00000002 R25: > 0x00000004 > > R26: 0x0080bd40 R27: 0x0000ffff > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb R28: 0x00000001 R29: > 0x00000001 > > R30: 0x4f38271c R31: 0x4f382714 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb MSR: 0x00089030 CTR: > 0x00000239 > > Link:0x002e34c8 SP: 0x00799450 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb CCR: 0x48002028 XER: > 0x20000000 > > PC: 0x00460320 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb DSISR: 0x00000000 DAR: > 0x00000000 > > K_MSR: 0x00000030 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Stack Traceback: > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 01: sp = 0x00799450, pc > = > > 0x0000c001 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 02: sp = 0x00799468, pc > = > > 0x002e4d74 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 03: sp = 0x00799498, pc > = > > 0x002e35e0 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 04: sp = 0x007994b8, pc > = > > 0x002e3bb0 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 05: sp = 0x007994c0, pc > = > > 0x00058818 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 06: sp = 0x007994d8, pc > = > > 0x0003df34 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 07: sp = 0x00799500, pc > = > > 0x003b4488 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 08: sp = 0x00799530, pc > = > > 0x003b4660 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 09: sp = 0x00799548, pc > = > > 0x003b3ed0 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 10: sp = 0x007995c8, pc > = > > 0x003b3d30 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 11: sp = 0x007995e8, pc > = > > 0x000b9f6c > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 12: sp = 0x00799610, pc > = > > 0x000b8928 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 13: sp = 0x00799628, pc > = > > 0x00448518 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 14: sp = 0x00799678, pc > = > > 0x00442d00 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 15: sp = 0x00799698, pc > = > > 0x0003a500 > > Aug 14 23:38:52 JuniperM10i-HMNDLAMA cfeb Frame 16: sp = 0x007996b0, pc > = > > 0x0003b268 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA /kernel: rdp keepalive expired, > > connection dropped - src 1:1021 dest 2:15360 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: Major alarm set, > CFEB > > not online, the box is not forwarding > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: CFEB > > color=RED, class=CHASSIS, reason=CFEB not online, the box is not > forwarding > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 43 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_SHUTDOWN_NOTICE: Shutdown reason: CFEB connection lost > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(0) > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA mib2d[3111]: SNMP_TRAP_LINK_DOWN: > > ifIndex 77, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-0/0/0 > > > > (Lots of BGP notifications due to interface down issues) > > > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA snmpd[3132]: SNMPD_SEND_FAILURE: > > trap_io_send_trap_now: send to (207.29.223.55) failure: Network is down > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: shutting down > chassisd > > connection: chassisd ipc pipe read error > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: > > craftd_user_conn_shutdown: socket 5, errno = 0 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: chassisd connection > > succeeded after 0 retries > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: chassisd connection > > succeeded after 0 retries > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA mib2d[3111]: SNMP_TRAP_LINK_DOWN: > > ifIndex 80, ifAdminStatus down(2), ifOperStatus down(2), ifName ge- > 1/0/0.462 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: resending alarm > state > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 43 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: CFEB > > color=RED, class=CHASSIS, reason=CFEB not online, the box is not > forwarding > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE > color=RED, > > class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 43 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE > color=RED, > > class=CHASSIS, reason=Host 1 fxp0: Ethernet Link Down > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 43 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA /kernel: rdp keepalive expired, > > connection dropped - src 1:1020 dest 2:15361 > > Aug 14 23:38:56 JuniperM10i-HMNDLAMA /kernel: pfe_listener_disconnect: > conn > > dropped: listener idx=0, tnpaddr=0x2, reason: socket error > > Aug 14 23:39:41 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed > > [system warm] > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_SNMP_TRAP10: > > SNMP trap generated: FRU power on (jnxFruContentsIndex 6, jnxFruL1Index > 1, > > jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName CFEB 0, jnxFruType 4, > > jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, > jnxFruLastPowerOn > > 0) > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(0) > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_DETACH_FPC: ifdev_detach(1) > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_DETACH_ALL_PSEUDO: ifdev_detach(pseudo devices: all) > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA craftd[2999]: Major alarm cleared, > > Host 0 fxp0: Ethernet Link Down > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm cleared: RE > > color=RED, class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 44 > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot > 0: > > CFEB not online, the box is not forwarding > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot > 0: > > Host 0 fxp0: Ethernet Link Down > > Aug 14 23:40:09 JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot > 1: > > Host 1 fxp0: Ethernet Link Down > > Aug 14 23:40:10 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_FRU_EVENT: > > fpc_m40_recv_restart: restarted FPC 0 > > Aug 14 23:40:10 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_FRU_EVENT: > > fpc_m40_recv_restart: restarted FPC 1 > > Aug 14 23:40:12 JuniperM10i-HMNDLAMA craftd[2999]: Major alarm set, > Host 0 > > fxp0: Ethernet Link Down > > Aug 14 23:40:12 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm set: RE > color=RED, > > class=CHASSIS, reason=Host 0 fxp0: Ethernet Link Down > > Aug 14 23:40:12 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 43 > > Aug 14 23:40:12 JuniperM10i-HMNDLAMA cfeb CM: ALARM CLEAR: Slot 0: Host > 0 > > fxp0: Ethernet Link Down > > Aug 14 23:40:17 JuniperM10i-HMNDLAMA craftd[2999]: Major alarm cleared, > > CFEB not online, the box is not forwarding > > Aug 14 23:40:17 JuniperM10i-HMNDLAMA alarmd[2998]: Alarm cleared: CFEB > > color=RED, class=CHASSIS, reason=CFEB not online, the box is not > forwarding > > Aug 14 23:40:17 JuniperM10i-HMNDLAMA craftd[2999]: forwarding display > > request to chassisd: type = 4, subtype = 44 > > Aug 14 23:40:17 JuniperM10i-HMNDLAMA cfeb CM: ALARM SET: (Major) Slot > 0: > > Host 0 fxp0: Ethernet Link Down > > Aug 14 23:40:32 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal > speed > > Aug 14 23:40:33 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_FRU_EVENT: > > scb_recv_slot_attach: attached FPC 0 > > Aug 14 23:40:55 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_FRU_EVENT: > > scb_recv_slot_attach: attached FPC 1 > > Aug 14 23:40:57 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_SNMP_TRAP10: > > SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index > 1, > > jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @ > > 0/0/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2, > > jnxFruLastPowerOff 0, jnxFruLastPowerOn 0) > > Aug 14 23:40:57 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_SNMP_TRAP10: > > SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index > 2, > > jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC: 1x G/E, 1000 BASE-SX @ > > 1/0/*, jnxFruType 11, jnxFruSlot 2, jnxFruOfflineReason 2, > > jnxFruLastPowerOff 0, jnxFruLastPowerOn 0) > > Aug 14 23:40:57 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for > > ge-0/0/0 > > Aug 14 23:40:58 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for > > ge-1/0/0 > > Aug 14 23:40:58 JuniperM10i-HMNDLAMA chassisd[2997]: > CHASSISD_SNMP_TRAP10: > > SNMP trap generated: FRU power on (jnxFruContentsIndex 7, jnxFruL1Index > 1, > > jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: @ 0/*/*, jnxFruType > 3, > > jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, > jnxFruLastPowerOn > > 0) > > > > (BGP notifications that peers are responding) > > > > > > Aug 14 23:42:22 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed > > [system warm] > > Aug 14 23:43:22 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal > speed > > Aug 14 23:44:02 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED_FULL: Fans and impellers being set to full speed > > [system warm] > > Aug 14 23:44:37 JuniperM10i-HMNDLAMA chassisd[2997]: > > CHASSISD_BLOWERS_SPEED: Fans and impellers are now running at normal > speed > > _______________________________________________ > > juniper-nsp mailing list [email protected] > > https://puck.nether.net/mailman/listinfo/juniper-nsp > > . > > > > > _______________________________________________ > juniper-nsp mailing list [email protected] > https://puck.nether.net/mailman/listinfo/juniper-nsp _______________________________________________ juniper-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/juniper-nsp
