Hi all, I had an odd event where the fan tray for one of my ASR 9001 routers suddenly disappeared. About 10 minutes later it returned and the alarm cleared. Apart from the obvious of someone having actually removed it, which I'm pretty certain did not happen, are there any known bugs that might cause this? I was going to open a TAC case, but I figured if someone has seen this before and knows this to be a known issue, I'll save the hassle. This device is running IOS XR 4.2.3. Relevant logs of the event below:
RP/0/RSP0/CPU0:Oct 20 07:42:19.759 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:42:19.767 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT RP/0/RSP0/CPU0:Oct 20 07:42:19.772 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:42:22.497 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_CLEAR : Fan tray failure cleared on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:42:22.497 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event. RP/0/RSP0/CPU0:Oct 20 07:42:22.498 : eem_ed_hardware[188]: Messge received content : Event 9 NodeId: 0x4a0 Threshold FANTRAY_FAIL Info Fantray Failure on 0/FT0/SP C Severity 0x64 Frutype 0x20 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:42:22.775 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:42:22.802 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed RP/0/RSP0/CPU0:Oct 20 07:42:22.803 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:42:22.804 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:42:22.804 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:42:22.805 : eem_ed_hardware[188]: Messge received content : Event 7 NodeId: 0x4a0 Threshold FANTRAY_ALARM Info FANTRAY ALARM on 0/FT0/SP Cle Severity 0x8 Frutype 0x20 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:42:22.809 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:42:22.810 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational RP/0/RSP0/CPU0:Oct 20 07:42:22.810 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold All Fantrays Missing Info All Fantrays Missing Severity 0x64 Frutype 0x1 Alarm_raised 1 Alarm_type 1 RP/0/RSP0/CPU0:Oct 20 07:42:31.022 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational RP/0/RSP0/CPU0:Oct 20 07:42:31.022 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold ALL_FANTRAYS_MISSING Info "All Fantrays Missing" Resolv Severity 0x64 Frutype 0x1 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:42:31.459 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis RP/0/RSP0/CPU0:Oct 20 07:42:34.761 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT RP/0/RSP0/CPU0:Oct 20 07:42:37.771 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:42:37.782 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted RP/0/RSP0/CPU0:Oct 20 07:42:37.782 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:43:29.949 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_FAIL : Fan tray failure on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:43:29.949 : eem_ed_hardware[188]: Messge received content : Event 9 NodeId: 0x4a0 Threshold FANTRAY_FAIL Info Fantray 0/FT0/SP Failed Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:30.235 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED RP/0/RSP0/CPU0:Oct 20 07:43:30.265 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 1, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:30.266 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 1 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:30.590 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis RP/0/RSP0/CPU0:Oct 20 07:43:30.590 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM : Fan tray alarm on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:43:30.591 : eem_ed_hardware[188]: Messge received content : Event 7 NodeId: 0x4a0 Threshold FANTRAY_ALARM Info FANTRAY ALARM on 0/FT0/SP Severity 0x8 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:30.950 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 2 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:32.941 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 4, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:32.941 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 4 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:33.942 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 5, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:33.942 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 5 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:36.945 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 8, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:36.945 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 8 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:37.946 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 9, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:37.946 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 9 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:39.948 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 11, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:39.948 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 11 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:43:40.949 : envmon[205]: %PLATFORM-ENVMON-2-FAN_FAIL : Fan failure on slot 0/FT0/SP, fan 12, RPM 0 RP/0/RSP0/CPU0:Oct 20 07:43:40.949 : eem_ed_hardware[188]: Messge received content : Event 6 NodeId: 0x4a0 Threshold FAN_FAIL Info Fan: 12 RPM: 0 Severity 0x64 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:44:31.231 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_CLEARED : Clear alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED RP/0/RSP0/CPU0:Oct 20 07:44:31.231 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_GREEN_FAILED RP/0/RSP0/CPU0:Oct 20 07:50:24.763 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:50:24.769 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT RP/0/RSP0/CPU0:Oct 20 07:50:24.773 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:50:26.850 : envmon[205]: %PLATFORM-ENVMON-2-FANTRAY_CLEAR : Fan tray failure cleared on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:50:26.850 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event. RP/0/RSP0/CPU0:Oct 20 07:50:26.851 : eem_ed_hardware[188]: Messge received content : Event 9 NodeId: 0x4a0 Threshold FANTRAY_FAIL Info Fantray Failure on 0/FT0/SP C Severity 0x64 Frutype 0x20 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:50:27.777 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:50:27.803 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed RP/0/RSP0/CPU0:Oct 20 07:50:27.803 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:50:27.805 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:50:27.805 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:50:27.806 : eem_ed_hardware[188]: Messge received content : Event 7 NodeId: 0x4a0 Threshold FANTRAY_ALARM Info FANTRAY ALARM on 0/FT0/SP Cle Severity 0x8 Frutype 0x20 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:50:27.811 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:50:27.811 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational RP/0/RSP0/CPU0:Oct 20 07:50:27.812 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold All Fantrays Missing Info All Fantrays Missing Severity 0x64 Frutype 0x1 Alarm_raised 1 Alarm_type 1 RP/0/RSP0/CPU0:Oct 20 07:53:05.741 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational RP/0/RSP0/CPU0:Oct 20 07:53:05.741 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 online event. RP/0/RSP0/CPU0:Oct 20 07:53:05.742 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold ALL_FANTRAYS_MISSING Info "All Fantrays Missing" Resolv Severity 0x64 Frutype 0x1 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:53:06.124 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis RP/0/RSP0/CPU0:Oct 20 07:53:06.137 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM : Fan tray alarm on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:53:06.138 : eem_ed_hardware[188]: Messge received content : Event 7 NodeId: 0x4a0 Threshold FANTRAY_ALARM Info FANTRAY ALARM on 0/FT0/SP Severity 0x8 Frutype 0x20 Alarm_raised 1 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:53:06.140 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:53:06.141 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis RP/0/RSP0/CPU0:Oct 20 07:53:09.766 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT RP/0/RSP0/CPU0:Oct 20 07:53:12.776 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:53:12.786 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted RP/0/RSP0/CPU0:Oct 20 07:53:12.787 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:53:34.764 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:53:34.770 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:NOT-PRESENT RP/0/RSP0/CPU0:Oct 20 07:53:34.773 : invmgr[254]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/FT0/SP, state: BRINGDOWN RP/0/RSP0/CPU0:Oct 20 07:53:37.779 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:53:37.800 : invmgr[254]: %PLATFORM-INV-6-OIROUT : OIR: Node 0/FT0/SP removed RP/0/RSP0/CPU0:Oct 20 07:53:37.801 : eem_ed_oir[190]: Messge received content : Event 1 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:53:39.016 : envmon[205]: %PLATFORM-ENVMON-3-FANTRAY_ALARM_CLEAR : Fan tray alarm cleared on slot 0/FT0/SP RP/0/RSP0/CPU0:Oct 20 07:53:39.016 : shelfmgr[394]: %PLATFORM-SHELFMGR_HAL-7-CAN_INFO : Can API: slot 10 offline event. RP/0/RSP0/CPU0:Oct 20 07:53:39.017 : eem_ed_hardware[188]: Messge received content : Event 7 NodeId: 0x4a0 Threshold FANTRAY_ALARM Info FANTRAY ALARM on 0/FT0/SP Cle Severity 0x8 Frutype 0x20 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : eem_ed_hardware[188]: get_envmon_card_data : Bag found NULL, returning RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis RP/0/RSP0/CPU0:Oct 20 07:53:39.020 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING : No fan trays present in the chassis, chassis cooling non-operational RP/0/RSP0/CPU0:Oct 20 07:53:39.021 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold All Fantrays Missing Info All Fantrays Missing Severity 0x64 Frutype 0x1 Alarm_raised 1 Alarm_type 1 RP/0/RSP0/CPU0:Oct 20 07:53:39.457 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition exists in the chassis RP/0/RSP0/CPU0:Oct 20 07:53:42.239 : envmon[205]: %PLATFORM-ENVMON-2-ALL_FANTRAYS_MISSING_CLEAR : One or more fan trays now plugged in, chassis cooling operational RP/0/RSP0/CPU0:Oct 20 07:53:42.239 : eem_ed_hardware[188]: Messge received content : Event 1 NodeId: 0x1 Threshold ALL_FANTRAYS_MISSING Info "All Fantrays Missing" Resolv Severity 0x64 Frutype 0x1 Alarm_raised 0 Alarm_type 2 RP/0/RSP0/CPU0:Oct 20 07:53:42.685 : envmon[205]: %PLATFORM-ENVMON-2-ENV_CONDITION : Outstanding environmental condition cleared in the chassis RP/0/RSP0/CPU0:Oct 20 07:53:44.766 : shelfmgr[394]: %PLATFORM-SHELFMGR-6-NODE_STATE_CHANGE : 0/FT0/SP FAN TRAY state:PRESENT RP/0/RSP0/CPU0:Oct 20 07:53:47.779 : invmgr[254]: arb_lib_asic_get_ver: msg send problem: Invalid argument RP/0/RSP0/CPU0:Oct 20 07:53:47.789 : invmgr[254]: %PLATFORM-INV-6-OIRIN : OIR: Node 0/FT0/SP inserted RP/0/RSP0/CPU0:Oct 20 07:53:47.789 : eem_ed_oir[190]: Messge received content : Event 0 NodeId: 0x4a0 RP/0/RSP0/CPU0:Oct 20 07:54:41.444 : canb-server[150]: %PLATFORM-CANB_SERVER-3-ALARM_INDICATION : Raise alarm from CBC in slot 0/FT0/SP, alarm code CBC_ALRM_FT_LED_RED_FAILED Thanks! -Vinny _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
