Hello,
We had the ixgbe radios in one of our systems on an overnight test. To my
knowledge,
we have never seen this particular issue before. Please let me know if you
have any
ideas on what caused it or how we can get better logs to debug it. We plan to
replace
the NIC and re-run in case it is hardware issue.
The logs below are filtered on 'ixgbe', but I can provide full logs if that
would help.
This is from 6.11.11 + local patches kernel, but not many changes from stock
kernel in the Ethernet
stack or driver.
root@ct523c-6987:~# grep ixgbe log.txt
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe: Intel(R) 10 Gigabit PCI Express
Network Driver
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe: Copyright (c) 1999-2016 Intel
Corporation.
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: Multiqueue Enabled: Rx
Queue count = 20, Tx Queue count = 20 XDP Queue count = 0
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: 31.504 Gb/s available
PCIe bandwidth (8.0 GT/s PCIe x4 link)
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: MAC: 4, PHY: 0, PBA No:
H86377-005
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: 3c:fd:fe:e1:c6:c6
Mar 07 17:34:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: Intel(R) 10 Gigabit
Network Connection
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1: Multiqueue Enabled: Rx
Queue count = 20, Tx Queue count = 20 XDP Queue count = 0
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1: 31.504 Gb/s available
PCIe bandwidth (8.0 GT/s PCIe x4 link)
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1: MAC: 4, PHY: 0, PBA No:
H86377-005
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1: 3c:fd:fe:e1:c6:c7
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1: Intel(R) 10 Gigabit
Network Connection
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.0 enp21s0f0: renamed from
eth2
Mar 07 17:34:49 ct523c-6987 kernel: ixgbe 0000:15:00.1 enp21s0f1: renamed from
eth3
Mar 07 17:34:53 ct523c-6987 kernel: ixgbe 0000:15:00.1 eth3: renamed from
enp21s0f1
Mar 07 17:34:53 ct523c-6987 kernel: ixgbe 0000:15:00.0 eth2: renamed from
enp21s0f0
Mar 07 17:49:27 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 17:49:27 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 07 17:49:27 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 17:49:28 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 07 17:49:28 ct523c-6987 kernel: ixgbe 0000:15:00.1: registered PHC device
on eth3
Mar 07 17:49:28 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 17:49:28 ct523c-6987 kernel: ixgbe 0000:15:00.1: removed PHC on eth3
Mar 07 17:49:29 ct523c-6987 kernel: ixgbe 0000:15:00.1: registered PHC device
on eth3
Mar 07 17:49:29 ct523c-6987 kernel: ixgbe 0000:15:00.1: removed PHC on eth3
Mar 07 17:49:29 ct523c-6987 kernel: ixgbe 0000:15:00.1: registered PHC device
on eth3
Mar 07 17:49:34 ct523c-6987 kernel: ixgbe 0000:15:00.0 eth2: NIC Link is Up 10
Gbps, Flow Control: RX/TX
Mar 07 17:49:34 ct523c-6987 kernel: ixgbe 0000:15:00.1 eth3: NIC Link is Up 10
Gbps, Flow Control: RX/TX
Mar 07 17:58:24 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 07 17:58:25 ct523c-6987 kernel: ixgbe 0000:15:00.1: removed PHC on eth3
Mar 07 17:58:48 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 18:00:42 ct523c-6987 kernel: ixgbe 0000:15:00.0 eth2: NIC Link is Up 10
Gbps, Flow Control: RX/TX
Mar 07 18:47:08 ct523c-6987 kernel: nfs_acl lockd grace sch_fq_codel sunrpc fuse zram raid1 dm_raid raid456 libcrc32c async_raid6_recov async_memcpy async_pq
async_xor xor async_tx raid6_pq xe drm_ttm_helper gpu_sched drm_suballoc_helper drm_gpuvm drm_exec i915 i2c_algo_bit cec rc_core drm_buddy intel_gtt
drm_display_helper drm_kms_helper ttm agpgart e1000e igc ixgbe mdio dca hwmon drm xhci_pci mei_wdt i2c_core xhci_pci_renesas video wmi pinctrl_alderlake
efivarfs [last unloaded: nfnetlink]
Mar 07 18:58:35 ct523c-6987 kernel: ixgbe 0000:15:00.0 eth2: NIC Link is Down
Mar 07 18:58:56 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 07 18:58:56 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 18:58:57 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 07 18:58:57 ct523c-6987 kernel: ixgbe 0000:15:00.0: registered PHC device
on eth2
Mar 07 18:59:03 ct523c-6987 kernel: ixgbe 0000:15:00.1: registered PHC device
on eth3
Mar 07 18:59:03 ct523c-6987 kernel: ixgbe 0000:15:00.1: removed PHC on eth3
Mar 07 18:59:04 ct523c-6987 kernel: ixgbe 0000:15:00.1: registered PHC device
on eth3
Mar 07 18:59:08 ct523c-6987 kernel: ixgbe 0000:15:00.0 eth2: NIC Link is Up 10
Gbps, Flow Control: RX/TX
Mar 07 18:59:08 ct523c-6987 kernel: ixgbe 0000:15:00.1 eth3: NIC Link is Up 10
Gbps, Flow Control: RX/TX
Mar 09 06:08:19 ct523c-6987 kernel: ixgbe 0000:15:00.0: Adapter removed
Mar 09 06:08:19 ct523c-6987 kernel: ixgbe 0000:15:00.0: Warning firmware error
detected FWSM: 0xFFFFFFFF
Mar 09 06:08:19 ct523c-6987 kernel: ixgbe 0000:15:00.0: Firmware recovery mode detected. Limiting functionality. Refer to the Intel(R) Ethernet Adapters and
Devices User Guide for details on firmware recovery mode.
Mar 09 06:08:19 ct523c-6987 kernel: ixgbe 0000:15:00.0: removed PHC on eth2
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe-mdio-0000:15:00.0: not in
UNREGISTERED state
Mar 09 06:08:21 ct523c-6987 kernel: nfs_acl lockd grace sch_fq_codel sunrpc fuse zram raid1 dm_raid raid456 libcrc32c async_raid6_recov async_memcpy async_pq
async_xor xor async_tx raid6_pq xe drm_ttm_helper gpu_sched drm_suballoc_helper drm_gpuvm drm_exec i915 i2c_algo_bit cec rc_core drm_buddy intel_gtt
drm_display_helper drm_kms_helper ttm agpgart e1000e igc ixgbe mdio dca hwmon drm xhci_pci mei_wdt i2c_core xhci_pci_renesas video wmi pinctrl_alderlake
efivarfs [last unloaded: nfnetlink]
Mar 09 06:08:21 ct523c-6987 kernel: Workqueue: ixgbe ixgbe_service_task [ixgbe]
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe_service_task+0xb9e/0x12f0 [ixgbe]
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe 0000:15:00.1: Adapter removed
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe 0000:15:00.1: Warning firmware error
detected FWSM: 0xFFFFFFFF
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe 0000:15:00.1: Firmware recovery mode detected. Limiting functionality. Refer to the Intel(R) Ethernet Adapters and
Devices User Guide for details on firmware recovery mode.
Mar 09 06:08:21 ct523c-6987 kernel: ixgbe 0000:15:00.1: removed PHC on eth3
Mar 09 06:08:22 ct523c-6987 kernel: ixgbe-mdio-0000:15:00.1: not in
UNREGISTERED state
Mar 09 06:08:22 ct523c-6987 kernel: nfs_acl lockd grace sch_fq_codel sunrpc fuse zram raid1 dm_raid raid456 libcrc32c async_raid6_recov async_memcpy async_pq
async_xor xor async_tx raid6_pq xe drm_ttm_helper gpu_sched drm_suballoc_helper drm_gpuvm drm_exec i915 i2c_algo_bit cec rc_core drm_buddy intel_gtt
drm_display_helper drm_kms_helper ttm agpgart e1000e igc ixgbe mdio dca hwmon drm xhci_pci mei_wdt i2c_core xhci_pci_renesas video wmi pinctrl_alderlake
efivarfs [last unloaded: nfnetlink]
Mar 09 06:08:22 ct523c-6987 kernel: Workqueue: ixgbe ixgbe_service_task [ixgbe]
Mar 09 06:08:22 ct523c-6987 kernel: ixgbe_service_task+0xb9e/0x12f0 [ixgbe]
root@ct523c-6987:~# uname -a
Linux ct523c-6987 6.11.11+ #39 SMP PREEMPT_DYNAMIC Fri Feb 28 15:53:45 PST 2025
x86_64 GNU/Linux
root@ct523c-6987:~# ifconfig eth2
eth2: error fetching interface information: Device not found
root@ct523c-6987:~# ifconfig eth3
eth3: error fetching interface information: Device not found
Thanks,
Ben
--
Ben Greear <[email protected]>
Candela Technologies Inc http://www.candelatech.com
