Hi Bernhard,
On Wed, Sep 30, 2026 at 09:57:56PM +0200, Bernhard Schmidt wrote:
> Package: src:linux
> Version: 6.12.111-1
> Severity: important
>
> After upgrading to todays security kernel a pair of BCM57416 onboard
> ports fail after a couple of minutes uptime with the following error
> message in the log
>
> Sep 30 21:33:27 nm3 kernel: Linux version 6.12.111+deb13-amd64
> ([email protected]) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19)
> 14.2.0,
> [...]
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.0 eth0: Broadcom BCM57416
> NetXtreme-E 10GBase-T Ethernet found at mem 9dc10000, node addr
> 00:0a:f7:e7:f0:5c
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.0: 63.008 Gb/s available PCIe
> bandwidth (8.0 GT/s PCIe x8 link)
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.1 (unnamed net_device)
> (uninitialized): Device requests max timeout of 100 seconds, may trigger hung
> task watchdog (kernel default 120s)
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.1 eth1: Broadcom BCM57416
> NetXtreme-E 10GBase-T Ethernet found at mem 9dc00000, node addr
> 00:0a:f7:e7:f0:5d
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.1: 63.008 Gb/s available PCIe
> bandwidth (8.0 GT/s PCIe x8 link)
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: renamed from eth0
> Sep 30 21:33:27 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: renamed from eth1
> [...]
> Sep 30 21:37:42 nm3 kernel: DMAR: DRHD: handling fault status reg 2
> Sep 30 21:37:42 nm3 kernel: DMAR: [DMA Read NO_PASID] Request device
> [19:00.0] fault addr 0xefffc000 [fault reason 0x06] PTE Read access is not set
> Sep 30 21:37:42 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: Fatal firmware
> reset event, data1: 0x201, data2: 0xda27, min wait 1300 ms, max wait 4200 ms
> Sep 30 21:37:42 nm3 systemd-networkd[1269]: eno2np1: Lost carrier
> Sep 30 21:37:42 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x39f} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:42 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3a0} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:42 nm3 kernel: bond0: (slave eno2np1): link status definitely
> down, disabling slave
> Sep 30 21:37:43 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x23
> seq id 0x3a1 error 0xf
> Sep 30 21:37:43 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb4
> seq id 0x3a2 error 0xf
> Sep 30 21:37:44 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: Device requests max
> timeout of 100 seconds, may trigger hung task watchdog (kernel default 120s)
> Sep 30 21:37:50 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: NIC Link is Down
> Sep 30 21:37:50 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3a3} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:50 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3a4} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:51 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3a5} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:51 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3a6} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:52 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3a7} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:52 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3a8} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:52 nm3 systemd-networkd[1269]: eno2np1: Gained carrier
> Sep 30 21:37:52 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: NIC Link is Up,
> 1000 Mbps full duplex, Flow control: none
> Sep 30 21:37:52 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: EEE is not active
> Sep 30 21:37:52 nm3 kernel: bnxt_en 0000:19:00.1 eno2np1: FEC autoneg off
> encoding: None
> Sep 30 21:37:52 nm3 kernel: bond0: (slave eno2np1): link status up, enabling
> it in 60000 ms
> Sep 30 21:37:53 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3aa} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:54 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3ab} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:54 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3ac} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:55 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3ad} len: 0 due to firmware status: 0x2000001
> Sep 30 21:37:55 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3ae} len: 0 due to firmware status: 0x2000001
> [...]
> Sep 30 21:38:26 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3eb} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:26 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3ec} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:27 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3ed} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:27 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3ee} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:28 nm3 systemd[1607]: oxidized.service: Scheduled restart job,
> restart counter is at 28.
> Sep 30 21:38:28 nm3 systemd[1607]: Started oxidized.service - Oxidized
> Network Configuration Backup.
> Sep 30 21:38:28 nm3 oxidized[10462]: A server is already running. Check
> /home/oxidized/.config/oxidized/pid
> Sep 30 21:38:28 nm3 systemd[1607]: oxidized.service: Main process exited,
> code=exited, status=1/FAILURE
> Sep 30 21:38:28 nm3 systemd[1607]: oxidized.service: Failed with result
> 'exit-code'.
> Sep 30 21:38:28 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3ef} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:28 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3f0} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:29 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x23 0x3f1} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:29 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0xb4 0x3f2} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: NETDEV WATCHDOG:
> CPU: 27: transmit queue 13 timed out 5376 ms
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: TX timeout
> detected, starting reset task!
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [0.0]: tx{fw_ring:
> 0 prod: 76f cons: 75d}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [0]: rx{fw_ring: 1
> prod: 4c29} rx_agg{fw_ring: 15 agg_prod: 80a sw_agg_prod: a}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [0]: cp{fw_ring: 0
> raw_cons: 96e3}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [1.0]: tx{fw_ring:
> 1 prod: 352 cons: 350}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [1]: rx{fw_ring: 2
> prod: 2b5} rx_agg{fw_ring: 16 agg_prod: 801 sw_agg_prod: 1}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [1]: cp{fw_ring: 16
> raw_cons: 294}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [2.0]: tx{fw_ring:
> 2 prod: df cons: df}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [2]: rx{fw_ring: 3
> prod: 225} rx_agg{fw_ring: 17 agg_prod: 805 sw_agg_prod: 5}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [2]: cp{fw_ring: 17
> raw_cons: a4}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [3.0]: tx{fw_ring:
> 3 prod: 105 cons: 105}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [3]: rx{fw_ring: 4
> prod: 252} rx_agg{fw_ring: 18 agg_prod: 805 sw_agg_prod: 5}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [3]: cp{fw_ring: 18
> raw_cons: 120}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [4.0]: tx{fw_ring:
> 4 prod: 249 cons: 235}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [4]: rx{fw_ring: 5
> prod: 208} rx_agg{fw_ring: 19 agg_prod: 800 sw_agg_prod: 0}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [4]: cp{fw_ring: 19
> raw_cons: e6}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [5.0]: tx{fw_ring:
> 5 prod: bb cons: bb}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [5]: rx{fw_ring: 6
> prod: 231} rx_agg{fw_ring: 20 agg_prod: 870 sw_agg_prod: 70}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [5]: cp{fw_ring: 20
> raw_cons: 123}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [6.0]: tx{fw_ring:
> 6 prod: a4b cons: a4b}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [6]: rx{fw_ring: 7
> prod: 211} rx_agg{fw_ring: 21 agg_prod: 801 sw_agg_prod: 1}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [6]: cp{fw_ring: 21
> raw_cons: 39e}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [7.0]: tx{fw_ring:
> 7 prod: 245 cons: 245}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [7]: rx{fw_ring: 8
> prod: 203} rx_agg{fw_ring: 22 agg_prod: 7fd sw_agg_prod: 7fd}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [7]: cp{fw_ring: 22
> raw_cons: d3}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [8.0]: tx{fw_ring:
> 8 prod: 72 cons: 6e}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [8]: rx{fw_ring: 9
> prod: 20c} rx_agg{fw_ring: 23 agg_prod: 800 sw_agg_prod: 0}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [8]: cp{fw_ring: 23
> raw_cons: 46}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [9.0]: tx{fw_ring:
> 9 prod: 8d cons: 8d}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [9]: rx{fw_ring: 10
> prod: 24a} rx_agg{fw_ring: 24 agg_prod: 801 sw_agg_prod: 1}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [9]: cp{fw_ring: 24
> raw_cons: e1}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [10.0]: tx{fw_ring:
> 10 prod: cf cons: cf}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [10]: rx{fw_ring:
> 11 prod: 207} rx_agg{fw_ring: 25 agg_prod: 7ff sw_agg_prod: 7ff}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [10]: cp{fw_ring:
> 25 raw_cons: 5b}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [11.0]: tx{fw_ring:
> 11 prod: 2b cons: 2b}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [11]: rx{fw_ring:
> 12 prod: 206} rx_agg{fw_ring: 26 agg_prod: 7ff sw_agg_prod: 7ff}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [11]: cp{fw_ring:
> 26 raw_cons: 21}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [12.0]: tx{fw_ring:
> 12 prod: db cons: db}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [12]: rx{fw_ring:
> 13 prod: 2a6} rx_agg{fw_ring: 27 agg_prod: 7fd sw_agg_prod: 7fd}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [12]: cp{fw_ring:
> 27 raw_cons: 19a}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [13.0]: tx{fw_ring:
> 13 prod: eb6 cons: eae}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [13]: rx{fw_ring:
> 14 prod: c67} rx_agg{fw_ring: 28 agg_prod: 99c sw_agg_prod: 19c}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: [13]: cp{fw_ring:
> 28 raw_cons: 1b70}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x91
> seq id 0x3f3 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x46 0x3f4} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x71 0x3f5} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x44 0x3f6} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm vnic set tpa
> failure rc for vnic 0: fffffff0
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Discarding out of
> seq response: 0x3f6 for msg {0x41 0x3f7}
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3f7 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3f8 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3f9 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3fa error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3fb error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3fc error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3fd error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3fe error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x3ff error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x400 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x401 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x402 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x403 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x404 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x41
> seq id 0x405 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 1 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 2 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Resp cmpl intr
> abandoning msg: 0x51 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 2 failed. rc:fffffff0 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x430 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x431 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x432 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x433 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x434 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x435 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x436 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x437 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x438 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x439 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x43a error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x43b error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x43c error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x51
> seq id 0x43d error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm_ring_free type
> 0 failed. rc:fffffff3 err:3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x43e error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x43f error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x440 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x441 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x442 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x443 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x444 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x445 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0x61
> seq id 0x446 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb1
> seq id 0x456 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb1
> seq id 0x457 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb1
> seq id 0x458 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb1
> seq id 0x459 error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm req_type 0xb0
> seq id 0x45a error 0x3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: hwrm stat ctx alloc
> failure rc: fffffff3
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: bnxt_init_nic err:
> fffffff3
> Sep 30 21:38:30 nm3 systemd-networkd[1269]: eno1np0: Link DOWN
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: nic open fail (rc:
> fffffff3)
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg
> {0x20 0x45b} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 kernel: bnxt_en 0000:19:00.0 eno1np0: Abandoning msg {0xd
> 0x45c} len: 0 due to firmware status: 0x2000001
> Sep 30 21:38:30 nm3 systemd-networkd[1269]: eno1np0: Lost carrier
> Sep 30 21:38:30 nm3 kernel: bond0: (slave eno1np0): link status definitely
> down, disabling slave
> Sep 30 21:38:30 nm3 kernel: bond0: (slave eno2np1): making interface the new
> active one 37200 ms earlier
>
> It's an active/backup bond, then the second port is down it stays down
> until the box has been rebooted. Downgrading to 6.12.107 fixes the
> problem.
Do you have non-production machine with the same setup which you
couls spare to bisect the changes? If you can that would be the most
helpful step to isolate the breaking change.
Or if you can spare the machine for a couple of cycles then bisecting
the issue would be most efficient forward in the following way:
git clone --single-branch -b linux-6.12.y
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
git checkout v6.12.107
cp /boot/config-$(uname -r) .config
yes '' | make localmodconfig
make savedefconfig
mv defconfig arch/x86/configs/my_defconfig
# test 6.12.107 to ensure this is "good"
make my_defconfig
make -j $(nproc) bindeb-pkg
... install the resulting .deb package and confirm problem does not exist
# test 6.12.111 to ensure this is "bad"
git checkout v6.12.111
make my_defconfig
make -j $(nproc) bindeb-pkg
... install the resulting .deb package and confirm problem exists
With that confirmed, the bisection can start:
git bisect start
git bisect good v6.12.107
git bisect bad v6.12.111
In each bisection step git checks out a state between the oldest
known-bad and the newest known-good commit. In each step test using:
make my_defconfig
make -j $(nproc) bindeb-pkg
... install, if problem exists
and if the problem is hit run:
git bisect bad
and if the problem doesn't trigger run:
git bisect good
. Please pay attention to always select the just built kernel for
booting, it won't always be the default kernel picked up by grub.
Iterate until git announces to have identified the first bad commit.
Then provide the output of
git bisect log
In the course of the bisection you might have to uninstall previous
kernels again to not exhaust the disk space in /boot. Also in the end
uninstall all self-built kernels again.
Regards,
Salvatore
