Public bug reported:
Since moving from 6.17.0-1022-azure to 7.0.0-1017-azure, the MANA VF on our
production VM keeps
locking up. While it is locked up, all traffic stops for about 3-6 minutes,
then the driver resets
itself. We saw this on three different physical hosts, and on the third host it
happened during
the first boot. That makes a host hardware fault unlikely.
Environment
- Ubuntu 24.04.4 LTS, Azure, East US 2 (zone 1), Standard_D4s_v7, Trusted
Launch (Secure Boot on)
- Kernel: Ubuntu 7.0.0-1017.17~24.04.1-azure 7.0.14 (linux-azure-7.0, HWE)
- NIC: 7870:00:00.0 Ethernet controller [0200]: Microsoft Corporation Device
[1414:00ba] (MANA),
Accelerated Networking enabled, single NIC, netplan matches eth0 by driver
hv_netvsc
- CONFIG_MICROSOFT_MANA=y, CONFIG_MANA_INFINIBAND=m (mana_ib is loaded but
unused, no RDMA workload)
- Workload: game backend in Docker, about 120 MB/min outbound, mostly small
WebSocket packets
Timeline (UTC)
- Until 2026-10-20 the VM ran 6.17.0-1022-azure. Our kern.log history goes back
to 2026-09-20,
which is a month on 6.17, and it has no "HWC: Request timed out" and no
"NETDEV WATCHDOG" lines.
The only VF event in that time was an err -71 removal during host maintenance
on 10-20 17:12,
and failover worked.
- 2026-10-20 17:22: the platform rebooted the VM and it came up on
7.0.0-1017-azure.
- Host A: lockups on 10-20 17:55 and on 10-21 06:32 and 07:05.
- 2026-10-21 07:10: deallocate/start, which moved the VM to host B.
- Host B: lockups on 10-21 19:44, then on 10-22 at 14:53, 15:24, 15:45, 15:50
and 16:23. The gaps
got shorter each time. Resource Health showed "Degraded" during these windows.
- 2026-10-22 16:24: redeploy to host C. Host B locked up once more just before
shutdown.
- Host C: the HWC timeout happened in the first boot, about 214 s after start
(log below). The VF
never came back after that: ens1 stays DOWN, vf_tx_bytes is stuck at 6166,
and all traffic goes
through the synthetic hv_netvsc path. The VM has been stable for more than 13
hours this way.
Each lockup follows the same pattern. Example from host B, 2026-10-22:
15:50:27 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 3: transmit queue 3
timed out 15556 ms
15:50:42 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 3: transmit queue 0
timed out 15005 ms
... (one line about every 15 s) ...
15:53:33 mana 7870:00:00.0 ens1: NETDEV WATCHDOG: CPU: 3: transmit queue 0
timed out 185852 ms
15:53:39 mana 7870:00:00.0: HWC: Request timed out: 60000 ms
15:53:39 mana 7870:00:00.0: Failed to send mana message: -110, 0x0
15:53:39 mana 7870:00:00.0 ens1: Failed to configure vPort RX: -110
15:53:39 mana 7870:00:00.0 ens1: Failed to disable vPort: -110
15:53:39 mana 7870:00:00.0: HWC: Request timed out: 1 ms
15:53:39 mana 7870:00:00.0: Failed to query GF stats: -110
15:53:39 mana 7870:00:00.0: Gf stats wk handler: gf stats query timed out.
15:53:39 mana 7870:00:00.0: Start MANA service type:135
15:53:39 mana 7870:00:00.0: MANA reset cycle start
15:53:39 mana 7870:00:00.0 ens1: Failed to fence RQ 0: -110
15:53:39 mana 7870:00:00.0 mana_0: Failed to destroy ud qp err -110
15:53:39 mana 7870:00:00.0 mana_0: Failed to destroy cq err -110
15:53:39 mana 7870:00:00.0: Failed to destroy MR: -110, 0x0
15:53:39 mana 7870:00:00.0 mana_0: Failed to config IP addr err -110
15:53:39 mana 7870:00:00.0 ens1: mana_attach failed: -110
15:53:39 mana 7870:00:00.0 mana_0: Failed to destroy RNIC adapter err -110
Host C, first boot (dmesg time since boot):
[ 6.231396] mana 7870:00:00.0 eth1: joined to eth0
[ 12.626953] hv_netvsc ... eth0: Data path switched to VF: ens1
[ 12.627693] mana 7870:00:00.0 mana_0: Port: 1 Link ACTIVE
[ 18.320839] hv_netvsc ... eth0: Data path switched from VF: ens1
[ 214.875614] mana 7870:00:00.0: HWC: Request timed out: 60000 ms
[ 214.875655] mana 7870:00:00.0: Failed to send mana message: -110, 0x0
[ 214.875674] mana 7870:00:00.0 ens1: Failed to configure vPort RX: -110
[ 214.882680] mana 7870:00:00.0: Start MANA service type:135
[ 214.882684] mana 7870:00:00.0: MANA reset cycle start
[ 214.884522] mana 7870:00:00.0 ens1: Failed to fence RQ 0: -110
[ 215.343517] mana 7870:00:00.0 mana_0: Failed to destroy RNIC adapter err
-110
[ 232.076748] mana 7870:00:00.0: MANA reset cycle completed
During the reset, mana_ib/ib_core also prints WARNINGs (Workqueue: events
mana_serv_func):
WARNING: drivers/infiniband/core/verbs.c:395 at ib_dealloc_pd_user+0x77/0x80
[ib_core]
WARNING: include/rdma/ib_verbs.h:3615 at ib_mad_port_close+0x14c/0x180
[ib_core]
WARNING: drivers/infiniband/core/restrack.c:52 at
rdma_restrack_clean+0x43/0x60 [ib_core]
call trace: mana_serv_func -> mana_do_service -> mana_rdma_remove ->
mana_ib_remove
Notes
- On host C, hv_netvsc moved the data path off the VF at 18 s, before any HWC
error was logged.
- During the lockups, Azure Resource Health and VM availability metrics usually
report the VM as
available. The only clear signs are Network Out dropping to near zero and the
kernel log.
- We know about the upstream "net: mana: HW channel reliability and hardening
fixes" series
(netdev, Aug 2026) and LP #2168373 in 7.0.0-1018. We could not tell whether
either one covers
this case.
- This is a production VM, so we can't experiment with kernels on it. We can
provide full kern.log
for the affected boots.
** Affects: linux-azure (Ubuntu)
Importance: Undecided
Status: New
** Tags: azure mana noble
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172019
Title:
linux-azure 7.0.0-1017 (noble): MANA HWC requests time out (-110), TX
watchdog fires, network down for 3-6 minutes; happens on 3 different
hosts
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2172019/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
