Public bug reported:
Summary
-------
On an Azure VM with Accelerated Networking, the kernel oopsed in the hv_pci
eject
worker when the host removed the SR-IOV VF (presumably during host servicing).
The
worker exited with IRQs disabled, the host re-offered the VF about a minute
later but
the guest never probed it again, and from then on the guest could not make
outbound
connections and the Azure Linux guest agent became unresponsive. Inbound
connections to
already-running services kept working. Only a reboot recovered the VM.
Environment
-----------
- Ubuntu 24.04.5 LTS (noble)
- linux-image-azure 7.0.0-1017.17~24.04.1 (linux-azure-7.0, from noble-updates)
- Azure VM size Standard_D2as_v5 (AMD), Gen2 (Hyper-V UEFI v4.1)
- Accelerated Networking enabled (Mellanox VF, mlx5_core)
- Kernel not tainted
- Uptime at the time of the oops: about 20 days (1747322 s)
- Date of the oops: 2026-10-24 22:10 UTC
Timeline (kernel time, serial console)
--------------------------------------
- 1747322.2 BUG: kernel NULL pointer dereference in strlen(), from
hv_eject_device_work -> pci_stop_and_remove_bus_device ->
pci_stop_bus_device -> pci_remove_sysfs_dev_files ->
pci_remove_resource_files -> sysfs_remove_bin_file ->
kernfs_remove_by_name_ns -> kernfs_find_ns (full trace below)
- 1747330.4 note: kworker/u8:4[...] exited with irqs disabled
- 1747381.9 hv_netvsc <guid> eth0: VF slot 1 removed
- 1747393.6 hv_netvsc <guid> eth0: VF slot 1 added
- After that, no further kernel messages: the re-added VF was never probed (no
"hv_pci ...: PCI VMBus probing" and no mlx5_core probe, unlike at boot).
Impact observed after the oops
------------------------------
- Outbound HTTPS requests from the running application (to
login.microsoftonline.com
and graph.microsoft.com) failed with connection errors or timeouts.
- All cron jobs stopped producing output, including a purely local hourly SQLite
backup job that needs no network.
- Azure reported the guest agent as "Not Ready - VM Agent is unresponsive", so
Run
Command could not be used.
- Inbound HTTPS to the already-running nginx and uvicorn processes kept being
served
normally.
- The state lasted about 37 hours, until the VM was restarted from Azure. After
the
reboot everything worked again and the VF was added and probed normally at
boot.
Possible explanation (not verified): the worker died inside
kernfs_remove_by_name_ns(),
which takes the kernfs root rwsem for writing, and hv_eject_device_work() runs
under
pci_lock_rescan_remove(). If either lock stayed held, every later sysfs access
or PCI
rescan/remove would block, which would match the unresponsive guest agent, the
re-added
VF never being probed and new processes stalling.
Expected behaviour
------------------
The VF hot-remove completes without an oops (traffic falls back to the synthetic
netvsc path) and the VF is probed again when the host re-adds it.
Reproducibility
---------------
Seen once. It was the first host-initiated VF removal recorded for this VM (the
serial
console log covers nine boots, from 6.17.0-1008-azure onwards), and we cannot
trigger
one on demand. Another VM on the same kernel without Accelerated Networking was
not
affected, as expected, since it has no VF.
Workaround
----------
Reboot the VM. Disabling Accelerated Networking avoids this code path.
Oops (serial console; VMBus instance GUIDs and the hv_pci domain number
replaced by
placeholders)
-----------------------------------------------------------------------------------
[1747322.218312] BUG: kernel NULL pointer dereference, address: 0000000000000000
[1747322.220610] #PF: supervisor read access in kernel mode
[1747322.222295] #PF: error_code(0x0000) - not-present page
[1747322.223942] PGD 0 P4D 0
[1747322.224805] Oops: Oops: 0000 [#1] SMP NOPTI
[1747322.226163] CPU: 1 UID: 0 PID: 783587 Comm: kworker/u8:4 Not tainted
7.0.0-1017-azure #17~24.04.1-Ubuntu PREEMPTLAZY
[1747322.229369] Hardware name: Microsoft Corporation Virtual Machine/Virtual
Machine, BIOS Hyper-V UEFI Release v4.1 01/08/2026
[1747322.235660] Workqueue: hv_pci_xxxx hv_eject_device_work
[1747322.237544] RIP: 0010:strlen+0x0/0x30
[1747322.238801] Code: 48 89 f8 31 d2 31 f6 31 ff e9 87 3b a6 fe 66 66 2e 0f 1f
84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <80> 3f 00
74 16 48 89 f8 48 83 c0 01 80 38 00 75 f7 48 29 f8 31 ff
[1747322.244334] RSP: 0018:ffffcd2f45d37cb0 EFLAGS: 00010246
[1747322.245996] RAX: ffff8b6d43b93dd0 RBX: 0000000000000000 RCX:
0000000000000000
[1747322.248422] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[1747322.250666] RBP: ffffcd2f45d37cc8 R08: 0000000000000000 R09:
0000000000000000
[1747322.252921] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[1747322.255155] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8b6d40b4f000
[1747322.257503] FS: 0000000000000000(0000) GS:ffff8b6f546c8000(0000)
knlGS:0000000000000000
[1747322.259943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1747322.261828] CR2: 0000000000000000 CR3: 0000000107836002 CR4:
0000000000b70ef0
[1747322.264136] Call Trace:
[1747322.265006] <TASK>
[1747322.265812] ? kernfs_name_hash+0x95/0xc0
[1747322.267118] kernfs_find_ns+0x5b/0x130
[1747322.268341] kernfs_remove_by_name_ns+0x5f/0x120
[1747322.269818] sysfs_remove_bin_file+0x17/0x30
[1747322.271221] pci_remove_resource_files+0x68/0xb0
[1747322.272739] pci_remove_sysfs_dev_files+0x21/0x30
[1747322.274319] pci_stop_bus_device+0x79/0x90
[1747322.275678] pci_stop_and_remove_bus_device+0x12/0x30
[1747322.277298] hv_eject_device_work+0x9d/0x1d0
[1747322.278769] ? srso_alias_return_thunk+0x5/0xfbef5
[1747322.280791] ? finish_task_switch.isra.0+0x95/0x2f0
[1747322.282373] ? srso_alias_return_thunk+0x5/0xfbef5
[1747322.284012] process_one_work+0x19c/0x3d0
[1747322.285358] worker_thread+0x1bf/0x350
[1747322.286622] ? __pfx_worker_thread+0x10/0x10
[1747322.288057] kthread+0xfb/0x140
[1747322.289157] ? __pfx_kthread+0x10/0x10
[1747322.290371] ret_from_fork+0x1f8/0x2b0
[1747322.291604] ? __pfx_kthread+0x10/0x10
[1747322.292884] ret_from_fork_asm+0x1a/0x30
[1747322.294255] </TASK>
[1747322.295061] Modules linked in: qrtr wireguard libcurve25519 ip6_udp_tunnel
udp_tunnel cfg80211 8021q garp mrp stp llc xt_owner xt_tcpudp xt_conntrack
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables binfmt_misc
mlx5_ib ib_uverbs macsec ib_core mlx5_fwctl fwctl nls_iso8859_1 intel_rapl_msr
intel_rapl_common kvm_amd ccp mlx5_core kvm joydev mac_hid hid_generic mlxfw
irqbypass hid_hyperv psample serio_raw ghash_clmulni_intel hyperv_drm tls hid
vmgenid aesni_intel hyperv_keyboard sch_fq_codel dm_multipath msr i2c_dev
nvme_fabrics efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4
[1747322.315857] CR2: 0000000000000000
[1747322.317247] ---[ end trace 0000000000000000 ]---
[1747330.407786] RIP: 0010:strlen+0x0/0x30
[1747330.408175] Code: 48 89 f8 31 d2 31 f6 31 ff e9 87 3b a6 fe 66 66 2e 0f 1f
84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <80> 3f 00
74 16 48 89 f8 48 83 c0 01 80 38 00 75 f7 48 29 f8 31 ff
[1747330.408853] RSP: 0018:ffffcd2f45d37cb0 EFLAGS: 00010246
[1747330.409207] RAX: ffff8b6d43b93dd0 RBX: 0000000000000000 RCX:
0000000000000000
[1747330.409544] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[1747330.409887] RBP: ffffcd2f45d37cc8 R08: 0000000000000000 R09:
0000000000000000
[1747330.410220] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[1747330.410533] R13: 0000000000000000 R14: 0000000000000000 R15:
ffff8b6d40b4f000
[1747330.410834] FS: 0000000000000000(0000) GS:ffff8b6f546c8000(0000)
knlGS:0000000000000000
[1747330.411128] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1747330.411413] CR2: 0000000000000000 CR3: 0000000211244001 CR4:
0000000000b70ef0
[1747330.411666] note: kworker/u8:4[783587] exited with irqs disabled
[1747381.857943] hv_netvsc <guid> eth0: VF slot 1 removed
[1747393.585096] hv_netvsc <guid> eth0: VF slot 1 added
** Affects: linux-azure (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172226
Title:
linux-azure 7.0.0-1017: NULL pointer dereference in
hv_eject_device_work when the host removes the Accelerated Networking
VF
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2172226/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
