Hello, I did the verification for resolute
CURRENT SITUATION
$ apt policy qemu-utils
qemu-utils:
Installed: 1:10.2.1+ds-1ubuntu3.1
Candidate: 1:10.2.1+ds-1ubuntu3.1
Version table:
1:10.2.1+ds-1ubuntu3.2 100
100 http://archive.ubuntu.com/ubuntu resolute-proposed/main amd64
Packages
*** 1:10.2.1+ds-1ubuntu3.1 500
500 http://archive.ubuntu.com/ubuntu resolute-updates/main amd64
Packages
100 /var/lib/dpkg/status
1:10.2.1+ds-1ubuntu3 500
500 http://archive.ubuntu.com/ubuntu resolute/main amd64 Packages
$ QEMU_MODULE_DIR=/tmp/qemu-modules/ qemu-img --trace "module_*" info
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
qemu-img: Could not open
'https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img':
failed to initialize module: /tmp/qemu-modules//block-curl.so
Only modules from the same build can be loaded.
WITH -PROPOSED:
$ apt policy qemu-utils
qemu-utils:
Installed: 1:10.2.1+ds-1ubuntu3.2
Candidate: 1:10.2.1+ds-1ubuntu3.2
Version table:
*** 1:10.2.1+ds-1ubuntu3.2 100
100 http://archive.ubuntu.com/ubuntu resolute-proposed/main amd64
Packages
100 /var/lib/dpkg/status
1:10.2.1+ds-1ubuntu3.1 500
500 http://archive.ubuntu.com/ubuntu resolute-updates/main amd64
Packages
1:10.2.1+ds-1ubuntu3 500
500 http://archive.ubuntu.com/ubuntu resolute/main amd64 Packages
$ QEMU_MODULE_DIR=/tmp/qemu-modules/ qemu-img --trace "module_*" info
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
module_load_module file /usr/lib/x86_64-linux-gnu/qemu/block-curl.so
image:
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
file format: qcow2
virtual size: 3.5 GiB (3758096384 bytes)
disk size: unavailable
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
Child node '/file':
filename:
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
protocol type: https
file length: 819 MiB (858388480 bytes)
disk size: unavailable
** Description changed:
- [ Impact ]
-
- After upgrading QEMU packages on a host, QEMU fails to load some modules
+ [ Impact ]
+
+ After upgrading QEMU packages on a host, QEMU fails to load some modules
in /run/qemu/<version>
- This might affect running instances in many ways but the original report is
about long-running instances that started with the old build can
- no longer hot-attach module-backed block devices such as Ceph RBD volumes.
-
- This is a regression introduced by upstream commit c551fb0b53d ("module: add
Error arguments to module_load and module_load_qom") which broke the directory
- fallback logic in module_load(). The CONFIG_MODULE_UPGRADES feature
- (enabled in Ubuntu builds) explicitly adds /run/qemu/<version>/ to the
- search list for this scenario — but the broken logic prevents it from
+ This might affect running instances in many ways but the original report is
about long-running instances that started with the old build can
+ no longer hot-attach module-backed block devices such as Ceph RBD volumes.
+
+ This is a regression introduced by upstream commit c551fb0b53d ("module: add
Error arguments to module_load and module_load_qom") which broke the directory
+ fallback logic in module_load(). The CONFIG_MODULE_UPGRADES feature
+ (enabled in Ubuntu builds) explicitly adds /run/qemu/<version>/ to the
+ search list for this scenario — but the broken logic prevents it from
ever being looked at.
-
- The fix restores the correct fallback behaviour (continue searching
- remaining directories when a file exists but fails to load) and fixes a
- related memory leak in module_load_dso() that caused the assertion crash
- on retry.
-
- [ Test Plan ]
-
- Prerequisites:
-
- - Ubuntu noble (24.04) host with QEMU installed
- - A VM using a module-backed device driver NOT loaded at boot (e.g. RBD)
- - Access to two consecutive QEMU package versions (e.g. 0ubuntu1.12 and
- 0ubuntu1.13) in noble-proposed or a local repo
-
- Steps to reproduce (and verify the fix):
-
- 1. Install QEMU version 1:8.2.2+ds-0ubuntu1.X on the host.
-
- 2. Start a VM that does NOT use an RBD disk at boot time.
- (The module block-rbd.so must not be loaded at QEMU start.)
-
- 3. Upgrade QEMU on the host to 0ubuntu1.(X+1) while the VM keeps running.
-
- 4. Verify the retained module directory exists:
- ls /run/qemu/Debian_1_8.2.2+ds-0ubuntu1.X/block-rbd.so
- The file should be present and readable.
-
- 5. Without the fix — hot-attach a Ceph RBD volume:
- openstack server add volume <instance> <ceph-volume>
- Expected (broken): libvirt reports "Unknown driver 'rbd'".
- Check VM log for:
- failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so
- Only modules from the same build can be loaded.
-
- 6. Retry hot-attach without the fix — second attempt:
- Expected (broken): QEMU crashes:
- qemu-system-x86_64: util/module.c:165: module_load_dso:
- Assertion `QTAILQ_EMPTY(&dso_init_list)' failed.
- VM ends up in SHUTOFF state.
-
- 7. With the fix applied, repeat steps 1–4, then hot-attach:
- Expected (fixed): the attach succeeds. QEMU loads block-rbd.so from
- /run/qemu/Debian_1_8.2.2+ds-0ubuntu1.X/block-rbd.so and the volume
- is accessible inside the VM.
-
- Minimal test without full OpenStack:
- - Use virsh/virt-manager to hot-attach a Ceph RBD disk to a running VM
- after a QEMU version upgrade, or
- - Write a small test that calls qemu QMP "blockdev-add" with driver=rbd
- after the host QEMU has been upgraded.
-
- [ Where problems could occur ]
-
- The change only affects the module directory search loop in
- util/module.c:module_load() and the cleanup path in
- util/module.c:module_load_dso(). No other code paths are touched.
-
- Regressions are possible in module loading. The most common real-world
- scenario is hot-attaching block/network devices (RBD, virtio-fs, etc.)
- that load modules late in the VM lifecycle.
-
- The fix is conservative: it restores the pre-regression legacy behaviour
- of continuing the search after a failed module load. The only observable
- difference is that QEMU will now successfully load the correct retained
- module from /run/qemu/<old-version>/ instead of stopping at the first
- (mismatched) module in /usr/lib/.
-
- [ Other Info ]
-
- Upstream submission: [PATCH v2 1/1] module: fix early stop for load module
function
- https://lists.gnu.org/archive/html/qemu-devel/2026-04/msg06376.html
-
- Upstream tracking issue:
- https://gitlab.com/qemu-project/qemu/-/work_items/3354
-
+
+ The fix restores the correct fallback behaviour (continue searching
+ remaining directories when a file exists but fails to load) and fixes a
+ related memory leak in module_load_dso() that caused the assertion crash
+ on retry.
+
+ [ Test Plan ]
+
+ The idea of the test plan is to demonstrate that QEMU stops searching
+ for the module at the first module load failure.
+
+ For that we can use the curl-block.so module that is easy to trigger the
loading
+ in QEMU.
+
+ Steps to reproduce (and verify the fix):
+
+ 1. Create /tmp/modules/
+ 2. Get a curl-block.so from a qemu-utils package that does not match the
version of the qemu-utils we have on the system.
+ On Resolute, we can take:
https://launchpad.net/ubuntu/+archive/primary/+files/qemu-block-extra_10.2.1+ds-1ubuntu3_amd64.deb
+ 3. Extract the contents and copy the block-curl.so to /tmp/qemu-modules
+ 4. Run qemu-img:
+
+ qemu-img --trace "module_*" info https://cloud-
+ images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
+
+ WITHOUT THE FIX:
+
+ This command will fail:
+
+ qemu-img: Could not open
'https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img':
failed to initialize module: /tmp/qemu-modules//block-curl.so
+ Only modules from the same build can be loaded.
+
+
+ WITH THE FIX:
+
+ This command succeeds:
+
+ qemu-img: failed to initialize module: /tmp/qemu-modules//block-curl.so
+ Only modules from the same build can be loaded.
+ module_load_module file /usr/lib/x86_64-linux-gnu/qemu/block-curl.so
+ image:
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
+ file format: qcow2
+ virtual size: 3.5 GiB (3758096384 bytes)
+ disk size: unavailable
+ cluster_size: 65536
+ Format specific information:
+ compat: 1.1
+ compression type: zlib
+ lazy refcounts: false
+ refcount bits: 16
+ corrupt: false
+ extended l2: false
+ Child node '/file':
+ filename:
https://cloud-images.ubuntu.com/stonking/current/stonking-server-cloudimg-amd64.img
+ protocol type: https
+ file length: 819 MiB (858388480 bytes)
+ disk size: unavailable
+
+ [ Where problems could occur ]
+
+ The change only affects the module directory search loop in
+ util/module.c:module_load() and the cleanup path in
+ util/module.c:module_load_dso(). No other code paths are touched.
+
+ Regressions are possible in module loading. The most common real-world
+ scenario is hot-attaching block/network devices (RBD, virtio-fs, etc.)
+ that load modules late in the VM lifecycle.
+
+ The fix is conservative: it restores the pre-regression legacy behaviour
+ of continuing the search after a failed module load. The only observable
+ difference is that QEMU will now successfully load the correct retained
+ module from /run/qemu/<old-version>/ instead of stopping at the first
+ (mismatched) module in /usr/lib/.
+
+ [ Other Info ]
+
+ Upstream submission: [PATCH v2 1/1] module: fix early stop for load module
function
+ https://lists.gnu.org/archive/html/qemu-devel/2026-04/msg06376.html
+
+ Upstream tracking issue:
+ https://gitlab.com/qemu-project/qemu/-/work_items/3354
Original bug report
---
## Package
qemu-system-x86 (Ubuntu noble)
## Affects
qemu (Ubuntu)
## Related bugs
- LP #1847361 (Upgrade of qemu binaries causes running instances to be unable
to hot-attach)
- LP #1913421 (module retention improvements)
## Description
### Summary
After upgrading QEMU packages on a compute node (e.g. from
`1:8.2.2+ds-0ubuntu1.12` to `0ubuntu1.13`), long-running VM instances
started with the older build can no longer hot-attach Ceph RBD volumes —
even though `/run/qemu/` contains the retained modules for the old
build.
The first attach attempt fails with "Unknown driver 'rbd'". A second
attempt crashes QEMU with an assertion failure.
This is a regression in the module-retention mechanism introduced for LP
#1847361.
### Root cause
Two bugs in `util/module.c` (confirmed identical on current QEMU master
as of 2026-03-26):
**Bug A — module_load() does not fall back on build mismatch:**
The directory search loop (lines 282–303) only continues to the next
directory when the module file is not found (`ENOENT`). When the file
exists but `module_load_dso()` fails (build mismatch), the loop hits
`goto out` immediately — never reaching `/run/qemu/<version>/`.
`CONFIG_MODULE_UPGRADES` is enabled in the Ubuntu noble build
(`debian/rules`: `$(if ${enable-system},--enable-module-upgrades)`), so
the `/run/qemu/<version>/` path is added to the search list — but it is
never reached because the system path (`/usr/lib/x86_64-linux-
gnu/qemu/`) contains the new build's modules, which exist but fail the
stamp check.
**Bug B — module_load_dso() leaks dso_init_list on failure:**
When `g_module_open()` loads a `.so`, its constructors populate
`dso_init_list`. On build mismatch, `g_module_close()` is called but
`dso_init_list` is not drained. On the next module load attempt,
`assert(QTAILQ_EMPTY(&dso_init_list))` fires and QEMU aborts.
### Environment
- Ubuntu 24.04 (noble), OpenStack compute nodes (Nova Victoria, libvirt/kvm,
Cinder/Ceph RBD)
- Kernel: `6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC x86_64`
- QEMU: `qemu-system-x86 1:8.2.2+ds-0ubuntu1.13`
- libvirt: 10.0.0-2ubuntu8.12
- AppArmor: enabled, no DENIED entries for `/run/qemu` or `block-rbd.so`
- `/run/qemu` mounted as tmpfs (rw, no noexec)
### Observed symptoms
**Instance started with QEMU 0ubuntu1.11, host upgraded to
0ubuntu1.13:**
Instance log (first attach attempt):
```
failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so
Only modules from the same build can be loaded.
```
libvirt:
```
internal error: unable to execute QEMU command 'blockdev-add': Unknown driver
'rbd'
```
VM continues running, but attach fails. `/proc/$PID/maps` shows no
mapping of `block-rbd.so`.
Second attempt — instance log:
```
qemu-system-x86_64: util/module.c:165: module_load_dso: Assertion
`QTAILQ_EMPTY(&dso_init_list)' failed.
```
QEMU exits (`reason=crashed`), VM ends up in SHUTOFF state.
At the time of the failure, the retained modules exist:
```
/run/qemu/Debian_1_8.2.2+ds-0ubuntu1.12/block-rbd.so (40312 bytes, readable)
/run/qemu/Debian_1_8.2.2+ds-0ubuntu1.11/block-rbd.so (40312 bytes, readable)
```
This has been reproduced across multiple minor build upgrades
(0ubuntu1.11→12 and 0ubuntu1.12→13).
### Steps to reproduce
1. Start an OpenStack instance on a compute node running QEMU
`1:8.2.2+ds-0ubuntu1.X`. The instance must not use RBD at boot.
2. Upgrade QEMU on the host to `0ubuntu1.(X+1)` while the instance keeps
running.
3. Verify `/run/qemu/Debian_1_8.2.2+ds-0ubuntu1.X/block-rbd.so` exists.
4. Hot-attach a Cinder/Ceph RBD volume (`openstack server add volume`).
5. First attempt: "Unknown driver 'rbd'".
6. Second attempt: QEMU assertion crash.
### Impact
- Long-running VMs that predate a QEMU package upgrade cannot hot-attach RBD
volumes (or any other module-backed driver not already loaded).
- Second attempt crashes the VM, causing unplanned downtime.
- Defeats the purpose of the `/run/qemu/` module-retention mechanism (LP
#1847361, LP #1913421).
### Proposed fix
See upstream QEMU GitLab issue (https://gitlab.com/qemu-
project/qemu/-/work_items/3354) for detailed code analysis and patch
proposals. Summary:
- **Bug A:** On `module_load_dso()` failure, clear the error and `continue`
to the next directory instead of `goto out`.
- **Bug B:** In `module_load_dso()`, drain `dso_init_list` before
`g_module_close()` when the stamp check fails.
Both fixes are against upstream `util/module.c` — the code is identical
on current QEMU master.
### Current workaround
Proactively reboot or live-migrate any instance whose running QEMU
version (via QMP `query-version`) does not match the installed package
version, before hot-attaching RBD volumes.
** Tags removed: verification-needed-resolute
** Tags added: verification-done-resolute
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2146445
Title:
qemu-system-x86: module upgrade fallback in /run/qemu/ broken —
"Unknown driver 'rbd'" + crash on retry
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2146445/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
