Just theorizing for now ...
For a packaging solution we would need:
- qemu-block-extra / qemu-system-gui binary packages would need
sort of a -$buildid in the name. That could be the version
string (sanitized for package name)
- /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid
- loading of modules from qemu would need to consider $buildid
when creating module names.
util/module.c in module_load_one / module_load_file
It already searches in multiple dirs, maybe it could insert
the $buildid there
- We'd need a way of detecting running versions of qemu binaries
and only make them uninstallable once the binaries are all
gone. I have not seen something like that in apt yet (kernel
is easy in comparison as only one can be loaded at a time).
ALTERNATIVES:
- disable loadable module support
- add an option to load all modules in advance (unlikely to be
liked upstream) and not desirable for many setups using qemu
(especially not as default)
- add an option to load a module (e.g via QMP/HMP) which would
allow an admin
to decide doing so for the few setups that benefit.
- that could down the road then even get a libvirt interface
for easier consumption
Heads up - None of the above would be SRUable
** Changed in: qemu (Ubuntu)
Importance: Undecided => Wishlist
** Changed in: qemu (Ubuntu)
Status: Confirmed => Triaged
** Description changed:
+ --- initial report ---
+
Upgrading qemu binaries causes the on-disk versions to change, but the
in-memory running instances still attempt to dynamically load a library
which matches its same version. This can cause running instances to fail
actions like hotplugging devices. This can be alleviated by migrating
the instance to a new host or restarting the instance, however in cloud
type environments there may be instances that cannot be migrated (sriov,
etc) or the cloud operator does not have permission to reboot.
This may be resolvable for many situations by changing the packaging to
keep older versions of qemu libraries around on disk (similar to how the
kernel package keeps older kernel versions around).
+
+ --- testcase ---
+
+ $ apt install uvtool-libvirt
+ $ uvt-simplestreams-libvirt --verbose sync --source
http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
+ $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic
label=daily
+
+ cat > curldisk.xml << EOF
+ <disk type='network' device='disk'>
+ <driver name='qemu' type='raw'/>
+ <source protocol="http"
name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso">
+ <host name="archive.ubuntu.com" port="80"/>
+ </source>
+ <target dev='vdc' bus='virtio'/>
+ <readonly/>
+ </disk>
+ EOF
+
+ # Here up or downgrade the installed packages, even a minor
+ # version or a rebuild of the same version
+
+ $ virsh attach-device lateload curldisk.xml
+ Reported issue happens on attach:
+ root@b:~# virsh attach-device lateload cdrom-curl.xml
+ error: Failed to attach device from cdrom-curl.xml
+ error: internal error: unable to execute QEMU command 'device_add': Property
'virtio-blk-device.drive' can't find value 'drive-virtio-disk2'
+
+ In the log we can see:
+ Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so
+ Note: only modules from the same build can be loaded.
+
+ --- solution options (WIP) ---
+
+ For a packaging solution we would need:
+ - qemu-block-extra / qemu-system-gui binary packages would need
+ sort of a -$buildid in the name. That could be the version
+ string (sanitized for package name)
+ - /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid
+ - loading of modules from qemu would need to consider $buildid
+ when creating module names.
+ util/module.c in module_load_one / module_load_file
+ It already searches in multiple dirs, maybe it could insert
+ the $buildid there
+ - We'd need a way of detecting running versions of qemu binaries
+ and only make them uninstallable once the binaries are all
+ gone. I have not seen something like that in apt yet (kernel
+ is easy in comparison as only one can be loaded at a time).
+
+ ALTERNATIVES:
+ - disable loadable module support
+ - add an option to load all modules in advance (unlikely to be
+ liked upstream) and not desirable for many setups using qemu
+ (especially not as default)
+ - add an option to load a module (e.g via QMP/HMP) which would
+ allow an admin
+ to decide doing so for the few setups that benefit.
+ - that could down the road then even get a libvirt interface
+ for easier consumption
+
+ Heads up - None of the above would be SRUable
+
+ --- mitigation options ---
+
+ - live migrate for upgrades
+ - prohibited by SR-IOV usage
+ - Tech to get SR-IOV migratable is coming (e.g. via net_failover,
+ bonding in DPDK, ...)
+ - load the modules you need in advance
+ - Note: lacking an explicit "load module" command makes this
+ slightly odd for now
+ - but using iscsi or ceph is never spontaneous, a deployment
+ has or hasn't the setup to use those
+ - Create a single small read-only node and attach this to each guest,
+ that will load the driver and render you immune to the issue. While
+ more clunky, this isn't so much different than how it would be
+ with an explicit "load module" command.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1847361
Title:
Upgrade of qemu binaries causes running instances not able to
dynamically load modules
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1847361/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
