Yeah, the workaround can be a disk that "fails" which helps to be not
guest detectable:
Check libs before the workaround:
$ sudo lsof -p 9952 | awk '{print $9}' | grep '\.so' | sort | uniq > pre.libs
Then attach a XML that is valid for libvirt, but will fail to attach a disk
(after loading the .so files)
<disk type='network' device='disk'>
<driver name='qemu' type='raw'/>
<source protocol="http" name="does-not-exist">
<host name="localhost" port="80"/>
</source>
<target dev='vdc' bus='virtio'/>
<readonly/>
</disk>
You'll get a fail (intentionally) like:
$ virsh attach-device guest1 non-disk.xml
error: Failed to attach device from on-disk.xml
error: internal error: unable to execute QEMU command 'device_add': Property
'virtio-blk-device.drive' can't find value 'drive-virtio-disk2'
Check after this:
$ sudo lsof -p 9952 | awk '{print $9}' | grep '\.so' | sort | uniq > post.libs
And we see all of them loaded:
$ diff -Naur pre.libs post.libs | grep '^+'
+++ post.libs 2019-11-18 07:01:55.461495910 +0100
+/lib/x86_64-linux-gnu/libcom_err.so.2.1
+/lib/x86_64-linux-gnu/libcrypt-2.27.so
+/lib/x86_64-linux-gnu/libkeyutils.so.1.5
+/usr/lib/x86_64-linux-gnu/libasn1.so.8.0.0
+/usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4.5.0
+/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2.2
+/usr/lib/x86_64-linux-gnu/libgssapi.so.3.0.0
+/usr/lib/x86_64-linux-gnu/libhcrypto.so.4.1.0
+/usr/lib/x86_64-linux-gnu/libheimbase.so.1.0.0
+/usr/lib/x86_64-linux-gnu/libheimntlm.so.0.1.0
+/usr/lib/x86_64-linux-gnu/libhx509.so.5.0.0
+/usr/lib/x86_64-linux-gnu/libk5crypto.so.3.1
+/usr/lib/x86_64-linux-gnu/libkrb5.so.26.0.0
+/usr/lib/x86_64-linux-gnu/libkrb5.so.3.3
+/usr/lib/x86_64-linux-gnu/libkrb5support.so.0.1
+/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2.10.8
+/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2.10.8
+/usr/lib/x86_64-linux-gnu/libnghttp2.so.14.15.2
+/usr/lib/x86_64-linux-gnu/libpsl.so.5.2.0
+/usr/lib/x86_64-linux-gnu/libroken.so.18.1.0
+/usr/lib/x86_64-linux-gnu/librtmp.so.1
+/usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
+/usr/lib/x86_64-linux-gnu/libwind.so.0.0.0
+/usr/lib/x86_64-linux-gnu/qemu/block-curl.so
** Description changed:
--- initial report ---
Upgrading qemu binaries causes the on-disk versions to change, but the
in-memory running instances still attempt to dynamically load a library
which matches its same version. This can cause running instances to fail
actions like hotplugging devices. This can be alleviated by migrating
the instance to a new host or restarting the instance, however in cloud
type environments there may be instances that cannot be migrated (sriov,
etc) or the cloud operator does not have permission to reboot.
This may be resolvable for many situations by changing the packaging to
keep older versions of qemu libraries around on disk (similar to how the
kernel package keeps older kernel versions around).
--- testcase ---
$ apt install uvtool-libvirt
$ uvt-simplestreams-libvirt --verbose sync --source
http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic
$ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic
label=daily
cat > curldisk.xml << EOF
- <disk type='network' device='disk'>
- <driver name='qemu' type='raw'/>
- <source protocol="http"
name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso">
- <host name="archive.ubuntu.com" port="80"/>
- </source>
- <target dev='vdc' bus='virtio'/>
- <readonly/>
- </disk>
+ <disk type='network' device='disk'>
+ <driver name='qemu' type='raw'/>
+ <source protocol="http"
name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso">
+ <host name="archive.ubuntu.com" port="80"/>
+ </source>
+ <target dev='vdc' bus='virtio'/>
+ <readonly/>
+ </disk>
EOF
# Here up or downgrade the installed packages, even a minor
# version or a rebuild of the same version
$ virsh attach-device lateload curldisk.xml
Reported issue happens on attach:
root@b:~# virsh attach-device lateload cdrom-curl.xml
error: Failed to attach device from cdrom-curl.xml
error: internal error: unable to execute QEMU command 'device_add': Property
'virtio-blk-device.drive' can't find value 'drive-virtio-disk2'
In the log we can see:
Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so
Note: only modules from the same build can be loaded.
--- solution options (WIP) ---
For a packaging solution we would need:
- qemu-block-extra / qemu-system-gui binary packages would need
- sort of a -$buildid in the name. That could be the version
- string (sanitized for package name)
+ sort of a -$buildid in the name. That could be the version
+ string (sanitized for package name)
- /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid
- loading of modules from qemu would need to consider $buildid
- when creating module names.
- util/module.c in module_load_one / module_load_file
- It already searches in multiple dirs, maybe it could insert
- the $buildid there
+ when creating module names.
+ util/module.c in module_load_one / module_load_file
+ It already searches in multiple dirs, maybe it could insert
+ the $buildid there
- We'd need a way of detecting running versions of qemu binaries
- and only make them uninstallable once the binaries are all
- gone. I have not seen something like that in apt yet (kernel
- is easy in comparison as only one can be loaded at a time).
+ and only make them uninstallable once the binaries are all
+ gone. I have not seen something like that in apt yet (kernel
+ is easy in comparison as only one can be loaded at a time).
ALTERNATIVES:
- disable loadable module support
- add an option to load all modules in advance (unlikely to be
- liked upstream) and not desirable for many setups using qemu
- (especially not as default)
+ liked upstream) and not desirable for many setups using qemu
+ (especially not as default)
- add an option to load a module (e.g via QMP/HMP) which would
- allow an admin
- to decide doing so for the few setups that benefit.
- - that could down the road then even get a libvirt interface
- for easier consumption
+ allow an admin
+ to decide doing so for the few setups that benefit.
+ - that could down the road then even get a libvirt interface
+ for easier consumption
Heads up - None of the above would be SRUable
--- mitigation options ---
- live migrate for upgrades
- - prohibited by SR-IOV usage
- - Tech to get SR-IOV migratable is coming (e.g. via net_failover,
- bonding in DPDK, ...)
+ - prohibited by SR-IOV usage
+ - Tech to get SR-IOV migratable is coming (e.g. via net_failover,
+ bonding in DPDK, ...)
- load the modules you need in advance
- - Note: lacking an explicit "load module" command makes this
- slightly odd for now
- - but using iscsi or ceph is never spontaneous, a deployment
- has or hasn't the setup to use those
- - Create a single small read-only node and attach this to each guest,
- that will load the driver and render you immune to the issue. While
- more clunky, this isn't so much different than how it would be
- with an explicit "load module" command.
+ - Note: lacking an explicit "load module" command makes this
+ slightly odd for now
+ - but using iscsi or ceph is never spontaneous, a deployment
+ has or hasn't the setup to use those
+ - Create a single small read-only node and attach this to each guest,
+ that will load the driver and render you immune to the issue. While
+ more clunky, this isn't so much different than how it would be
+ with an explicit "load module" command.
+ Actually the target doesn't have to exist it can fail to attach
+ and still achieves what is needed comment #17 has an example.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1847361
Title:
Upgrade of qemu binaries causes running instances not able to
dynamically load modules
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1847361/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
