** Description changed: + [Impact] - for SRU template: - QEMU_MODULE_DIR="/tmp/" qemu-system-x86_64 -cdrom localhost::/foo + * An infrequent but annoying issue is qemus problem to not be able to + hot-add capabilities IF since starting the instance qemu has been + upgraded. This is due to qemu modules only working with exactly the + same build. + + * We brought changes upstream that allow the packaging to keep the old + files around and make qemu look after them as a fallback. + + [Test Case] + + I: + * $ apt install uvtool-libvirt + $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic + $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic label=daily + + cat > curldisk.xml << EOF + <disk type='network' device='disk'> + <driver name='qemu' type='raw'/> + <source protocol="http" name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso"> + <host name="archive.ubuntu.com" port="80"/> + </source> + <target dev='vdc' bus='virtio'/> + <readonly/> + </disk> + EOF + + # Here up or downgrade the installed packages, even a minor + # version or a rebuild of the same version + # Instead if you prefer (easier) you can run + $ apt install --reinstall qemu-* + + Next check if they appeared (action of the maintainer scripts) + in the /var/run/qemu/<version> directory + + # And then rm/mv the original .so files of qemu-block-extra + # Trying to load a .so now would after an upgrade fail as the old qemu can't load the build id + + $ virsh attach-device lateload curldisk.xml + Reported issue happens on attach: + root@b:~# virsh attach-device lateload cdrom-curl.xml + error: Failed to attach device from cdrom-curl.xml + error: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk2' + + In the log we can see: + Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so + + One can also check files mapped into a process and we should see the + /var/run/.. path being used now. + + + II: + * As it had issues in the first iteration of the fix worth a + try is also the use of an environment var for an extra path: + $ QEMU_MODULE_DIR="/tmp/" qemu-system-x86_64 -cdrom localhost::/foo + + + [Regression Potential] + + I: + * libvirt just allows a few more paths to be read from in the apparmor + isolation that is usually safe unless these paths are considered + sensitive. But /var/run/qemu is new, /var/run in general not meant + for permanent or secure data and as always if people want to ramp up + isolation they can always add deny rules to the local overrides. + + II: + * the qemu change has two components. + In qemu code it looks for another path if the former ones failed. + I see no issues there yet, but can imagine that odd versions might + make it access odd paths which would then be denied by apparmor or + just don't exist. But that is no different than the former built-in + paths it tries, so nothing bad should happen. + The code change to the maintainer scripts has to backup the files. + If that goes wrong upgrades could be broken, but so far no tests have + shown issues. + + + [Other Info] + + * To really use the functionality users will need the new qemu AND the + new libvirt that are uploaded for this bug. + But it felt wrong to add versioned dependencies from qemu->libvirt + (that is the semantically correct direction) also conflicts/breaks + might cause issues in many places that want to control these. OTOH + while the fix is great for some installations the majority of users + won't care and therefore be happy if extra dependencies are not + causing any oddity on apt upgrade. Therefore no versioned + dependencies were added intentionally. + --- [Feature Freeze Exception] Hi, this is IMHO a just a bugfix. But since it involves some bigger changes I wanted to be on the safe side and get an ack by the release Team. Problem: - on upgrade qemu processes are left running as they represent a guest VM - later trying to add features e.g. ceph disk hot-add will need to load .so files e.g. from qemu-block-extra package - those modules can on ly be loaded from the same build, but those are gone after upgrade Solution: - If qemu fails to load from its usual paths it will now also look in /var/run/<version/ - package upgrade code will place the .so's there - things will be cleaned on reboot which is much simpler and error-proof than trying to detect which versions binaries are running - libvirt has a change to allow just reading and mapping from that path (apparmor) @Release team it would be great if you would agree to this being safe for an FFe. --- initial report --- Upgrading qemu binaries causes the on-disk versions to change, but the in-memory running instances still attempt to dynamically load a library which matches its same version. This can cause running instances to fail actions like hotplugging devices. This can be alleviated by migrating the instance to a new host or restarting the instance, however in cloud type environments there may be instances that cannot be migrated (sriov, etc) or the cloud operator does not have permission to reboot. This may be resolvable for many situations by changing the packaging to keep older versions of qemu libraries around on disk (similar to how the kernel package keeps older kernel versions around). - --- testcase --- - - $ apt install uvtool-libvirt - $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic - $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic label=daily - - cat > curldisk.xml << EOF - <disk type='network' device='disk'> - <driver name='qemu' type='raw'/> - <source protocol="http" name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso"> - <host name="archive.ubuntu.com" port="80"/> - </source> - <target dev='vdc' bus='virtio'/> - <readonly/> - </disk> - EOF - - # Here up or downgrade the installed packages, even a minor - # version or a rebuild of the same version - - $ virsh attach-device lateload curldisk.xml - Reported issue happens on attach: - root@b:~# virsh attach-device lateload cdrom-curl.xml - error: Failed to attach device from cdrom-curl.xml - error: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk2' - - In the log we can see: - Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so - Note: only modules from the same build can be loaded. --- solution options (WIP) --- For a packaging solution we would need: - qemu-block-extra / qemu-system-gui binary packages would need sort of a -$buildid in the name. That could be the version string (sanitized for package name) - /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid - loading of modules from qemu would need to consider $buildid when creating module names. util/module.c in module_load_one / module_load_file It already searches in multiple dirs, maybe it could insert the $buildid there - We'd need a way of detecting running versions of qemu binaries and only make them uninstallable once the binaries are all gone. I have not seen something like that in apt yet (kernel is easy in comparison as only one can be loaded at a time). ALTERNATIVES: - disable loadable module support - add an option to load all modules in advance (unlikely to be liked upstream) and not desirable for many setups using qemu (especially not as default) - add an option to load a module (e.g via QMP/HMP) which would allow an admin to decide doing so for the few setups that benefit. - that could down the road then even get a libvirt interface for easier consumption Heads up - None of the above would be SRUable --- mitigation options --- - live migrate for upgrades - prohibited by SR-IOV usage - Tech to get SR-IOV migratable is coming (e.g. via net_failover, bonding in DPDK, ...) - load the modules you need in advance - Note: lacking an explicit "load module" command makes this slightly odd for now - but using iscsi or ceph is never spontaneous, a deployment has or hasn't the setup to use those - Create a single small read-only node and attach this to each guest, that will load the driver and render you immune to the issue. While more clunky, this isn't so much different than how it would be with an explicit "load module" command. Actually the target doesn't have to exist it can fail to attach and still achieves what is needed comment #17 has an example.
** Description changed: [Impact] - * An infrequent but annoying issue is qemus problem to not be able to - hot-add capabilities IF since starting the instance qemu has been - upgraded. This is due to qemu modules only working with exactly the - same build. + * An infrequent but annoying issue is QEMUs problem to not be able to + hot-add capabilities IF since starting the instance qemu has been + upgraded. This is due to qemu modules only working with exactly the + same build. - * We brought changes upstream that allow the packaging to keep the old - files around and make qemu look after them as a fallback. + * We brought changes upstream that allow the packaging to keep the old + files around and make qemu look after them as a fallback. [Test Case] - I: - * $ apt install uvtool-libvirt - $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic - $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic label=daily + I: + * $ apt install uvtool-libvirt + $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic + $ uvt-kvm create --password ubuntu lateload arch=amd64 release=bionic label=daily cat > curldisk.xml << EOF <disk type='network' device='disk'> <driver name='qemu' type='raw'/> <source protocol="http" name="ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso"> <host name="archive.ubuntu.com" port="80"/> </source> <target dev='vdc' bus='virtio'/> <readonly/> </disk> EOF # Here up or downgrade the installed packages, even a minor # version or a rebuild of the same version # Instead if you prefer (easier) you can run - $ apt install --reinstall qemu-* + $ apt install --reinstall qemu-* Next check if they appeared (action of the maintainer scripts) in the /var/run/qemu/<version> directory # And then rm/mv the original .so files of qemu-block-extra # Trying to load a .so now would after an upgrade fail as the old qemu can't load the build id $ virsh attach-device lateload curldisk.xml Reported issue happens on attach: root@b:~# virsh attach-device lateload cdrom-curl.xml error: Failed to attach device from cdrom-curl.xml error: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk2' In the log we can see: Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so One can also check files mapped into a process and we should see the /var/run/.. path being used now. + II: + * As it had issues in the first iteration of the fix worth a + try is also the use of an environment var for an extra path: + $ QEMU_MODULE_DIR="/tmp/" qemu-system-x86_64 -cdrom localhost::/foo - II: - * As it had issues in the first iteration of the fix worth a - try is also the use of an environment var for an extra path: - $ QEMU_MODULE_DIR="/tmp/" qemu-system-x86_64 -cdrom localhost::/foo + [Regression Potential] + I: + * libvirt just allows a few more paths to be read from in the apparmor + isolation that is usually safe unless these paths are considered + sensitive. But /var/run/qemu is new, /var/run in general not meant + for permanent or secure data and as always if people want to ramp up + isolation they can always add deny rules to the local overrides. - [Regression Potential] - - I: - * libvirt just allows a few more paths to be read from in the apparmor - isolation that is usually safe unless these paths are considered - sensitive. But /var/run/qemu is new, /var/run in general not meant - for permanent or secure data and as always if people want to ramp up - isolation they can always add deny rules to the local overrides. - - II: - * the qemu change has two components. - In qemu code it looks for another path if the former ones failed. - I see no issues there yet, but can imagine that odd versions might - make it access odd paths which would then be denied by apparmor or - just don't exist. But that is no different than the former built-in - paths it tries, so nothing bad should happen. - The code change to the maintainer scripts has to backup the files. - If that goes wrong upgrades could be broken, but so far no tests have - shown issues. - + II: + * the qemu change has two components. + In qemu code it looks for another path if the former ones failed. + I see no issues there yet, but can imagine that odd versions might + make it access odd paths which would then be denied by apparmor or + just don't exist. But that is no different than the former built-in + paths it tries, so nothing bad should happen. + The code change to the maintainer scripts has to backup the files. + If that goes wrong upgrades could be broken, but so far no tests have + shown issues. [Other Info] - - * To really use the functionality users will need the new qemu AND the - new libvirt that are uploaded for this bug. - But it felt wrong to add versioned dependencies from qemu->libvirt - (that is the semantically correct direction) also conflicts/breaks - might cause issues in many places that want to control these. OTOH - while the fix is great for some installations the majority of users - won't care and therefore be happy if extra dependencies are not - causing any oddity on apt upgrade. Therefore no versioned - dependencies were added intentionally. + * To really use the functionality users will need the new qemu AND the + new libvirt that are uploaded for this bug. + But it felt wrong to add versioned dependencies from qemu->libvirt + (that is the semantically correct direction) also conflicts/breaks + might cause issues in many places that want to control these. OTOH + while the fix is great for some installations the majority of users + won't care and therefore be happy if extra dependencies are not + causing any oddity on apt upgrade. Therefore no versioned + dependencies were added intentionally. --- [Feature Freeze Exception] Hi, this is IMHO a just a bugfix. But since it involves some bigger changes I wanted to be on the safe side and get an ack by the release Team. Problem: - on upgrade qemu processes are left running as they represent a guest VM - later trying to add features e.g. ceph disk hot-add will need to load .so files e.g. from qemu-block-extra package - those modules can on ly be loaded from the same build, but those are gone after upgrade Solution: - If qemu fails to load from its usual paths it will now also look in /var/run/<version/ - package upgrade code will place the .so's there - things will be cleaned on reboot which is much simpler and error-proof than trying to detect which versions binaries are running - libvirt has a change to allow just reading and mapping from that path (apparmor) @Release team it would be great if you would agree to this being safe for an FFe. --- initial report --- Upgrading qemu binaries causes the on-disk versions to change, but the in-memory running instances still attempt to dynamically load a library which matches its same version. This can cause running instances to fail actions like hotplugging devices. This can be alleviated by migrating the instance to a new host or restarting the instance, however in cloud type environments there may be instances that cannot be migrated (sriov, etc) or the cloud operator does not have permission to reboot. This may be resolvable for many situations by changing the packaging to keep older versions of qemu libraries around on disk (similar to how the kernel package keeps older kernel versions around). - --- solution options (WIP) --- For a packaging solution we would need: - qemu-block-extra / qemu-system-gui binary packages would need sort of a -$buildid in the name. That could be the version string (sanitized for package name) - /usr/lib/x86_64-linux-gnu/qemu/*.so would need a -$buildid - loading of modules from qemu would need to consider $buildid when creating module names. util/module.c in module_load_one / module_load_file It already searches in multiple dirs, maybe it could insert the $buildid there - We'd need a way of detecting running versions of qemu binaries and only make them uninstallable once the binaries are all gone. I have not seen something like that in apt yet (kernel is easy in comparison as only one can be loaded at a time). ALTERNATIVES: - disable loadable module support - add an option to load all modules in advance (unlikely to be liked upstream) and not desirable for many setups using qemu (especially not as default) - add an option to load a module (e.g via QMP/HMP) which would allow an admin to decide doing so for the few setups that benefit. - that could down the road then even get a libvirt interface for easier consumption Heads up - None of the above would be SRUable --- mitigation options --- - live migrate for upgrades - prohibited by SR-IOV usage - Tech to get SR-IOV migratable is coming (e.g. via net_failover, bonding in DPDK, ...) - load the modules you need in advance - Note: lacking an explicit "load module" command makes this slightly odd for now - but using iscsi or ceph is never spontaneous, a deployment has or hasn't the setup to use those - Create a single small read-only node and attach this to each guest, that will load the driver and render you immune to the issue. While more clunky, this isn't so much different than how it would be with an explicit "load module" command. Actually the target doesn't have to exist it can fail to attach and still achieves what is needed comment #17 has an example. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1847361 Title: Upgrade of qemu binaries causes running instances not able to dynamically load modules To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1847361/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
