[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
Thanks for agreeing to my suggested approach to resolve all of this! I have already taken out the deny rules since Cosmic and we had no bad feedback that ceph (the original reason to add them) would trigger log storms. Therefore this bug is completely done. Status already reflects this, I'll remove David as assignee to completely clean up. ** Changed in: ubuntu-power-systems Assignee: David Britton (davidpbritton) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
Hi Anish, IMHO none of the messages are related to this bug. I'll reply per mail with some info. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
Christian, Seeing following errors on our end as well - fyi Create/delete of our XXL Guest vm. Sep 1 08:54:54 my-system libvirtd[4126]: 2018-09-01 15:54:54.426+: 4126: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error Sep 1 08:54:55 my-system libvirtd[4126]: 2018-09-01 15:54:55.009+: 4126: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error Sep 1 08:54:55 my-system libvirtd[4126]: 2018-09-01 15:54:55.177+: 4126: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error Sep 1 08:54:55 my-system libvirtd[4126]: 2018-09-01 15:54:55.325+: 4126: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error Sep 1 08:54:56 my-system libvirtd[4126]: 2018-09-01 15:54:56.936+: 4126: error : qemuMonitorIORead:610 : Unable to read from monitor: Connection reset by peer Sep 1 08:55:09 my-system libvirtd[4126]: 2018-09-01 15:55:09.503+: 4202: error : virProcessKillPainfully:401 : Failed to terminate process 48500 with SIGKILL: Device or resource busy Sep 1 08:55:16 my-system libvirtd[4126]: 2018-09-01 15:55:16.959+: 4201: error : virSecurityDACSetOwnershipInternal:619 : unable to set user and group to '0:0' on '/var/lib/libvirt/images/dgx2vm-labSat0847-16g0-15_dgx-kvm-image-4.0.0~180823-22df0e.1.qcow2': No such file or directory Sep 1 08:55:16 my-system libvirtd[4126]: 2018-09-01 15:55:16.959+: 4201: error : virSecurityDACSetOwnershipInternal:619 : unable to set user and group to '0:0' on '/raid/dgx-kvm/vol-dgx2vm-labSat0847-16g0-15': No such file or directory Sep 1 09:22:11 my-system libvirtd[4126]: 2018-09-01 16:22:11.098+: 4203: error : virProcessKillPainfully:401 : Failed to terminate process 64980 with SIGKILL: Device or resource busy Sep 1 09:22:11 my-system libvirtd[4126]: 2018-09-01 16:22:11.099+: 4126: error : qemuMonitorIO:719 : internal error: End of file from qemu monitor Sep 1 09:22:26 my-system libvirtd[4126]: 2018-09-01 16:22:26.110+: 51440: error : virProcessKillPainfully:401 : Failed to terminate process 64980 with SIGKILL: Device or resource busy thanks, Anish -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
TBH - I haven't taken the former comment as a call for further action. It was more of a summary how docs and output could be better. Let me answer: 1. document that --bypass-cache would help Yeah it might be nice, but then it is just such a general thing. It only affects apparmor users (not all libvirt users). It only affects /tmp wi I wonder how such a hint might look like. Checking the doc there is a Note on disk corruption for virsh restore - maybe there as another Note entry. But I'm still not all in for this. 2. on older releases "error out or warn in Libvirt when performing save in denial paths" It is not really possible to predetect and differentiate if such a denial was the reason. Looking into the future I think we might use per-guest overrides. I was thinking on that more, the fact that all other but /tmp (for the explicit deny) just work, like: $ virsh save xenial-testshutdown-0 /var/anythingbuttmp.state $ virsh restore /var/anythingbuttmp.state That annoys me a lot. I'd suggest otherwise, we keep the past as it is without modifying man pages or anything like it (after all it is no regression I can SRU and a very special case choosing /tmp only). But I want to make it better thinking forward. I thought about it again and again, and revisited the old bug that added those deny rules. I think it is time to take them out in the next release. That would mean it would generally work, and even if there is a deny it would at least be in the log. See also: - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1403648/comments/6 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1403648/comments/12 I think the old assumptions don't hold true. So for the current and stable-releases we keep it as is, to not regress anyone (with too much logs). But forward I'd drop the deny rules and then all of this (and similar things where users WANT e.g. images in /tmp) would work. Part of it would be to check (way more modern and recent) openstack that it no more has those issues and if it has as part of the fix look for something better e.g. adapt how openstack sets the ceph config to no more trigger /tmp /tmp/var access. There are also rules like owner /tmp/pulse-*/ rw, in the meantime which get trumped by the deny. TL;DR - taking out the deny and making the save/restore case of this bug no more a special case would be much better IMHO. If you are ok with that I'd create a new bug to: 1. take out the deny rules to /tmp early in Ubuntu 18.10 2. do an analysis with recent openstack+ceph if they still trigger access there So are you ok with that approach? P.S. If you really really (...really*) want/need a man page entry for this special case we could work something out, but I think that would not qualify as an SRU [1] so thinking forward is much better anyway. [1]: https://wiki.ubuntu.com/StableReleaseUpdates -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
>From comment #22: "Distro team, Is there any update based on the previous update we posted. ?" I'm afraid this bug report is now a little complex and involved, and it is now marked as "Fix Released" in Launchpad. Could you please be specific about the comment or question that you require more information about? Alternatively, it may be best to raise a new bug report. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
** Changed in: libvirt (Ubuntu) Assignee: ChristianEhrhardt (paelzer) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
Thanks for the full dmesg. It seems to me that: "unable to set AppArmor profile 'libvirt-81b387d9-1dfc-4f55-8b98-0318f1f94442'" means there is an issue in loading the profile after your change. That matches: audit: type=1400 audit(1519028363.683:12417): apparmor="DENIED" operation="change_profile" info="label not found" error=-2 profile="/usr/sbin/libvirtd" name="libvirt-81b387d9-1dfc-4f55-8b98-0318f1f94442" pid=12949 comm="libvirtd" It is not getting to the actual restore, it is failing when spawning the guest to to the changes in the apparmor profile. I tried to check what you hit: $ virsh save bionic-test --file /var/tmp/bionic-test.save --verbose Guest is shut-off and I have -rw--- 1 root root 527808329 Feb 19 12:34 /var/tmp/bionic-test.save The restore hits the (silent) denial we discussed. #deny /tmp/{,**} r, #deny /var/tmp/{,**} r, Changed the two lines above to a comment. Then restored again, just worked: $ virsh restore /var/tmp/bionic-test.save Domain restored from /var/tmp/bionic-test.save To quote jdstrand from bug 1403648: "We should not allow access to /tmp and /var/tmp as that breaks application isolation." That said we are in the following situation: 1. /tmp and /var/tmp are not allowed to be read (apparmor default for app isolation) 2. read denies there are silenced via explicit denies in /etc/apparmor.d/abstractions/libvirt-qemu 3. I see your point: 3.1 on save libvirt writes to that place (libvirt is allowed to do so, while qemu is not) 3.2 on restore qemu wants to read it and is denied. And you wonder about the asymetric behavior of 3.1 and 3.2. I agree that it is somewhat unexpected, but wonder what would be better 1. We could also deny /var /tmp for the lbivirt daemon (which intentionally has a rather lenient apparmor profile). Then already on the save people would be denied, maybe for a new release - but not as an SRU to not break people relying on that access working. 2. And on the new release we already have the --bypass-cache fixes you referred to to get the restore working there as a workaround - so the benefit of preventing libvirt to access there isn't too big either. So forbidding the access on "save" for libvirt there would make that useless. I'm unsure how to continue. To better brain-storm with you on how to proceed do you have a clear preferred solution (other than the already included bypass-cache fixes) or is it just "not nice in general" that the denial should be consistent for save/restore? Separate to the discussion above: To find how your modified apparmor profile breaks your guest start you could share it - as I mentioned it worked for me right away (no need to restart libvirt after changing btw, the one we change it loaded on guest load). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
** Changed in: ubuntu-power-systems Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
This bug was fixed in the package libvirt - 4.0.0-1ubuntu1 --- libvirt (4.0.0-1ubuntu1) bionic; urgency=medium * Merged with Debian unstable (4.0) This closes several bugs: - Error generating apparmor profile when hostname contains spaces (LP: #77) - qemu 2.10 locks files, libvirt shared now sets share-rw=on (LP: #1716028) - libvirt usb passthrough throws apparmor denials related to /run/udev/data/+usb (LP: #1727311) - AppArmor denies access to /sys/block/*/queue/max_segments (LP: #1729626) - iohelper improvements to let bypass-cache work without opening up the apparmor isolation (LP: #1719579) - nodeinfo on s390x to contain more CPU info (LP: #1733688) - Upgrade libvirt >= 4.0 (LP: #1745934) * Remaining changes: - Disable libssh2 support (universe dependency) - Disable firewalld support (universe dependency) - Disable selinux - Set qemu-group to kvm (for compat with older ubuntu) - Additional apport package-hook - Modifications to adapt for our delayed switch away from libvirt-bin (can be dropped >18.04). + d/p/ubuntu/libvirtd-service-add-bin-alias.patch: systemd: define alias to old service name so that old references work + d/p/ubuntu/libvirtd-init-add-bin-alias.patch: sysv init: define alias to old service name so that old references work + d/control: transitional package with the old name and maintainer scripts to handle the transition - Backwards compatible handling of group rename (can be dropped >18.04). - config details and autostart of default bridged network. Creating that is now the default in general, yet our solution provides the following on top as of today: + autostart the default network by default + do not autostart if subnet is already taken (e.g. in guests). - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is the group based access to libvirt functions as it was used in Ubuntu for quite long. + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests due to the group access change. - ubuntu/parallel-shutdown.patch: set parallel shutdown by default. - d/p/ubuntu/enable-kvm-spice.patch: compat with older Ubuntu qemu/kvm which provided a separate kvm-spice. - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: this change was split. The section that adapts the path of the emulator to the Debian/Ubuntu packaging is kept. - d/p/ubuntu/ubuntu-libxl-Fix-up-VRAM-to-minimum-requirements.patch: auto set VRAM to minimum requirements - d/p/ubuntu/xen-default-uri.patch: set default URI on xen hosts - Add libxl log directory - libvirt-uri.sh: Automatically switch default libvirt URI for users on Xen dom0 via user profile (was missing on changelogs before) - d/p/ubuntu/apibuild-skip-libvirt-common.h: drop libvirt-common.h from included_files to avoid build failures due to duplicate definitions. - Update README.Debian with Ubuntu changes - Convert libvirt0, libnss_libvirt and libvirt-dev to multi-arch. - Enable some additional features on ppc64el and s390x (for arch parity) + systemtap, zfs, numa and numad on s390x. + systemtap on ppc64el. - fix conffile upgrade handling to avoid obsolete files and inactive duplicates (LP 1694159) - d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making vmlinuz available and accessible (Debian bug 848314) - d/test/smoke-lxc workaround for debbug 848317/867379 - d/t/control, d/t/smoke-lxc: fix up lxc smoke test (Debian bug 848317) - Add dnsmasq configuration to work with system wide dnsmasq (drop >18.04, no more UCA onto Xenial then which has global dnsmasq by default). - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx - conffile handling of files dropped in 3.5 (can be dropped >18.04) + /etc/init.d/virtlockd was sysv init only + /etc/apparmor.d/local/usr.sbin.libvirtd and /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper are now generated by dh_apparmor as needed - Reworked apparmor Delta, especially the more complex delta is dropped now, also our former delta is now split into logical pieces, has improved comments and is part of a continuous upstreaming effort. Listing related remaining changes: + d/p/0001-apparmor-Allow-pygrub-to-run-on-Debian-Ubuntu.patch: apparmor: Allow pygrub to run on Debian/Ubuntu + d/p/0003-apparmor-libvirt-qemu-Allow-read-access-to-overcommi.patch: apparmor, libvirt-qemu: Allow read access to overcommit_memory + d/p/0007-apparmor-libvirt-qemu-Allow-owner-read-access-to-PRO.patch: apparmor, libvirt-qemu: Allow owner read access to @{PROC}/*/auxv + d/p/0017-apparmor-virt-aa-helper-Allow-access-to-tmp-director.patch: apparmor, virt-aa-helper: Allo
[Bug 1719579] Re: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save
** Summary changed: - [Ubuntu 16.04.2] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save + [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1719579 Title: [Ubuntu 18.04] [libvirt] virsh restore fails from state file saved in /var/tmp folder using virsh save To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1719579/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs