Hi,
I encountered the same problem.
It looks like libvirt denies write access to guest base image by
modifying the guest related apparmor profile once the guest runs on a
snapshot referring to this base image.
Here's what i tried:
Guest runs on base image:
root@host/storage/vm# virsh domblklist test
Target Source
--------------------------------
vda /storage/vm/test.img
Generated aa profile not contains deny rule for base image:
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/test.log" w,
"/var/lib/libvirt/qemu/domain-test/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-8-test/*" rw,
"/var/run/libvirt/**/test.pid" rwk,
"/run/libvirt/**/test.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.test" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.test" rw,
"/storage/vm/test.img" rwk, <---------------------------------------
"/dev/vhost-net" rw,
"/var/lib/libvirt/qemu/domain-8-test/{,**}" rwk,
"/var/lib/libvirt/qemu/channel/target/domain-8-test/{,**}" rwk,
"/var/lib/libvirt/qemu/domain-8-test/master-key.aes" rwk,
"/dev/net/tun" rwk,
After generating the snapshot, there's still no deny rule (but we also not
tried to merge back to base image yet):
root@host/storage/vm# virsh snapshot-create-as --domain test --name
backup_overlay --no-metadata --atomic --disk-only --diskspec
vda,snapshot=external
Domain snapshot backup_overlay created
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/test.log" w,
"/var/lib/libvirt/qemu/domain-test/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-8-test/*" rw,
"/var/run/libvirt/**/test.pid" rwk,
"/run/libvirt/**/test.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.test" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.test" rw,
"/storage/vm/test.img" rwk, <---------------------------------------
"/dev/pts/2" rw,
"/dev/pts/2" rw,
"/var/lib/libvirt/qemu/channel/target/domain-8-test/org.qemu.guest_agent.0" rw,
"/dev/vhost-net" rw,
"/storage/vm/test.backup_overlay" rwk,
When shutting down and starting guest again, host specific aa profile gets
regenerated.
Now the deny rule is present. Note that guest runs on snapshot:
root@host/storage/vm# virsh domblklist test
Target Source
-------------------------------------------
vda /storage/vm/test.backup_overlay
root@host/storage/vm# virsh shutdown test
Domain test is being shutdown
root@host/storage/vm# virsh start test
Domain test started
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
...
"/storage/vm/test.img" rk, <-------------------------------------------
# don't audit writes to readonly files
deny "/storage/vm/test.img" w, <----------------------------------------
"/dev/vhost-net" rw,
...
And it seems like this rule prevents us from merging snapshot back to base
image:
root@host/storage/vm# virsh blockcommit test vda --active --pivoterror:
internal error: unable to execute QEMU command 'block-commit': Could not reopen
file: Permission denied
As soon as the guest aa profile gets disabled, blockcommit works as expected:
root@host/storage/vm# aa-disable /etc/apparmor.d/libvirt/libvirt-<uuid>
Disabling /etc/apparmor.d/libvirt/libvirt-<uuid>.
root@host/storage/vm# virsh blockcommit test vda --active --pivot
Successfully pivoted
After merge, stop and start guest, the deny rule is gone again:
root@host/storage/vm# virsh shutdown test
Domain test is being shutdown
root@host/storage/vm# virsh start test
Domain test started
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
...
"/storage/vm/test.img" rwk,
"/dev/vhost-net" rw,
...
root@host/storage/vm# virsh domblklist test
Target Source
--------------------------------
vda /storage/vm/test.img
As stated above, the deny rule is not created immediately when creating the
snapshot
if guest not gets stopped/started. But it gets added as soon as we try to merge
back
to the base image:
root@host/storage/vm# virsh snapshot-create-as --domain test --name
backup_overlay --no-metadata --atomic --disk-only --diskspec
vda,snapshot=external
Domain snapshot backup_overlay created
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
...
"/storage/vm/test.img" rwk,
...
root@host/storage/vm# virsh blockcommit test vda --active --pivoterror:
internal error: unable to execute QEMU command 'block-commit': Could not reopen
file: Permission denied
root@host/storage/vm# cat /etc/apparmor.d/libvirt/libvirt-<uuid>.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
...
"/storage/vm/test.img" rk,
# don't audit writes to readonly files
deny "/storage/vm/test.img" w,
<--------------------------------------------
"/dev/pts/2" rw,
"/dev/pts/2" rw,
"/var/lib/libvirt/qemu/channel/target/domain-10-test/org.qemu.guest_agent.0" rw,
"/dev/vhost-net" rw,
"/storage/vm/test.img" rwk,
I fear this is not solvable by just fixing some apparmor base profiles because
libvirt creates and modifies aa profiles on the fly via virt-aa-helper.
Would be good to know if this procedure fails as well on other distros if
apparmor is enabled, then it's probably a general problem with libvirt and
apparmor.
Thanks,
Kind Regards,
Robert
