Bug#1061725: Info received (Bug#1061725: libvirt-daemon: Deleting external snapshot for non-running system VM fails with Permission Denied)

[Martin Pitt]

I can't make head or tail of this. aa-complain still enforces deny
rules, there is no (discoverable) way to log deny rules, and

  grep -r deny /etc/apparmor.d | grep virt | grep -v /sys | grep -v /dev

doesn't show anything which would apply to /var/lib/libvirt/.

`aa-disable 
/etc/apparmor.d/libvirt/libvirt-a5aa3a67-6967-43a5-9d61-b0b380bd14e6` also
doesn't work because it references a non-existing
libvirt/libvirt-a5aa3a67-6967-43a5-9d61-b0b380bd14e6.files.

The only thing that works is

  aa-disable libvirtd
  systemctl restart libvirtd

(That requires apparmor-utils)

After that, the snapshot-delete command works.

I don't know what else I could try here to debug this properly, so a hint from
someone AppArmor-savvy would be much appreciated.

Bug#1061725: libvirt-daemon: Deleting external snapshot for non-running system VM fails with Permission Denied

[Martin Pitt]

Control: retitle -1 libvirt-daemon: Deleting external snapshot for non-running 
system VM fails with AppArmor

when stracing libvirt, this is what happens:

6557  openat(AT_FDCWD, "/var/lib/libvirt/images/test2.qcow2", O_RDWR|O_CLOEXEC) 
= -1 EACCES (Permission denied)
6557  sendmsg(13, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="{\"id\": 
\"libvirt-443\", \"error\": {\"class\": \"GenericError\", \"desc\
": \"Could not open '/var/lib/libvirt/images/test2.qcow2': Permission 
denied\"}}\r\n", iov_len=142}], msg_iovlen=1, msg_controllen=0, msg_flags
=0}, 0 

and the most recent geteuid() call responded with "0". So it actually *does*
smell like an AppArmor issue, even though it's weird that it would work for a
running VM then. Running `aa-teardown` before the creation of the VM doesn't
work, nor does "aa-complain libvirtd". But after `dpkg -P apparmor; reboot` it
does work.

So AppArmor breaks this without even logging about it, i.e. some "deny" rule. I
don't know how to make AA log deny rules -- the profile has tons of them
(albeit to /proc, /dev/, etc.), and it's further complicated by the dynamic
profile creation through virt-aa-helper.

As this works in current Ubuntu, it's perhaps worth looking at
https://patches.ubuntu.com/libv/libvirt/libvirt_9.6.0-1ubuntu2.patch
The most plausible one may be 
debian/patches/ubuntu-aa/0031-virt-aa-helper-Ask-for-no-deny-rule-for-readonly-dis.patch
but that requires rebuilding libvirt. But also, that patch is from 2017, and
it's still broken in Ubuntu 22.04.

Bug#1061725: libvirt-daemon: Deleting external snapshot for non-running system VM fails with Permission Denied

[Martin Pitt]

Package: libvirt-daemon
Version: 10.0.0-1

When creating a trivial VM and doing an external snapshot if the VM is *not* 
running,
deleting the snapshot fails. As root:

  qemu-img create -f qcow2 /var/lib/libvirt/images/test1.qcow2 10G
  virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 
--wait 0 --name test1 --disk target=vda,path=/var/lib/libvirt/images/test1.qcow2
  virsh destroy test1
  virsh snapshot-create-as --domain test1 --name snap1 --disk-only --diskspec 
vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap1
  virsh snapshot-delete test1 snap1

The last command fails with

| error: Failed to delete snapshot snap1
| error: internal error: unable to execute QEMU command 'block-commit': Could 
not open '/var/lib/libvirt/images/test1.qcow2': Permission denied

The error message may be a bit misleading -- this is the *image* file, which
has wide-open libvirt-qemu:libvirt-qemu 666 permissions. "test1-snap1" in the
same directory is much more restrictive root:root 644; but even trying to
chown/chmod that doesn't unbreak this. So perhaps it's trying to do something
funky to the actual image file after all.

It also happens with the automatically created disk image, which will then be
IDE "hda", not virtio "vda":

  virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 
--wait 0 --name test2
  virsh destroy test2
  virsh snapshot-create-as --domain test2 --name snap1 --disk-only --diskspec 
hda,snapshot=external,file=/var/lib/libvirt/images/test2-snap1
  virsh snapshot-delete test2 snap1

It works when doing a snapshot from a *running* VM, either disk-only or with 
memory:

  virsh start test1
  virsh snapshot-create-as --domain test1 --name snap2 --disk-only --diskspec 
vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap2
  virsh snapshot-create-as --domain test1 --name snap3 --memspec 
file=/var/lib/libvirt/qemu/snapshot/test1-snap3-memory  --diskspec 
vda,snapshot=external,file=/var/lib/libvirt/images/test1-snap3

Then both snap2 and snap3 can be deleted. But still not snap1, so the running
state matters at the time of snapshot creation, not deletion.

This also happens with libvirt 9.0.0-4 in Debian stable and libvirt
8.0.0-1ubuntu7.8 in Ubuntu 22.04 LTS, but curiously not with libvirt
9.6.0-1ubuntu1 in Ubuntu 23.10. It also works fine in Fedora, CentOS/RHEL 8/9,
and Arch Linux, so this is somehow specific to Debian.

I tried `aa-teardown` just in case it's apparmor, but that doesn't seem to
influence it.

There is no useful/relevant journal message about this other than the
"Permission denied" line that's already on stderr.

I also tried this as user:

  virt-install --memory 50 --pxe --virt-type qemu --os-variant alpinelinux3.8 
--wait 0 --name test1
  virsh destroy test1
  virsh snapshot-create-as --domain test1 --name snap1 --disk-only --diskspec 
hda,snapshot=external,file=$HOME/.local/share/libvirt/images/test1-snap1
  virsh snapshot-delete test1 snap1

This works as well. So this is specific to qemu:///system, session works fine.

Martin