[Kernel-packages] [Bug 2056441] [NEW] [UBUNTU 20.04] Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu
Public bug reported:
---Problem Description (by far...@us.ibm.com) ---
Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04
Contact Information = Eric Farman
---uname output---
Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022
s390x s390x s390x GNU/Linux
---Additional Hardware Info---
ECKD DASD, connected as a mediated device for KVM device passthrough
Machine Type = IBM z14 (3906) LPAR
---Debugger---
A debugger is not configured
---Steps to Reproduce---
Attempting to spawn a guest with a vfio-ccw hostdev device fails with an
AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU
attempts to open for the device. The failure also occurs when trying to hotplug
such a device, which I'll use in these steps to keep the XML simple:
eric@kvmhost:~# chzdev -ea ca8b
eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind
eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 >
/sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create
eric@kvmhost:~# cat hostdev.xml
eric@kvmhost:~# virsh attach-device guest hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@kvmhost:~# dmesg | grep DENIED
[ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED"
operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a"
name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
While the failure occurs with the pimpampom file for the subchannel,
there are two others that QEMU would attempt to open after this:
eric:qemu$ git grep -B 2 -pn fopen hw/s390x/
hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2585-fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids",
hw/s390x/css.c-2586- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2587:fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2619-fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom",
hw/s390x/css.c-2620- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2621:fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t
*type,
--
hw/s390x/css.c-2649-fid_path =
g_strdup_printf("/sys/devices/css%x/chp0.%02x/type",
hw/s390x/css.c-2650- dev_id->cssid, chpid);
hw/s390x/css.c:2651:fd = fopen(fid_path, "r");
The first two directories are links to the third, so I made the
following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu which
Works For Me:
eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
/sys/devices/css0/** r,
This is of course a very broad brush, so perhaps there's a better
deterministic way to the files in question for the subchannel(s) that
are requested. (I apologize if that deterministic logic is tied up in
the "hostdev networks" bug I see here:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398)
For what it's worth, those files are not ones that remain open once the
device is connected to the guest:
eric@kvmhost:~# cat
/etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/guest.log" w,
"/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-3-guest/*" rw,
"/run/libvirt/**/guest.pid" rwk,
"/run/libvirt/**/*.tunnelmigrate.dest.guest" rw,
"/dev/dasdb" rwk,
"/dev/pts/2" rw,
"/dev/vhost-net" rw,
"/dev/vfio/2" rwk,
(The passed through DASD device is /dev/vfio/2 in the above list, not
/dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.)
=== Comment: - 2024-03-06 13:30:45
=
Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the
manual sysfs changes described in the initial comment, and the more convenient
driverctl and mdevctl tooling.
=== Comment: - 2024-03-06 13:34:28
=
eric@host:~# virsh attach-device guest_3c4c hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@host:~# dmesg | grep 0165
[ 127.558194] vfio_ccw
[Kernel-packages] [Bug 2056441] [NEW] [UBUNTU 20.04] Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu
You have been subscribed to a public bug:
---Problem Description (by far...@us.ibm.com) ---
Cannot use vfio-ccw dasd passthrough for KVM guests under Ubuntu 20.04/22.04
Contact Information = Eric Farman
---uname output---
Linux m34mkvmt5 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 21:05:01 UTC 2022
s390x s390x s390x GNU/Linux
---Additional Hardware Info---
ECKD DASD, connected as a mediated device for KVM device passthrough
Machine Type = IBM z14 (3906) LPAR
---Debugger---
A debugger is not configured
---Steps to Reproduce---
Attempting to spawn a guest with a vfio-ccw hostdev device fails with an
AppArmor policy restriction on both 20.04 and 22.04, for files that QEMU
attempts to open for the device. The failure also occurs when trying to hotplug
such a device, which I'll use in these steps to keep the XML simple:
eric@kvmhost:~# chzdev -ea ca8b
eric@kvmhost:~# echo 0.0.ca8b > /sys/bus/ccw/drivers/dasd-eckd/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/io_subchannel/unbind
eric@kvmhost:~# echo 0.0.0b16 > /sys/bus/css/drivers/vfio_ccw/bind
eric@kvmhost:~# echo 11f2d2bc-4083-431d-a023-eff72715c4f0 >
/sys/bus/css/devices/0.0.0b16/mdev_supported_types/vfio_ccw-io/create
eric@kvmhost:~# cat hostdev.xml
eric@kvmhost:~# virsh attach-device guest hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@kvmhost:~# dmesg | grep DENIED
[ 5949.232089] audit: type=1400 audit(1670350246.709:22): apparmor="DENIED"
operation="open" profile="libvirt-0e995f6d-f85e-4ffe-b612-e07bfd62116a"
name="/sys/devices/css0/0.0.0b16/pimpampom" pid=1497 comm="qemu-system-s39"
requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
While the failure occurs with the pimpampom file for the subchannel,
there are two others that QEMU would attempt to open after this:
eric:qemu$ git grep -B 2 -pn fopen hw/s390x/
hw/s390x/css.c=2577=static int css_sch_get_chpids(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2585-fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/chpids",
hw/s390x/css.c-2586- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2587:fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2612=static int css_sch_get_path_masks(SubchDev *sch, CssDevId
*dev_id)
--
hw/s390x/css.c-2619-fid_path =
g_strdup_printf("/sys/bus/css/devices/%x.%x.%04x/pimpampom",
hw/s390x/css.c-2620- dev_id->cssid, dev_id->ssid,
dev_id->devid);
hw/s390x/css.c:2621:fd = fopen(fid_path, "r");
--
hw/s390x/css.c=2643=static int css_sch_get_chpid_type(uint8_t chpid, uint32_t
*type,
--
hw/s390x/css.c-2649-fid_path =
g_strdup_printf("/sys/devices/css%x/chp0.%02x/type",
hw/s390x/css.c-2650- dev_id->cssid, chpid);
hw/s390x/css.c:2651:fd = fopen(fid_path, "r");
The first two directories are links to the third, so I made the
following entry in /etc/apparmor.d/local/abstractions/libvirt-qemu which
Works For Me:
eric@kvmhost:~# cat /etc/apparmor.d/local/abstractions/libvirt-qemu
/sys/devices/css0/** r,
This is of course a very broad brush, so perhaps there's a better
deterministic way to the files in question for the subchannel(s) that
are requested. (I apologize if that deterministic logic is tied up in
the "hostdev networks" bug I see here:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398)
For what it's worth, those files are not ones that remain open once the
device is connected to the guest:
eric@kvmhost:~# cat
/etc/apparmor.d/libvirt/libvirt-43b44ca9-d1c2-46f7-a686-2329a5a1d425.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/guest.log" w,
"/var/lib/libvirt/qemu/domain-guest/monitor.sock" rw,
"/var/lib/libvirt/qemu/domain-3-guest/*" rw,
"/run/libvirt/**/guest.pid" rwk,
"/run/libvirt/**/*.tunnelmigrate.dest.guest" rw,
"/dev/dasdb" rwk,
"/dev/pts/2" rw,
"/dev/vhost-net" rw,
"/dev/vfio/2" rwk,
(The passed through DASD device is /dev/vfio/2 in the above list, not
/dev/dasdb. The latter is the guest rootfs, connected via virtio-blk.)
=== Comment: - 2024-03-06 13:30:45
=
Verified that this still misbehaves with 20.04.6 and 22.04.4. Both with the
manual sysfs changes described in the initial comment, and the more convenient
driverctl and mdevctl tooling.
=== Comment: - 2024-03-06 13:34:28
=
eric@host:~# virsh attach-device guest_3c4c hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add':
s390_ccw_realize: Failed to build initial schib: Invalid argument
eric@host:~# dmesg | grep 0165
[
