** Description changed: - By debugging various bugs I've found several entries missing in the - libvirt-qemu apparmor abstraction. + [Impact] + + * Attaching of virtual functions (SR-IOV) not possible without manually + modification of apparmor rules + * libvirt/qemu guest logfiles can not state the program that sent the + shutdown signal + * on ppc64el when starting on SMT enabled it failes (expected) but does + not present the hint about SMT being the reasons that was added in + >=zesty + * various apparmor denials in the log hide other issues + * Fix by opening up apparmor rules as needed for these use-cases + + [Test Case] + + * The rules covere a set of use cases, outlining them all here. Some only + apply to >=Zesty, so I mark each subtest with release names. + * All start with "Spawn a kvm guest e.g. via uvtool-libvirt" + * A) >=Xenial - shutdown message + - A1 - start the guest + - A2 - stop the guest + - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log + - A-check: without the fix the log ends with <unknown process> + With the fix it shall contain the name of the process + stopping it instead of. So libvirt shutdown will show + libvirtd, kill from bash shows bash, ...) + * B) >=Xenial + - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your + system, one example is outlined here: + # Prep VFs and attach + $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs + $ sudo modprobe vfio-pci + $ lspci -n -s 0005:01:01.3 + 0005:01:01.3 0200: 10df:e228 (rev 10) + $ cat VF-5.1.1.3.xml + <hostdev mode='subsystem' type='pci' managed='yes'> + <source> + <address domain='0x0005' bus='0x01' slot='0x01' + function='0x3'/> + </source> + </hostdev> + - B2 until bug 1679704 is fixed you need to manually increase the + locked memory limit of the qemu process before going on + - B3 attach the device + $ virsh attach-device z-test VF-5.1.1.3.xml + - B-check: without the fix it will fail for a set of apparmor denials + with the fix the attaching will succeed + * C) >=Zesty + - C1 - on a ppc64el system start and stop a guest + - C-check: without the fix apparmor will hold many denies for ppc64_cpu + and/or accessing /sys/devices/system/cpu/... + With the fix not only the denies are gone, but also if you + run on a system with SMT enabled (kvm will fail there) it + will report to you in the log about "Error: You must disable + SMT ..." + + [Regression Potential] + + * We are only further "opening up" the current apparmor profiles. So we + should not end up affecting existing use cases by new blocks. + * There is a very slight chance that due to accesses being allowed follow + on actions are triggered which people are not used to like "the VF hot + attach actually working". So if one relied on it not working it will be + different for them. + * Further I made some brainstorming on the potential effects but the only + other one that came to my mind being a regression of some sort is that + it now correctly states what send the shutdown signal. So when qemu did + not end by itself (guest stops) but is stopped (shutdown) the old + message was: + terminating on signal 15 from pid 22938 (<unknown process>) + And now will be like: + terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd) + For the hopefully super unlikely case that someone e.g. grepped for + unknown here it might be a regression. + And making that message more useful (along with getting rid of the + associated aa deny is part of the intended fix). + + + [Other Info] + + * Tested manually and via regression tests in advance on bileto tickets. + So far almost all green, unrelated issues in some dep8 due to what + seems transient network timeouts - if showing up in proposed as well we + will likely have to retrigger the tests until good. Latest ticket for + this is https://bileto.ubuntu.com/#/ticket/2738 + * Upload will be directly, since in some of the releases I will need + dpdk-genversion -v which bileto does not support (and it doesn't + support publishing selective releases) yet I wanted to show the extra + tests and checks made and therefore using bileto was beneficial even if + eventually uploading "old style" + + + + ----- + + + By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions /libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like: - # allow guest access to the generic base vfio interface (LP: #1678322) - /dev/vfio/vfio rw, + # allow guest access to the generic base vfio interface (LP: #1678322) + /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. - #2 + #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 - #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)"
** Description changed: [Impact] - * Attaching of virtual functions (SR-IOV) not possible without manually - modification of apparmor rules - * libvirt/qemu guest logfiles can not state the program that sent the - shutdown signal - * on ppc64el when starting on SMT enabled it failes (expected) but does - not present the hint about SMT being the reasons that was added in - >=zesty - * various apparmor denials in the log hide other issues - * Fix by opening up apparmor rules as needed for these use-cases + * Attaching of virtual functions (SR-IOV) not possible without manually + modification of apparmor rules + * libvirt/qemu guest logfiles can not state the program that sent the + shutdown signal + * on ppc64el when starting on SMT enabled it failes (expected) but does + not present the hint about SMT being the reasons that was added in + >=zesty + * various apparmor denials in the log hide other issues + * Fix by opening up apparmor rules as needed for these use-cases [Test Case] - * The rules covere a set of use cases, outlining them all here. Some only - apply to >=Zesty, so I mark each subtest with release names. - * All start with "Spawn a kvm guest e.g. via uvtool-libvirt" - * A) >=Xenial - shutdown message - - A1 - start the guest - - A2 - stop the guest - - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log - - A-check: without the fix the log ends with <unknown process> - With the fix it shall contain the name of the process - stopping it instead of. So libvirt shutdown will show - libvirtd, kill from bash shows bash, ...) - * B) >=Xenial - - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your - system, one example is outlined here: - # Prep VFs and attach - $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs - $ sudo modprobe vfio-pci - $ lspci -n -s 0005:01:01.3 - 0005:01:01.3 0200: 10df:e228 (rev 10) - $ cat VF-5.1.1.3.xml - <hostdev mode='subsystem' type='pci' managed='yes'> - <source> - <address domain='0x0005' bus='0x01' slot='0x01' - function='0x3'/> - </source> - </hostdev> - - B2 until bug 1679704 is fixed you need to manually increase the - locked memory limit of the qemu process before going on - - B3 attach the device - $ virsh attach-device z-test VF-5.1.1.3.xml - - B-check: without the fix it will fail for a set of apparmor denials - with the fix the attaching will succeed - * C) >=Zesty - - C1 - on a ppc64el system start and stop a guest - - C-check: without the fix apparmor will hold many denies for ppc64_cpu - and/or accessing /sys/devices/system/cpu/... - With the fix not only the denies are gone, but also if you - run on a system with SMT enabled (kvm will fail there) it - will report to you in the log about "Error: You must disable - SMT ..." + * The rules covere a set of use cases, outlining them all here. Some only + apply to >=Zesty, so I mark each subtest with release names. + * All start with "Spawn a kvm guest e.g. via uvtool-libvirt" + * A) >=Xenial - shutdown message + - A1 - start the guest + - A2 - stop the guest + - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log + - A-check: without the fix the log ends with <unknown process> + With the fix it shall contain the name of the process + stopping it instead of. So libvirt shutdown will show + libvirtd, kill from bash shows bash, ...) + * B) >=Xenial + - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your + system, one example is outlined here: + # Prep VFs and attach + $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs + $ sudo modprobe vfio-pci + $ lspci -n -s 0005:01:01.3 + 0005:01:01.3 0200: 10df:e228 (rev 10) + $ cat VF-5.1.1.3.xml + <hostdev mode='subsystem' type='pci' managed='yes'> + <source> + <address domain='0x0005' bus='0x01' slot='0x01' + function='0x3'/> + </source> + </hostdev> + - B2 until bug 1679704 is fixed you need to manually increase the + locked memory limit of the qemu process before going on + - B3 attach the device + $ virsh attach-device z-test VF-5.1.1.3.xml + - B-check: without the fix it will fail for a set of apparmor denials + with the fix the attaching will succeed + * C) >=Zesty + - C1 - on a ppc64el system start and stop a guest + - C-check: without the fix apparmor will hold many denies for ppc64_cpu + and/or accessing /sys/devices/system/cpu/... + With the fix not only the denies are gone, but also if you + run on a system with SMT enabled (kvm will fail there) it + will report to you in the log about "Error: You must disable + SMT ..." [Regression Potential] - * We are only further "opening up" the current apparmor profiles. So we - should not end up affecting existing use cases by new blocks. - * There is a very slight chance that due to accesses being allowed follow - on actions are triggered which people are not used to like "the VF hot - attach actually working". So if one relied on it not working it will be - different for them. - * Further I made some brainstorming on the potential effects but the only - other one that came to my mind being a regression of some sort is that - it now correctly states what send the shutdown signal. So when qemu did - not end by itself (guest stops) but is stopped (shutdown) the old - message was: - terminating on signal 15 from pid 22938 (<unknown process>) - And now will be like: - terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd) - For the hopefully super unlikely case that someone e.g. grepped for - unknown here it might be a regression. - And making that message more useful (along with getting rid of the - associated aa deny is part of the intended fix). - + * We are only further "opening up" the current apparmor profiles. So we + should not end up affecting existing use cases by new blocks. + * There is a very slight chance that due to accesses being allowed follow + on actions are triggered which people are not used to like "the VF hot + attach actually working". So if one relied on it not working it will be + different for them. + * Further I made some brainstorming on the potential effects but the only + other one that came to my mind being a regression of some sort is that + it now correctly states what send the shutdown signal. So when qemu did + not end by itself (guest stops) but is stopped (shutdown) the old + message was: + terminating on signal 15 from pid 22938 (<unknown process>) + And now will be like: + terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd) + For the hopefully super unlikely case that someone e.g. grepped for + unknown here it might be a regression. + And making that message more useful (along with getting rid of the + associated aa deny is part of the intended fix). [Other Info] - - * Tested manually and via regression tests in advance on bileto tickets. - So far almost all green, unrelated issues in some dep8 due to what - seems transient network timeouts - if showing up in proposed as well we - will likely have to retrigger the tests until good. Latest ticket for - this is https://bileto.ubuntu.com/#/ticket/2738 - * Upload will be directly, since in some of the releases I will need - dpdk-genversion -v which bileto does not support (and it doesn't - support publishing selective releases) yet I wanted to show the extra - tests and checks made and therefore using bileto was beneficial even if - eventually uploading "old style" - + * Tested manually and via regression tests in advance on bileto tickets. + So all are green there now, there were two unrelated in some dep8 tests + due to what seems transient network timeouts (worked on retry) - if + showing up in proposed as well we will likely have to retrigger the + tests until good. Latest ticket for these to look at sniff builds: + => https://bileto.ubuntu.com/#/ticket/2738 + * Upload will be directly, since in some of the releases I will need + dpdk-genversion -v which bileto does not support (and it doesn't + support publishing selective releases) yet I wanted to show the extra + tests and checks made and therefore using bileto was beneficial even if + eventually uploading "old style" ----- - - By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. + By debugging various bugs I've found several entries missing in the + libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions /libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like: # allow guest access to the generic base vfio interface (LP: #1678322) /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)" -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1680384 Title: libvirt-qemu apparmor profiles misses several important entries To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1680384/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
