Used the Kernel from Proposed:
apt-cache policy linux-image-4.15.0-42-generic
linux-image-4.15.0-42-generic:
Installed: 4.15.0-42.45
Candidate: 4.15.0-42.45
Libvirt/Qemu from PPA [1]
Having one device assigned to my LPAR atm:
$ ll /sys/bus/ap/devices/
total 0
drwxr-xr-x 2 root root 0 Nov 23 03:29 ./
drwxr-xr-x 4 root root 0 Nov 23 03:29 ../
lrwxrwxrwx 1 root root 0 Nov 23 03:29 00.0016 ->
../../../devices/ap/card00/00.0016/
lrwxrwxrwx 1 root root 0 Nov 23 03:29 card00 -> ../../../devices/ap/card00/
# mask out the adapters/queues of your choice that you want to virtualize
# In my case i have card 0 queue 16 (hex 16 dec 22 to match HMC config)
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 5
00.0016 CEX5C CCA-Coproc online 5
# so lets assign that to vfio-ap instead of zcrypt use
# Adapter
$ cat /sys/bus/ap/apmask
0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
$ echo -0x0 | sudo tee /sys/bus/ap/apmask
$ cat /sys/bus/ap/apmask
0x7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
# Domain
$ cat /sys/bus/ap/aqmask
0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
$ echo -0x16 | sudo tee /sys/bus/ap/aqmask
$ cat /sys/bus/ap/aqmask
0xfffffdffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
$ sudo modprobe vfio_ap
$ dmesg | tail
...
[272006.492864] vfio_ap matrix: MDEV: Registered
# create a new MDEV
$ uuid=$(uuidgen)
$ echo ${uuid} | sudo tee
/sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create
in Dmesg:
[272197.818811] iommu: Adding device 24f952b3-03d1-4df2-9967-0d5f7d63d5f2 to
group 0
[272197.818815] vfio_mdev 24f952b3-03d1-4df2-9967-0d5f7d63d5f2: MDEV: group_id
= 0
# Assign adapter 0 to vfio-ap
echo +0x0 > /sys/devices/vfio_ap/matrix/${uuid}/assign_adapter
# Assign domain 16 (22) to vfio-ap
$ echo +0x16 | sudo tee /sys/devices/vfio_ap/matrix/${uuid}/assign_domain
$ echo +0x16 | sudo tee
/sys/devices/vfio_ap/matrix/${uuid}/assign_control_domain
Check the matrix you have set up
$ cat /sys/devices/vfio_ap/matrix/${uuid}/matrix
00.0016
Get something to bootable to then start it with the MDEV assigned:
$ uvt-kvm create --memory=1024 --password=ubuntu bionic-vfio-ap arch=s390x
label=daily release=bionic
# wait until initialized and shut it down
$ virsh shutdown bionic-vfio-ap
# Modify to also use the MDEV
$ virsh edit bionic-vfio-ap
# add a snippet matching your UUID like:
<hostdev mode='subsystem' type='mdev' model='vfio-ap'>
<source>
<address uuid='24f952b3-03d1-4df2-9967-0d5f7d63d5f2'/>
</source>
</hostdev>
When restarting the guest this correctly adds the commandline argument:
-device
vfio-ap,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/24f952b3-03d1-4df2-9967-0d5f7d63d5f2
We also see virt-aa helper generting vfio rules
$ grep '/dev/vfio' /etc/apparmor.d/libvirt/$(virsh dominfo bionic-vfio-ap | awk
'/^Security label/ {print $3}').files
"/dev/vfio/vfio" rw,
"/dev/vfio/[0-9]*" rw,
And most importantly in the guest the adapter is present:
$ lszcrypt
CARD.DOMAIN TYPE MODE STATUS REQUEST_CNT
-------------------------------------------------
00 CEX5C CCA-Coproc online 1
00.0016 CEX5C CCA-Coproc online 1
Thanks Halil that I was watching [2] in Edinburgh :-)
And thanks cborntra for the WIP script to set those up.
[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3520
[2]:
https://events.linuxfoundation.org/wp-content/uploads/2017/12/vfio-ap-The-Perils-of-the-Weird-Halil-Pasic-IBM.pdf
** Description changed:
[Impact]
- * The ability to pass through more cryptographic capabilities is a very
- important feature for users of s390x as virtualization platform.
- Its availability upstream now and its backport in this bug allows to
- exploit the crypto cards as new HW for these virtualization use
- cases.
-
- * This falls under both "other safe cases" SRU exceptions:
- - For Long Term Support releases we regularly want to enable new
- hardware ...
- - For Long Term Support releases we sometimes want to introduce new
- features. They must not change the behaviour on existing
- installations ...
+ * The ability to pass through more cryptographic capabilities is a very
+ important feature for users of s390x as virtualization platform.
+ Its availability upstream now and its backport in this bug allows to
+ exploit the crypto cards as new HW for these virtualization use
+ cases.
- * This bug has three main components:
- - kernel (ability to do all of this)
- - qemu (add feature to exploit the new code)
- - libvirt (make the feature user consumable)
+ * This falls under both "other safe cases" SRU exceptions:
+ - For Long Term Support releases we regularly want to enable new
+ hardware ...
+ - For Long Term Support releases we sometimes want to introduce new
+ features. They must not change the behaviour on existing
+ installations ...
+
+ * This bug has three main components:
+ - kernel (ability to do all of this)
+ - qemu (add feature to exploit the new code)
+ - libvirt (make the feature user consumable)
[Test Case]
- * TBD: prepping commands atm ...
+ * In general this consists of a few steps
+ - get the updated kernel/qemu/libvirt
+ - mask the card & domains from the usual driver
+ - load vfio-ap
+ - assign card&domain to vfio-ap
+ - prepare a guest
+ - configure a guest to use the card
+
+ * See comment #66 how to do all of that in detail
+
+ https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1787405/comments/66
[Regression Potential]
- * The changes are mostly s390x only and adding a new feature so
- regressions to existing components should be low. But to backport it
- slight changes to the MDEV handling had to be applied as well.
- The potential regressions I can see are in that MDEV handling if one
- of the backports would be bad.
- Fortunately we know that without the related libvirt fixes we added
- here using MDEVs didn't work at all yet, and people very rarely use
- qemu without libvirt for anything else than experiments.
- Therefore I'm confident that even if there would be a flaw in the
- MDEV changes no one is hugely relying on it.
+ * The changes are mostly s390x only and adding a new feature so
+ regressions to existing components should be low. But to backport it
+ slight changes to the MDEV handling had to be applied as well.
+ The potential regressions I can see are in that MDEV handling if one
+ of the backports would be bad.
+ Fortunately we know that without the related libvirt fixes we added
+ here using MDEVs didn't work at all yet, and people very rarely use
+ qemu without libvirt for anything else than experiments.
+ Therefore I'm confident that even if there would be a flaw in the
+ MDEV changes no one is hugely relying on it.
[Other Info]
-
- * n/a
+ * n/a
== SRU Justification ==
(Kernel SRU)
Allow kvm to dedicate crypto adapters (and domains) as passthrough devices to
a KVM guest such that the hypervisor cannot observe the communication of the
guest with the device.
(Since all kernel patches/commits are from kernel 4.19, they will
automagically land in 'Disco'.)
== Fix ==
9ea5972 ("KVM: s390: vsie: simulate VCPU SIE entry/exit")
3194cdb ("KVM: s390: introduce and use KVM_REQ_VSIE_RESTART")
e585b24 ("KVM: s390: refactor crypto initialization")
1fde573 ("s390: vfio-ap: base implementation of VFIO AP device driver")
65f0671 ("s390: vfio-ap: register matrix device with VFIO mdev framework")
96d152b ("s390: vfio-ap: sysfs interfaces to configure adapters")
3211da0 ("s390: vfio-ap: sysfs interfaces to configure domains")
3b1eab7 ("s390: vfio-ap: sysfs interfaces to configure control domains")
81b2b4b ("s390: vfio-ap: sysfs interface to view matrix mdev matrix")
4210459 ("KVM: s390: interface to clear CRYCB masks")
258287c ("s390: vfio-ap: implement mediated device open callback")
e06670c ("s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl")
46a7263 ("s390: vfio-ap: zeroize the AP queues")
cd8a377 ("s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl")
6cc571b ("KVM: s390: Clear Crypto Control Block when using vSIE")
d6f6959 ("KVM: s390: vsie: Do the CRYCB validation first")
3af84de ("KVM: s390: vsie: Make use of CRYCB FORMAT2 clear")
56019f9 ("KVM: s390: vsie: Allow CRYCB FORMAT-2")
19fd83a ("KVM: s390: vsie: allow CRYCB FORMAT-1")
6ee7409 ("KVM: s390: vsie: allow CRYCB FORMAT-0")
c9ba8c2 ("KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1")
6b79de4 ("KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2")
9ee71f2 ("KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2")
37940fb ("KVM: s390: device attrs to enable/disable AP interpretation")
112c24d ("KVM: s390: CPU model support for AP virtualization")
492a6be ("s390: doc: detailed specifications for AP virtualization")
<-- till here in 'kvm/next'
(https://git.kernel.org/pub/scm/virt/kvm/kvm.git/) -->
8e41bd5 ("KVM: s390: fix locking for crypto setting error path")
0e237e4 ("KVM: s390: Tracing APCB changes")
76c7829 ("s390: vfio-ap: setup APCB mask using KVM dedicated function")
<-- till here in 'kvms390/next'
(https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git/) -->
<-- In addition to that some prereqs for the 'ap/crypto' driver are
necessary -->
ea3c418 ("s390/zcrypt: Add ZAPQ inline function.")
df80c03 ("s390/zcrypt: Review inline assembler constraints.")
f1b0a43 ("s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.")
2395103 ("s390/zcrypt: fix ap_instructions_available() returncodes")
7e0bdbe ("s390/zcrypt: AP bus support for alternate driver(s)")
3d8f60d3 ("s390/zcrypt: hex string mask improvements for apmask and aqmask.")
fa108f9 ("s390/zcrypt: remove VLA usage from the AP bus")
<--
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1787405/comments/12
-->
== PATCH ==
Above git commits are all from 4.19.
The git commands for 4.18 would be:
$ git cherry-pick <all from 'kvm/next' list>
(112c24d "KVM: s390: CPU model support for AP virtualization" may have a
trivial merge conflict with the etoken patch)
$ git cherry-pick <all from 'kvms390/next' list>
$ git cherry-pick <all from 'ap/zcrypt' list>
== Regression Potential ==
Low to mid:
- mid because in summary there are a lot of changes, but low
- they are all limited to the s390x architecture
- and again limited to KVM/s390x, vfio-ap and the zcrypt (aka ap) driver
- Test kernel was built for testting.
== Test Case ==
Setup a system for KVM use on an s390x LPAR that has CryptoExpress (aka
crypto-) adapters installed.
Verify that the AP bus created a sysfs device for each APQN, like:
/sys/devices/ap/card04/04.0006
/sys/devices/ap/card04/04.0047
/sys/devices/ap/card0a/0a.0006
/sys/devices/ap/card0a/0a.0047
Verify the APQN range via the following two sysfs files:
/sys/bus/ap/apmask
/sys/bus/ap/aqmask
Configure and start a guest.
More details see: 492a6be ("s390: doc: detailed specifications for AP
virtualization")
But for that an updated qemu and libvirt should be in place - that's
addressed in LP1787405, too.
(So this is only the kernel part of that ticket.)
__________
Description:
Allow kvm to dedicate crypto adapters (and domains) as passthrough devices to
a KVM guest such that the hypervisor cannot observe the communication of the
guest with the device.
This functionality will be contribute to following packages.
--kernel, qemu and libvirt.
Currently these functions are not finalized and therefore no git-commit are
avalable,
- kernel > 4.19
- libvirt > 4.6.0
- qemu > 3.0
We will provide these as soon as possible.
This request is launched against Ubuntu 18.10 to fulllfil the feature
integration process of Canonical.
But the main intention is, to get this integrated into 18.04 LTS !!!!!!
Thererfore, the backports will be required for both distros.!
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1787405
Title:
[FEAT] Guest-dedicated Crypto Adapters
Status in Ubuntu on IBM z Systems:
In Progress
Status in libvirt package in Ubuntu:
In Progress
Status in linux package in Ubuntu:
Fix Committed
Status in qemu package in Ubuntu:
In Progress
Status in libvirt source package in Bionic:
Triaged
Status in linux source package in Bionic:
Fix Committed
Status in qemu source package in Bionic:
Triaged
Status in libvirt source package in Cosmic:
Triaged
Status in linux source package in Cosmic:
Fix Committed
Status in qemu source package in Cosmic:
Triaged
Status in linux source package in Disco:
Fix Committed
Bug description:
[Impact]
* The ability to pass through more cryptographic capabilities is a very
important feature for users of s390x as virtualization platform.
Its availability upstream now and its backport in this bug allows to
exploit the crypto cards as new HW for these virtualization use
cases.
* This falls under both "other safe cases" SRU exceptions:
- For Long Term Support releases we regularly want to enable new
hardware ...
- For Long Term Support releases we sometimes want to introduce new
features. They must not change the behaviour on existing
installations ...
* This bug has three main components:
- kernel (ability to do all of this)
- qemu (add feature to exploit the new code)
- libvirt (make the feature user consumable)
[Test Case]
* In general this consists of a few steps
- get the updated kernel/qemu/libvirt
- mask the card & domains from the usual driver
- load vfio-ap
- assign card&domain to vfio-ap
- prepare a guest
- configure a guest to use the card
* See comment #66 how to do all of that in detail
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1787405/comments/66
[Regression Potential]
* The changes are mostly s390x only and adding a new feature so
regressions to existing components should be low. But to backport it
slight changes to the MDEV handling had to be applied as well.
The potential regressions I can see are in that MDEV handling if one
of the backports would be bad.
Fortunately we know that without the related libvirt fixes we added
here using MDEVs didn't work at all yet, and people very rarely use
qemu without libvirt for anything else than experiments.
Therefore I'm confident that even if there would be a flaw in the
MDEV changes no one is hugely relying on it.
[Other Info]
* n/a
== SRU Justification ==
(Kernel SRU)
Allow kvm to dedicate crypto adapters (and domains) as passthrough devices to
a KVM guest such that the hypervisor cannot observe the communication of the
guest with the device.
(Since all kernel patches/commits are from kernel 4.19, they will
automagically land in 'Disco'.)
== Fix ==
9ea5972 ("KVM: s390: vsie: simulate VCPU SIE entry/exit")
3194cdb ("KVM: s390: introduce and use KVM_REQ_VSIE_RESTART")
e585b24 ("KVM: s390: refactor crypto initialization")
1fde573 ("s390: vfio-ap: base implementation of VFIO AP device driver")
65f0671 ("s390: vfio-ap: register matrix device with VFIO mdev framework")
96d152b ("s390: vfio-ap: sysfs interfaces to configure adapters")
3211da0 ("s390: vfio-ap: sysfs interfaces to configure domains")
3b1eab7 ("s390: vfio-ap: sysfs interfaces to configure control domains")
81b2b4b ("s390: vfio-ap: sysfs interface to view matrix mdev matrix")
4210459 ("KVM: s390: interface to clear CRYCB masks")
258287c ("s390: vfio-ap: implement mediated device open callback")
e06670c ("s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl")
46a7263 ("s390: vfio-ap: zeroize the AP queues")
cd8a377 ("s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl")
6cc571b ("KVM: s390: Clear Crypto Control Block when using vSIE")
d6f6959 ("KVM: s390: vsie: Do the CRYCB validation first")
3af84de ("KVM: s390: vsie: Make use of CRYCB FORMAT2 clear")
56019f9 ("KVM: s390: vsie: Allow CRYCB FORMAT-2")
19fd83a ("KVM: s390: vsie: allow CRYCB FORMAT-1")
6ee7409 ("KVM: s390: vsie: allow CRYCB FORMAT-0")
c9ba8c2 ("KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1")
6b79de4 ("KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2")
9ee71f2 ("KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2")
37940fb ("KVM: s390: device attrs to enable/disable AP interpretation")
112c24d ("KVM: s390: CPU model support for AP virtualization")
492a6be ("s390: doc: detailed specifications for AP virtualization")
<-- till here in 'kvm/next'
(https://git.kernel.org/pub/scm/virt/kvm/kvm.git/) -->
8e41bd5 ("KVM: s390: fix locking for crypto setting error path")
0e237e4 ("KVM: s390: Tracing APCB changes")
76c7829 ("s390: vfio-ap: setup APCB mask using KVM dedicated function")
<-- till here in 'kvms390/next'
(https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git/)
-->
<-- In addition to that some prereqs for the 'ap/crypto' driver are
necessary -->
ea3c418 ("s390/zcrypt: Add ZAPQ inline function.")
df80c03 ("s390/zcrypt: Review inline assembler constraints.")
f1b0a43 ("s390/zcrypt: Integrate ap_asm.h into include/asm/ap.h.")
2395103 ("s390/zcrypt: fix ap_instructions_available() returncodes")
7e0bdbe ("s390/zcrypt: AP bus support for alternate driver(s)")
3d8f60d3 ("s390/zcrypt: hex string mask improvements for apmask and aqmask.")
fa108f9 ("s390/zcrypt: remove VLA usage from the AP bus")
<--
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1787405/comments/12
-->
== PATCH ==
Above git commits are all from 4.19.
The git commands for 4.18 would be:
$ git cherry-pick <all from 'kvm/next' list>
(112c24d "KVM: s390: CPU model support for AP virtualization" may have
a trivial merge conflict with the etoken patch)
$ git cherry-pick <all from 'kvms390/next' list>
$ git cherry-pick <all from 'ap/zcrypt' list>
== Regression Potential ==
Low to mid:
- mid because in summary there are a lot of changes, but low
- they are all limited to the s390x architecture
- and again limited to KVM/s390x, vfio-ap and the zcrypt (aka ap) driver
- Test kernel was built for testting.
== Test Case ==
Setup a system for KVM use on an s390x LPAR that has CryptoExpress (aka
crypto-) adapters installed.
Verify that the AP bus created a sysfs device for each APQN, like:
/sys/devices/ap/card04/04.0006
/sys/devices/ap/card04/04.0047
/sys/devices/ap/card0a/0a.0006
/sys/devices/ap/card0a/0a.0047
Verify the APQN range via the following two sysfs files:
/sys/bus/ap/apmask
/sys/bus/ap/aqmask
Configure and start a guest.
More details see: 492a6be ("s390: doc: detailed specifications for AP
virtualization")
But for that an updated qemu and libvirt should be in place - that's
addressed in LP1787405, too.
(So this is only the kernel part of that ticket.)
__________
Description:
Allow kvm to dedicate crypto adapters (and domains) as passthrough devices to
a KVM guest such that the hypervisor cannot observe the communication of the
guest with the device.
This functionality will be contribute to following packages.
--kernel, qemu and libvirt.
Currently these functions are not finalized and therefore no git-commit are
avalable,
- kernel > 4.19
- libvirt > 4.6.0
- qemu > 3.0
We will provide these as soon as possible.
This request is launched against Ubuntu 18.10 to fulllfil the feature
integration process of Canonical.
But the main intention is, to get this integrated into 18.04 LTS !!!!!!
Thererfore, the backports will be required for both distros.!
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-z-systems/+bug/1787405/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
