On 04/08/26 11:37 PM, Amit Machhiwal wrote:
On POWER systems, newer processor generations can operate in compatibility
modes corresponding to earlier generations (e.g., a Power11 system running
in Power10 compatibility mode). In such cases, the effective CPU level
exposed to guests differs from the physical processor generation.
This creates a problem for nested virtualization. When booting a nested KVM
guest (L2) inside a host KVM guest (L1) running in a compatibility mode,
userspace (e.g., QEMU) may derive the CPU model from the raw hardware PVR
and attempt to configure the nested guest accordingly. However, the L1
partition is constrained by the compatibility level negotiated with the
hypervisor (L0), and requests exceeding that level are rejected, leading to
guest boot failures such as:
KVM-NESTEDv2: couldn't set guest wide elements
This series provides a mechanism for userspace to query the effective CPU
compatibility modes supported by the host, so it can select an appropriate
CPU model for nested guests.
To achieve this, the series introduces a new KVM capability and ioctl
(KVM_CAP_PPC_COMPAT_CAPS / KVM_PPC_GET_COMPAT_CAPS) that expose the
compatibility modes supported by the host.
Why a new UAPI?
===============
While cpu-version is available in /proc/device-tree/cpus/<cpu#>/cpu-version
on both L1 booted on PowerNV and PowerVM LPARs, the UAPI approach is
preferable for several reasons:
1. pHYP (L0) capabilities: On PowerVM, we need to rely on capabilities
negotiated with pHYP in KVM, not just device tree properties. The
cpu-version property depicts the current compat mode but doesn't point
to what all compat modes are supported for the nested guest.
2. procfs dependency: Not all systems run with procfs enabled (CONFIG_PROC_FS
is optional). Minimal configurations like buildroot might disable it, but
KVM ioctl works regardless since it accesses kernel data structures
directly.
3. Kernel validation: The kernel validates and normalizes the compatibility
information, ensuring userspace gets validated, consistent data.
4. Abstraction & stability: /proc/device-tree is an implementation detail.
The UAPI provides a stable interface that won't break if the underlying
mechanism changes.
5. Semantic clarity: KVM_PPC_GET_COMPAT_CAPS clearly expresses what
compatibility modes can be used for KVM guests, vs. parsing device tree
which requires understanding the semantic meaning of cpu-version.
The implementation supports both:
- KVM on PowerVM (nested API v2), where compatibility information is
served from the cached nested_capabilities value, originally obtained
via the H_GUEST_GET_CAPABILITIES hypercall at module init.
- KVM on PowerNV (nested API v1), where compatibility is derived from the
device tree ("cpu-version") representing the effective processor
compatibility level.
This allows userspace (e.g., QEMU) to select a CPU model consistent with
the host compatibility mode, avoiding mismatches and enabling successful
nested guest boot.
Note: This series is built on top of patch [1] which must be applied first.
Patch [1] ensures arch_compat is validated against the host compatibility
mode before this series adds the capability query mechanism.
Commit e4de1b9cb3b5 ("powerpc/dt_cpu_ftrs: Set CPU_FTR_P11_PVR for Power11
and later processors") which was also a prerequisite has been merged upstream.
Changes in v6:
- Changed KVM_PPC_GET_COMPAT_CAPS ioctl number from 0xe4 to 0xb8 to
avoid placing it in the KVM_CREATE_DEVICE fd ioctl range (0xe0-0xe3);
relocated definition to sit alongside other PPC vm ioctls (patch 1)
- [Gautam]
- kvmppc_map_compat_capabilities(): changed parameter type from
'const __be32' to 'u32' to fix Sparse type annotation warning (patch 3)
- [Sashiko]
- kvmppc_get_compat_caps(): replaced of_get_property() + be32_to_cpup()
with of_property_read_u32() for implicit length validation and cleaner
endianness handling (patch 3) - [Sashiko]
- Documentation: corrected :Parameters: from (out) to (in/out) since
userspace must set size and flags before calling (patch 4) - [Sashiko]
Patch summary:
[1/4] Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
[2/4] Implement capability retrieval for KVM on PowerVM (API v2)
[3/4] Add KVM on PowerNV support (API v1)
[4/4] Document the new ioctl
Testing (with QEMU v4 patches and on top of patch [1]):
KVM APIv1 Testing
=================
On P10 PowerNV machine (L0)
---------------------------
- P10 L1 KVM guest -> works
- P10 nested L2 KVM guest -> works
- P9 compat nested L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 nested L2 KVM guest -> works
On Powernv11 TCG Guest (L0)
---------------------------
- P11 PowerNV TCG L0 guest -> works
- P11 L1 KVM guest -> works
- P11 L2 KVM guest -> works
- P10 compat L1 KVM guest -> works
- P10 L2 KVM guest -> works
- P9 compat L1 KVM guest -> works
- P9 L2 KVM guest -> works
KVM APIv2 Testing
=================
On P11 PowerVM LPAR (L1)
------------------------
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without QEMU patches but Linux patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
- Without Linux patches but QEMU patches
- P11 L2 KVM guest -> works
- P10 compat L2 KVM guest -> works
On P11 LPAR in P10 compat (L1)
------------------------------
- P10 (host compat) L2 KVM guest -> works
- Without QEMU patch but Linux patches
- P10 guest fails to boot as expected (error: kvm run failed Invalid
argument)
- Without Linux patch but QEMU patches
- P10 guest fails to boot as expected (KVM: unknown exit, hardware
reason ffffffffffffffea)
On P10 PowerVM LPAR (L1)
------------------------
- P10 L2 KVM guest -> works
- P9 compat L2 KVM guest fails to boot as expected
TCG pSeries Guest
=================
- P11 (default) pSeries guest boots fine
ABI Extensibility Testing (struct size 32, extra member)
=========================================================
- Newer struct on QEMU, older kernel -> works (kernel returns -E2BIG,
QEMU retries with correct size)
- New struct on Linux kernel, older QEMU -> works (kernel zero-pads
trailing fields, QEMU gets correct data)
With this series, nested guests boot successfully in configurations where
they previously failed due to compatibility mismatches.
Related QEMU series:
====================
A corresponding QEMU v5 series will be sent soon.
Previous QEMU versions:
v4: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v1: https://lore.kernel.org/all/[email protected]/
Previous versions:
==================
v5:
https://lore.kernel.org/linuxppc-dev/[email protected]/
v4:
https://lore.kernel.org/linuxppc-dev/[email protected]/
v3:
https://lore.kernel.org/linuxppc-dev/[email protected]/
v2:
https://lore.kernel.org/linuxppc-dev/[email protected]/
v1:
https://lore.kernel.org/linuxppc-dev/[email protected]/
References:
===========
[1] https://lore.kernel.org/all/[email protected]/
Amit Machhiwal (4):
KVM: PPC: Introduce KVM_CAP_PPC_COMPAT_CAPS and wire up ioctl
KVM: PPC: Book3S HV: Implement compat CPU capability retrieval for KVM
on PowerVM
KVM: PPC: Book3S HV: Add support for compat CPU capabilities for KVM
on PowerNV
KVM: PPC: Document KVM_PPC_GET_COMPAT_CAPS ioctl
Documentation/virt/kvm/api.rst | 79 +++++++++++++++++++++++++++++
arch/powerpc/include/asm/kvm_ppc.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 18 +++++++
arch/powerpc/kvm/book3s_hv.c | 56 ++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 71 ++++++++++++++++++++++++++
include/uapi/linux/kvm.h | 3 ++
6 files changed, 228 insertions(+)
base-commit: 848acc8ffe1b7cd5f1bf427b93069becfebc2c9d
prerequisite-patch-id: 7755786f0e4f415e47065ff1972765008727fe10
Hi Amit,
I have tested this patch and it works as expected. Here is my analysis :
I booted a host with Power10 compat mode and tried following scenarios -
lscpu on host :
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Model name: POWER10 (architected), altivec supported
Before applying the patch :
When I am trying to bringup the guest on a compat mode host it was
bringing up a Power11 guest and was failing as
[ 1411.578944] [ T2928] KVM-NESTEDv2: couldn't set guest wide elements
[ 1411.578963] [ T2928] vcpu 000000000b9c4155 (0):
[ 1411.578968] [ T2928] pc = 000000007daf9790 msr =
8000000000103000 trap = ffffffea
[ 1411.578973] [ T2928] r 0 = 8000000000003000 r16 = 0000000000000000
[ 1411.578978] [ T2928] r 1 = 000000007e581e20 r17 = 0000000000000000
[ 1411.578982] [ T2928] r 2 = 000000007db26c00 r18 = 0000000000000000
[ 1411.578985] [ T2928] r 3 = 0000000000000000 r19 = 0000000000000000
[ 1411.578989] [ T2928] r 4 = 0000000002e30c80 r20 = 0000000000000000
[ 1411.578993] [ T2928] r 5 = 000000007df80000 r21 = 0000000000000000
[ 1411.578996] [ T2928] r 6 = 0000000000200000 r22 = 00000000018c5fd6
[ 1411.579000] [ T2928] r 7 = 000000007df80000 r23 = 000000007db21cc0
[ 1411.579003] [ T2928] r 8 = 000000007db6e5d8 r24 = 000000007db66000
[ 1411.579006] [ T2928] r 9 = 000000007e6655d8 r25 = 000000007e665508
[ 1411.579010] [ T2928] r10 = 000000007db6e5d0 r26 = 00000000018c5fd6
[ 1411.579013] [ T2928] r11 = 0000000000003000 r27 = 0000000000000003
[ 1411.579017] [ T2928] r12 = 8000000000000001 r28 = 000000007db6e5e0
[ 1411.579020] [ T2928] r13 = 0000000000000000 r29 = 000000007db224b0
[ 1411.579024] [ T2928] r14 = 0000000000000000 r30 = 000000007daf274c
[ 1411.579028] [ T2928] r15 = 0000000000000000 r31 = 000000007db76000
[ 1411.579033] [ T2928] ctr = 000000007daf1b44 lr = 000000007daf1b7c
[ 1411.579037] [ T2928] srr0 = 000000007daf9790 srr1 = 8000000000102000
[ 1411.579041] [ T2928] sprg0 = 0000000000000000 sprg1 = 000000000000ff10
[ 1411.579045] [ T2928] sprg2 = 0000000000000000 sprg3 = 0000000000000000
[ 1411.579049] [ T2928] cr = 20000402 xer = 0000000020040000 dsisr =
00000000
[ 1411.579054] [ T2928] dar = 0000000000000000
[ 1411.579057] [ T2928] fault dar = 0000000000000000 dsisr = 00000000
[ 1411.579061] [ T2928] SLB (0 entries):
[ 1411.579064] [ T2928] lpcr = 0040000000020400 sdr1 =
0000000000000000 last_inst = ffffffffffffffff
[ 1411.579069] [ T2928] trap=0xffffffea | pc=0x7daf9790 |
msr=0x8000000000103000
After applying this patch along with the qemu built with it's dependent
patch
(https://lore.kernel.org/all/[email protected]/):
I am able to bringup a guest and it got boot up with Power10 by default:
lscpu on guest -
ltcbonn53-vm2:~ # lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Model name: POWER10 (architected), altivec supported
Model: 2.0 (pvr 0082 0200)
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
Please feel free to add my tested-by:
Tested-by: Anushree Mathur <[email protected]>
Thank you,
Anushree Mathur
