[ Note: I've spoken with both Mark Rutland and James Morse both of whom
might potentially have some overlapping changes, they've both agreed
that they'll deal with any rebasing if it's a question of this getting
in or something else. ]
I've removed the RFC tag from this version of the series, but the items
that I'm looking for feedback on remains the same:
- The userspace ABI, in particular:
- The vector length used for the SVE registers, access to the SVE
registers and access to ZA and (if available) ZT0 depending on
the current state of PSTATE.{SM,ZA}.
- The use of a single finalisation for both SVE and SME.
- There was some review from the qemu people who seemed OK with the
structure, and there's some initial patches from them at:
https://patchew.org/QEMU/[email protected]/
This series implements support for SME use in non-protected KVM guests.
Much of this is very similar to SVE, the main additional challenge that
SME presents is that it introduces a new vector length similar to the
SVE vector length and two new controls which change the registers seen
by guests:
- PSTATE.ZA enables the ZA matrix register and, if SME2 is supported,
the ZT0 LUT register.
- PSTATE.SM enables streaming mode, a new floating point mode which
uses the SVE register set with the separately configured SME vector
length. In streaming mode implementation of the FFR register is
optional.
It is also permitted to build systems which support SME without SVE, in
this case when not in streaming mode no SVE registers or instructions
are available. Further, there is no requirement that there be any
overlap in the set of vector lengths supported by SVE and SME in a
system, this is expected to be a common situation in practical systems.
Since there is a new vector length to configure we introduce a new
feature parallel to the existing SVE one with a new pseudo register for
the streaming mode vector length. Due to the overlap with SVE caused by
streaming mode rather than finalising SME as a separate feature we use
the existing SVE finalisation to also finalise SME, a new define
KVM_ARM_VCPU_VEC is provided to help make user code clearer. Finalising
SVE and SME separately would introduce complication with register access
since finalising SVE makes the SVE registers writeable by userspace and
doing multiple finalisations results in an error being reported.
Dealing with a state where the SVE registers are writeable due to one of
SVE or SME being finalised but may have their VL changed by the other
being finalised seems like needless complexity with minimal practical
utility, it seems clearer to just express directly that only one
finalisation can be done in the ABI.
Access to the floating point registers follows the architecture:
- When both SVE and SME are present:
- If PSTATE.SM == 0 the vector length used for the Z and P registers
is the SVE vector length.
- If PSTATE.SM == 1 the vector length used for the Z and P registers
is the SME vector length.
- If only SME is present:
- If PSTATE.SM == 0 the Z and P registers are inaccessible and the
floating point state accessed via the encodings for the V registers.
- If PSTATE.SM == 1 the vector length used for the Z and P registers
- The SME specific ZA and ZT0 registers are only accessible if SVCR.ZA is 1.
The VMM must understand this, in particular when loading state SVCR
should be configured before other state. It should be noted that while
the architecture refers to PSTATE.SM and PSTATE.ZA these PSTATE bits are
not preserved in SPSR_ELx, they are only accessible via SVCR.
There are a large number of subfeatures for SME, most of which only
offer additional instructions but some of which (SME2 and FA64) add
architectural state. These are configured via the ID registers as per
usual.
Protected KVM is supported, with the implementation maintaining the
existing restriction that the hypervisor will refuse to run if streaming
mode or ZA is enabled. This both simplfies the code and avoids the need
to allocate storage for host ZA and ZT0 state, there seems to be little
practical use case for supporting this and the memory usage would be
non-trivial. SME is not made available to protected guests, only normal
guests.
The new KVM_ARM_VCPU_VEC feature and ZA and ZT0 registers have not been
added to the get-reg-list selftest, the idea of supporting additional
features there without restructuring the program to generate all
possible feature combinations has been rejected. I will post a separate
series which does that restructuring.
There is some room for optimisation here, for example we could do more
work to suppress isb()s on SMCR changes (we don't really try in the
hypervisor), but this is already a very large series.
Signed-off-by: Mark Brown <[email protected]>
---
Changes in v13:
- Rebase onto v7.2-rc3.
- Remove existing special casing of ZCR_EL2 as an access trap and the
corresponding newly added SMCR_EL2 handling.
- Fix handling of nested EZT0/FA64, factoring out shared initialisation
for both SMCR_EL2 and nested EL2 traps.
- Document why we isb() after updating SMCR, and suppress redundant
isb()s for the host kernel.
- Add missing isb()s in the hypervisor.
- Use generic RES0 handling for SMPRIMAP_EL2.
- Fix comment on SMPRI_EL1.
- Emulate SCMR_EL2.EZT0 traps for nested guests.
- Remove a bitrotted comment on the SVE exit handler.
- Don't restore SMCR_EL1 after running non-SME guests with pKVM.
- Add an explicit reset of ZCR_EL1.
- Add a missing __init annotation to vec_virtualisable_vl().
- Add SMIDR_EL1 to the ABI documentation for KVM_CAP_ARM_WRITABLE_IMP_ID_REGS.
- Block attempts to set ID_AA64PFR1_EL1.SME to 0 when SME is enabled.
- Process vCPU resets before enumerating registers.
- Changelog updates.
- Link to v12:
https://patch.msgid.link/[email protected]
Changes in v12:
- Resend with commit rather than tag object specified as the base.
- Link to v11:
https://patch.msgid.link/[email protected]
Changes in v11:
- Rebase onto v7.2-rc2.
- Rework VL enumeration.
- Refactor ZCR and SMCR value generation for the host into helper
functions.
- Add handling of FEAT_IDST for hidden system registers.
- Ensure we manage traps correctly for emulated EL2.
- Always allocate space for ZT0 if the hardware supports it.
- Tighten the register enumeration ABI to reflect current VM state.
- Sanitise SMIDR_EL1.
- Fix generation of invalid values in set_id_regs.
- Various smaller fixups, including from Jean-Phillipe's review.
- Link to v10:
https://patch.msgid.link/[email protected]
Changes in v10:
- Define and use a SME_VQ_INVALID for the case where there is no
virtuablisable SME VL.
- Fix handling of SMCR_EL2 accesses.
- Correct VNCR constant for SMPRI_EL2.
- Correct trapping for SMPRI_EL1.
- Reject userspace access to FFR when in streaming mode without FA64.
- Constrain the VL set by sme_cond_update_smcr() to fit within LEN.
- Reject userspace access to ZA and ZT0 when SVCR.SM is 0.
- Use -EACCESS for inaccessible SME registers.
- Remove some unused functions.
- Further bugfixes from review.
- Commit log typo fixes.
- Link to v9:
https://patch.msgid.link/[email protected]
Changes in v9:
- Rebase onto v6.19-rc1.
- ABI document clarifications.
- Add changes dropping asserts on single bit wide bitfields in set_id_regs.
- Link to v8:
https://lore.kernel.org/r/[email protected]
Changes in v8:
- Small fixes in ABI documentation.
- Link to v7:
https://lore.kernel.org/r/[email protected]
Changes in v7:
- Rebase onto v6.17-rc1.
- Handle SMIDR_EL1 as a VM wide ID register and use this in feat_sme_smps().
- Expose affinity fields in SMIDR_EL1.
- Remove SMPRI_EL1 from vcpu_sysreg, the value is always 0 currently.
- Prevent userspace writes to SMPRIMAP_EL2.
- Link to v6:
https://lore.kernel.org/r/[email protected]
Changes in v6:
- Rebase onto v6.16-rc3.
- Link to v5:
https://lore.kernel.org/r/[email protected]
Changes in v5:
- Rebase onto v6.15-rc2.
- Add pKVM guest support.
- Always restore SVCR.
- Link to v4:
https://lore.kernel.org/r/[email protected]
Changes in v4:
- Rebase onto v6.14-rc2 and Mark Rutland's fixes.
- Expose SME to nested guests.
- Additional cleanups and test fixes following on from the rebase.
- Flush register state on VMM PSTATE.{SM,ZA}.
- Link to v3:
https://lore.kernel.org/r/[email protected]
Changes in v3:
- Rebase onto v6.12-rc2.
- Link to v2:
https://lore.kernel.org/r/[email protected]
Changes in v2:
- Rebase onto v6.7-rc3.
- Configure subfeatures based on host system only.
- Complete nVHE support.
- There was some snafu with sending v1 out, it didn't make it to the
lists but in case it hit people's inboxes I'm sending as v2.
---
Mark Brown (32):
arm64/sysreg: Define full value read/modify/write helpers
arm64/fpsimd: Ensure all of ZCR_EL1 is initialised from idle
arm64/fpsimd: Configure all ZCR/SMCR bits when loading task state
arm64/fpsimd: Decide to save ZT0 and streaming mode FFR at bind time
arm64/sve: Factor virtualizable VL discovery out of SVE specific code
arm64/fpsimd: Determine maximum virtualisable SME vector length
KVM: arm64: Remove bitrotted comment on handle_sve()
KVM: arm64: Handle FEAT_IDST for guest accesses to hidden registers
KVM: arm64: Pull ctxt_has_ helpers to start of sysreg-sr.h
KVM: arm64: Rename SVE finalization constants to be more general
KVM: arm64: Remove special case for FP state loading from ZCR_EL2 traps
KVM: arm64: Define internal features for SME
KVM: arm64: Rename sve_state_reg_region
KVM: arm64: Store vector lengths in an array
KVM: arm64: Factor SVE code out of fpsimd_lazy_switch_to_host()
KVM: arm64: Document the KVM ABI for SME
KVM: arm64: Implement SME vector length configuration
KVM: arm64: Support SME control registers
KVM: arm64: Support TPIDR2_EL0
KVM: arm64: Support SME identification registers for guests
KVM: arm64: Support SME priority registers
KVM: arm64: Support userspace access to streaming mode Z and P registers
KVM: arm64: Flush register state on writes to SVCR.SM and SVCR.ZA
KVM: arm64: Expose SME specific state to userspace
KVM: arm64: Context switch SME state for guests
KVM: arm64: Handle SME exceptions
KVM: arm64: Expose SME to nested guests
KVM: arm64: Provide interface for configuring and enabling SME for guests
KVM: arm64: selftests: Remove spurious check for single bit safe values
KVM: arm64: selftests: Skip impossible invalid value tests
KVM: arm64: selftests: Add SME system registers to get-reg-list
KVM: arm64: selftests: Add SME to set_id_regs test
Documentation/virt/kvm/api.rst | 127 +++++---
arch/arm64/include/asm/fpsimd.h | 11 +-
arch/arm64/include/asm/kvm_emulate.h | 16 +
arch/arm64/include/asm/kvm_host.h | 117 ++++++-
arch/arm64/include/asm/kvm_hyp.h | 2 +-
arch/arm64/include/asm/kvm_nested.h | 2 +
arch/arm64/include/asm/kvm_pkvm.h | 2 +-
arch/arm64/include/asm/sysreg.h | 8 +
arch/arm64/include/asm/vncr_mapping.h | 2 +
arch/arm64/include/uapi/asm/kvm.h | 34 +++
arch/arm64/kernel/cpufeature.c | 2 -
arch/arm64/kernel/fpsimd.c | 144 ++++-----
arch/arm64/kernel/suspend.c | 2 +-
arch/arm64/kvm/arm.c | 19 ++
arch/arm64/kvm/config.c | 13 +-
arch/arm64/kvm/emulate-nested.c | 6 +-
arch/arm64/kvm/fpsimd.c | 27 +-
arch/arm64/kvm/guest.c | 369 ++++++++++++++++++++---
arch/arm64/kvm/handle_exit.c | 23 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 262 +++++++++++++---
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 96 +++---
arch/arm64/kvm/hyp/nvhe/hyp-main.c | 133 ++++++--
arch/arm64/kvm/hyp/nvhe/pkvm.c | 90 ++++--
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +
arch/arm64/kvm/hyp/nvhe/sys_regs.c | 6 +
arch/arm64/kvm/hyp/vhe/switch.c | 28 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 7 +
arch/arm64/kvm/inject_fault.c | 15 +
arch/arm64/kvm/nested.c | 15 +-
arch/arm64/kvm/reset.c | 159 +++++++---
arch/arm64/kvm/sys_regs.c | 155 +++++++++-
include/uapi/linux/kvm.h | 1 +
tools/testing/selftests/kvm/arm64/get-reg-list.c | 15 +-
tools/testing/selftests/kvm/arm64/set_id_regs.c | 96 +++++-
34 files changed, 1602 insertions(+), 404 deletions(-)
---
base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa
change-id: 20230301-kvm-arm64-sme-06a1246d3636
Best regards,
--
Mark Brown <[email protected]>
