Re: [PATCH 12/15] arm-soc: omap: replace open coded VA->PA calculations
* Ard Biesheuvel[170805 13:54]: > This replaces a couple of open coded calculations to obtain the > physical address of a far symbol with calls to the new adr_l etc > macros. I gave this series a quick test and omap3 no longer boots it seems. Regards, Tony ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v2 06/16] arm64: entry.S: convert elX_sync
Hi James, On Fri, Jul 28, 2017 at 03:10:09PM +0100, James Morse wrote: > @@ -520,9 +514,16 @@ el1_preempt: > el0_sync: > kernel_entry 0 > mrs x25, esr_el1// read the syndrome register > + mrs x26, far_el1 Just checking, since we are going to access far_el1 even when we get a syscall, have you noticed any overhead? -- Catalin ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 01/11] KVM: arm64: Store vcpu on the stack during __guest_enter()
On Tue, Aug 08, 2017 at 05:48:29PM +0100, James Morse wrote: > Hi Christoffer, > > On 06/06/17 20:59, Christoffer Dall wrote: > > On Mon, May 15, 2017 at 06:43:49PM +0100, James Morse wrote: > >> KVM uses tpidr_el2 as its private vcpu register, which makes sense for > >> non-vhe world switch as only KVM can access this register. This means > >> vhe Linux has to use tpidr_el1, which KVM has to save/restore as part > >> of the host context. > >> > >> __guest_enter() stores the host_ctxt on the stack, do the same with > >> the vcpu. > > >> diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S > >> index 12ee62d6d410..113735df7d01 100644 > >> --- a/arch/arm64/kvm/hyp/entry.S > >> +++ b/arch/arm64/kvm/hyp/entry.S > >> @@ -159,9 +159,15 @@ abort_guest_exit_end: > >> ENDPROC(__guest_exit) > >> > >> ENTRY(__fpsimd_guest_restore) > >> + // x0: esr > >> + // x1: vcpu > >> + // x2-x29,lr: vcpu regs > >> + // vcpu x0-x1 on the stack > >>stp x2, x3, [sp, #-16]! > >>stp x4, lr, [sp, #-16]! > >> > >> + mov x3, x1 > >> + > > > > nit: can you avoid this by using x1 for the vcpu pointer in this routine > > instead? > > Unfortunately x1 is clobbered by the __fpsimd_{save,restore}_state() macros > that > are called further down this function. > > (its a bit obscure: > > fpsimd_save x0, 1 > that '1' is used to generate 'x1' or 'w1' in includes/asm/fpsimdmacros.h) > Ah, I guess I missed that. Thanks, -Christoffer ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 06/27] arm64/sve: System register and exception syndrome definitions
The SVE architecture adds some system registers, ID register fields and a dedicated ESR exception class. This patch adds the appropriate definitions that will be needed by the kernel. Signed-off-by: Dave Martin--- arch/arm64/include/asm/esr.h | 3 ++- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/sysreg.h | 16 arch/arm64/kernel/traps.c| 1 + 4 files changed, 20 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 8cabd57..813629e 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -43,7 +43,8 @@ #define ESR_ELx_EC_HVC64 (0x16) #define ESR_ELx_EC_SMC64 (0x17) #define ESR_ELx_EC_SYS64 (0x18) -/* Unallocated EC: 0x19 - 0x1E */ +#define ESR_ELx_EC_SVE (0x19) +/* Unallocated EC: 0x1A - 0x1E */ #define ESR_ELx_EC_IMP_DEF (0x1f) #define ESR_ELx_EC_IABT_LOW(0x20) #define ESR_ELx_EC_IABT_CUR(0x21) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 61d694c..dbf0537 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -185,6 +185,7 @@ #define CPTR_EL2_TCPAC (1 << 31) #define CPTR_EL2_TTA (1 << 20) #define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) +#define CPTR_EL2_TZ(1 << 8) #define CPTR_EL2_DEFAULT 0x33ff /* Hyp Debug Configuration Register bits */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 248339e..2d259e8 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -145,6 +145,7 @@ #define SYS_ID_AA64PFR0_EL1sys_reg(3, 0, 0, 4, 0) #define SYS_ID_AA64PFR1_EL1sys_reg(3, 0, 0, 4, 1) +#define SYS_ID_AA64ZFR0_EL1sys_reg(3, 0, 0, 4, 4) #define SYS_ID_AA64DFR0_EL1sys_reg(3, 0, 0, 5, 0) #define SYS_ID_AA64DFR1_EL1sys_reg(3, 0, 0, 5, 1) @@ -160,6 +161,8 @@ #define SYS_ACTLR_EL1 sys_reg(3, 0, 1, 0, 1) #define SYS_CPACR_EL1 sys_reg(3, 0, 1, 0, 2) +#define SYS_ZCR_EL1sys_reg(3, 0, 1, 2, 0) + #define SYS_TTBR0_EL1 sys_reg(3, 0, 2, 0, 0) #define SYS_TTBR1_EL1 sys_reg(3, 0, 2, 0, 1) #define SYS_TCR_EL1sys_reg(3, 0, 2, 0, 2) @@ -250,6 +253,8 @@ #define SYS_PMCCFILTR_EL0 sys_reg (3, 3, 14, 15, 7) +#define SYS_ZCR_EL2sys_reg(3, 4, 1, 2, 0) + #define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) #define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) #define SYS_FPEXC32_EL2sys_reg(3, 4, 5, 3, 0) @@ -331,6 +336,7 @@ #define ID_AA64ISAR1_JSCVT_SHIFT 12 /* id_aa64pfr0 */ +#define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_ASIMD_SHIFT20 #define ID_AA64PFR0_FP_SHIFT 16 @@ -339,6 +345,7 @@ #define ID_AA64PFR0_EL1_SHIFT 4 #define ID_AA64PFR0_EL0_SHIFT 0 +#define ID_AA64PFR0_SVE0x1 #define ID_AA64PFR0_FP_NI 0xf #define ID_AA64PFR0_FP_SUPPORTED 0x0 #define ID_AA64PFR0_ASIMD_NI 0xf @@ -440,6 +447,15 @@ #endif +#define ZCR_ELx_LEN_SHIFT 0 +#define ZCR_ELx_LEN_SIZE 9 +#define ZCR_ELx_LEN_MASK 0x1ff + +#define CPACR_EL1_ZEN_EL1EN(1 << 16) +#define CPACR_EL1_ZEN_EL0EN(1 << 17) +#define CPACR_EL1_ZEN (CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN) + + /* Safe value for MPIDR_EL1: Bit31:RES1, Bit30:U:0, Bit24:MT:0 */ #define SYS_MPIDR_SAFE_VAL (1UL << 31) diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 0f047e9..8964795 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -621,6 +621,7 @@ static const char *esr_class_str[] = { [ESR_ELx_EC_HVC64] = "HVC (AArch64)", [ESR_ELx_EC_SMC64] = "SMC (AArch64)", [ESR_ELx_EC_SYS64] = "MSR/MRS (AArch64)", + [ESR_ELx_EC_SVE]= "SVE", [ESR_ELx_EC_IMP_DEF]= "EL3 IMP DEF", [ESR_ELx_EC_IABT_LOW] = "IABT (lower EL)", [ESR_ELx_EC_IABT_CUR] = "IABT (current EL)", -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 00/27] ARM Scalable Vector Extension (SVE)
This series implements Linux kernel support for the ARM Scalable Vector Extension (SVE). [1] It supersedes the previous RFC: see [6] for link and a summary of changes. This series depends on some series that are headed for v4.14: see [3], [4], [5]. To reduce spam, some people may not been copied on the entire series. For those who did not receive the whole series, it can be found in the linux-arm-kernel archive. [2] *Note* The final two patches (26-27) of the series are still RFC -- before committing to this ABI it would be good to get feedback on whether the approach makes sense and whether it suitable for other architectures. These two patches are not required by the rest of the series and can be revised or merged later. Support for use of SVE by KVM guests is not currently included. Instead, such use will be trapped and reflected to the guest as undefined instruction execution. SVE is hidden from the view of the CPU feature registers visible to guests, so that guests will not expect it to work. This series has been build- and boot-tested on Juno r0 and the ARM FVP Base model with SVE plugin. Because there is no hardware with SVE support yet, testing of the SVE functionality has only been performed on the model. Regression testing using LTP is under way and has also been completed on previous versions of this series. Series summary: * Patches 1-5 contain some individual bits of preparatory spadework, which are indirectly related to SVE. Dave Martin (5): regset: Add support for dynamically sized regsets arm64: KVM: Hide unsupported AArch64 CPU features from guests arm64: efi: Add missing Kconfig dependency on KERNEL_MODE_NEON arm64: Port deprecated instruction emulation to new sysctl interface arm64: fpsimd: Simplify uses of {set,clear}_ti_thread_flag() Non-trivial changes among these are: * Patch 1: updates the regset core code to handle regsets whose size is not fixed at compile time. This avoids bloating coredumps even though the maximum theoretical SVE regset size is large. * Patch 2: extends KVM to modify the ARM architectural ID registers seen by guests, by trapping and emulating certain registers. For SVE this is a temporary measure, but it may be useful for other architecture extensions. This patch may also be built on in the future, since the only registers currently emulated are those required for hiding SVE. * Patches 6-10 add SVE-specific system register and structure layout definitions, and the low-level boot code and accessors needed for making use of SVE. Dave Martin (5): arm64/sve: System register and exception syndrome definitions arm64/sve: Low-level SVE architectural state manipulation functions arm64/sve: Kconfig update and conditional compilation support arm64/sve: Signal frame and context structure definition arm64/sve: Low-level CPU setup * Patches 11-13 implement the core context management facilities to provide each user task with its own SVE register context, signal handling facilities, and sane programmer's model interoperation between SVE and FPSIMD. Dave Martin (3): arm64/sve: Core task context handling arm64/sve: Support vector length resetting for new processes arm64/sve: Signal handling support * Patches 14-15 provide backend logic for detecting and making use of the different SVE vector lengths supported by the hardware. Dave Martin (2): arm64/sve: Backend logic for setting the vector length arm64/sve: Probe SVE capabilities and usable vector lengths * Patches 16-17 update the kernel-mode NEON / EFI FPSIMD frameworks to interoperate correctly with SVE. Dave Martin (2): arm64/sve: Preserve SVE registers around kernel-mode NEON use arm64/sve: Preserve SVE registers around EFI runtime service calls * Patches 18-20 implement the userspace frontend for managing SVE, comprising ptrace, some new arch-specific prctl() calls, and a new sysctl for init-time setup. Dave Martin (3): arm64/sve: ptrace and ELF coredump support arm64/sve: Add prctl controls for userspace vector length management arm64/sve: Add sysctl to set the default vector length for new processes * Patches 21-23 provide stub KVM extensions for using KVM only on the host, while denying guest access. (A future series will extend this with full support for SVE in guests.) Dave Martin (3): arm64/sve: KVM: Prevent guests from using SVE arm64/sve: KVM: Treat guest SVE use as undefined instruction execution arm64/sve: KVM: Hide SVE from CPU features exposed to guests And finally: * Patch 24 disengages the safety catch, enabling the kernel SVE runtime support and allowing userspace to use SVE. Dave Martin (1): arm64/sve: Detect SVE and activate runtime support * Patch 25 adds some basic documentation. Dave Martin (1): arm64/sve: Add documentation * Patches 26-27 (which may be considered RFC) propose a mechanism to report the maximum
[PATCH 05/27] arm64: fpsimd: Simplify uses of {set, clear}_ti_thread_flag()
The existing FPSIMD context switch code contains a couple of instances of {set,clear}_ti_thread(task_thread_info(task)). Since there are thread flag manipulators that operate directly on task_struct, this verbosity isn't strictly needed. For consistency, this patch simplifies the affected calls. This should have no impact on behaviour. Signed-off-by: Dave Martin--- arch/arm64/kernel/fpsimd.c | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 138fcfa..9c1f268e 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -159,11 +159,9 @@ void fpsimd_thread_switch(struct task_struct *next) if (__this_cpu_read(fpsimd_last_state) == st && st->cpu == smp_processor_id()) - clear_ti_thread_flag(task_thread_info(next), -TIF_FOREIGN_FPSTATE); + clear_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE); else - set_ti_thread_flag(task_thread_info(next), - TIF_FOREIGN_FPSTATE); + set_tsk_thread_flag(next, TIF_FOREIGN_FPSTATE); } } -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 04/27] arm64: Port deprecated instruction emulation to new sysctl interface
Currently, armv8_deprected.c takes charge of the "abi" sysctl directory, which makes life difficult for other code that wants to register sysctls in the same directory. There is a "new" [1] sysctl registration interface that removes the need to define ctl_tables for parent directories explicitly, which is ideal here. This patch ports register_insn_emulation_sysctl() over to the register_sysctl() interface and removes the redundant ctl_table for "abi". Signed-off-by: Dave Martin[1] fea478d4101a (sysctl: Add register_sysctl for normal sysctl users) The commit message notes an intent to port users of the pre-existing interfaces over to register_sysctl(), though the number of users of the new interface currently appears negligible. --- arch/arm64/kernel/armv8_deprecated.c | 15 +++ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/arch/arm64/kernel/armv8_deprecated.c b/arch/arm64/kernel/armv8_deprecated.c index f0e6d71..e15eb2d 100644 --- a/arch/arm64/kernel/armv8_deprecated.c +++ b/arch/arm64/kernel/armv8_deprecated.c @@ -228,15 +228,7 @@ static int emulation_proc_handler(struct ctl_table *table, int write, return ret; } -static struct ctl_table ctl_abi[] = { - { - .procname = "abi", - .mode = 0555, - }, - { } -}; - -static void __init register_insn_emulation_sysctl(struct ctl_table *table) +static void __init register_insn_emulation_sysctl(void) { unsigned long flags; int i = 0; @@ -262,8 +254,7 @@ static void __init register_insn_emulation_sysctl(struct ctl_table *table) } raw_spin_unlock_irqrestore(_emulation_lock, flags); - table->child = insns_sysctl; - register_sysctl_table(table); + register_sysctl("abi", insns_sysctl); } /* @@ -644,7 +635,7 @@ static int __init armv8_deprecated_init(void) cpuhp_setup_state_nocalls(CPUHP_AP_ARM64_ISNDEP_STARTING, "arm64/isndep:starting", run_all_insn_set_hw_mode, NULL); - register_insn_emulation_sysctl(ctl_abi); + register_insn_emulation_sysctl(); return 0; } -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 02/27] arm64: KVM: Hide unsupported AArch64 CPU features from guests
Currently, a guest kernel sees the true CPU feature registers (ID_*_EL1) when it reads them using MRS instructions. This means that the guest will observe features that are present in the hardware but the host doesn't understand or doesn't provide support for. A guest may legimitately try to use such a feature as per the architecture, but use of the feature may trap instead of working normally, triggering undef injection into the guest. This is not a problem for the host, but the guest may go wrong when running on newer hardware than the host knows about. This patch hides from guest VMs any AArch64-specific CPU features that the host doesn't support, by exposing to the guest the sanitised versions of the registers computed by the cpufeatures framework, instead of the true hardware registers. To achieve this, HCR_EL2.TID3 is now set for AArch64 guests, and emulation code is added to KVM to report the sanitised versions of the affected registers in response to MRS and register reads from userspace. The affected registers are removed from invariant_sys_regs[] (since the invariant_sys_regs handling is no longer quite correct for them) and added to sys_reg_desgs[], with appropriate access(), get_user() and set_user() methods. No runtime vcpu storage is allocated for the registers: instead, they are read on demand from the cpufeatures framework. This may need modification in the future if there is a need for userspace to customise the features visible to the guest. Attempts by userspace to write the registers are handled similarly to the current invariant_sys_regs handling: writes are permitted, but only if they don't attempt to change the value. This is sufficient to support VM snapshot/restore from userspace. Because of the additional registers, restoring a VM on an older kernel may not work unless userspace knows how to handle the extra VM registers exposed to the KVM user ABI by this patch. Under the principle of least damage, this patch makes no attempt to handle any of the other registers currently in invariant_sys_regs[], or to emulate registers for AArch32: however, these could be handled in a similar way in future, as necessary. Signed-off-by: Dave Martin--- arch/arm64/kvm/hyp/switch.c | 6 ++ arch/arm64/kvm/sys_regs.c | 224 +++- 2 files changed, 185 insertions(+), 45 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 945e79c..35a90b8 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -81,11 +81,17 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) * it will cause an exception. */ val = vcpu->arch.hcr_el2; + if (!(val & HCR_RW) && system_supports_fpsimd()) { write_sysreg(1 << 30, fpexc32_el2); isb(); } + + if (val & HCR_RW) /* for AArch64 only: */ + val |= HCR_TID3; /* TID3: trap feature register accesses */ + write_sysreg(val, hcr_el2); + /* Trap on AArch32 cp15 c15 accesses (EL1 or EL0) */ write_sysreg(1 << 15, hstr_el2); /* diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2e070d3..6583dd7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -892,6 +892,135 @@ static bool access_cntp_cval(struct kvm_vcpu *vcpu, return true; } +/* Read a sanitised cpufeature ID register by sys_reg_desc */ +static u64 read_id_reg(struct sys_reg_desc const *r, bool raz) +{ + u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, +(u32)r->CRn, (u32)r->CRm, (u32)r->Op2); + + return raz ? 0 : read_sanitised_ftr_reg(id); +} + +/* cpufeature ID register access trap handlers */ + +static bool __access_id_reg(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc const *r, + bool raz) +{ + if (p->is_write) { + kvm_inject_undefined(vcpu); + return false; + } + + p->regval = read_id_reg(r, raz); + return true; +} + +static bool access_id_reg(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return __access_id_reg(vcpu, p, r, false); +} + +static bool access_raz_id_reg(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return __access_id_reg(vcpu, p, r, true); +} + +static int reg_from_user(u64 *val, const void __user *uaddr, u64 id); +static int reg_to_user(void __user *uaddr, const u64 *val, u64 id); +static u64 sys_reg_to_index(const struct sys_reg_desc *reg); + +/* + * cpufeature ID register user accessors + * + * For now, these registers are immutable for userspace, so no values + * are stored, and for set_id_reg() we
[PATCH 01/27] regset: Add support for dynamically sized regsets
Currently the regset API doesn't allow for the possibility that regsets (or at least, the amount of meaningful data in a regset) may change in size. In particular, this results in useless padding being added to coredumps in a regset's current size is smaller than its theoretical maximum size. This patch adds a get_size() function to struct user_regset. Individual regset implementations can implement this function to return the current size of the regset data. A regset_size() function is added to provide callers with an abstract interface for determining the size of a regset without needing to know whether the regset is dynamically sized or not. The only affected user of this interface is the ELF coredump code: This patch ports ELF coredump to dump regsets with their actual size in the coredump. This has no effect except for new regsets that are dynamically sized and provide a get_size() implementation. Signed-off-by: Dave Martin--- fs/binfmt_elf.c| 6 ++--- include/linux/regset.h | 67 -- 2 files changed, 63 insertions(+), 10 deletions(-) diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 879ff9c..16d2403 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -1697,7 +1697,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t, long signr, size_t *total) { unsigned int i; - unsigned int regset_size = view->regsets[0].n * view->regsets[0].size; + unsigned int size = regset_size(t->task, >regsets[0]); /* * NT_PRSTATUS is the one special case, because the regset data @@ -1706,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t, * We assume that regset 0 is NT_PRSTATUS. */ fill_prstatus(>prstatus, t->task, signr); - (void) view->regsets[0].get(t->task, >regsets[0], 0, regset_size, + (void) view->regsets[0].get(t->task, >regsets[0], 0, size, >prstatus.pr_reg, NULL); fill_note(>notes[0], "CORE", NT_PRSTATUS, @@ -1726,7 +1726,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t, if (regset->core_note_type && regset->get && (!regset->active || regset->active(t->task, regset))) { int ret; - size_t size = regset->n * regset->size; + size_t size = regset_size(t->task, regset); void *data = kmalloc(size, GFP_KERNEL); if (unlikely(!data)) return 0; diff --git a/include/linux/regset.h b/include/linux/regset.h index 8e0c9fe..494ceda 100644 --- a/include/linux/regset.h +++ b/include/linux/regset.h @@ -107,6 +107,28 @@ typedef int user_regset_writeback_fn(struct task_struct *target, int immediate); /** + * user_regset_get_size_fn - type of @get_size function in user_regset + * @target:thread being examined + * @regset:regset being examined + * + * This call is optional; usually the pointer is %NULL. + * + * When provided, this function must return the current size of regset + * data, as observed by the @get function in user_regset. The + * value returned must be a multiple of @size. The returned size is + * required to be valid only until the next time (if any) @regset is + * modified for @target. + * + * This function is intended for dynamically sized regsets. A regset + * that is statically sized does not need to implement it. + * + * This function should not be called directly: instead, callers should + * call regset_size() to determine the current size of a regset. + */ +typedef unsigned int user_regset_get_size_fn(struct task_struct *target, +const struct user_regset *regset); + +/** * struct user_regset - accessible thread CPU state * @n: Number of slots (registers). * @size: Size in bytes of a slot (register). @@ -117,19 +139,33 @@ typedef int user_regset_writeback_fn(struct task_struct *target, * @set: Function to store values. * @active:Function to report if regset is active, or %NULL. * @writeback: Function to write data back to user memory, or %NULL. + * @get_size: Function to return the regset's size, or %NULL. * * This data structure describes a machine resource we call a register set. * This is part of the state of an individual thread, not necessarily * actual CPU registers per se. A register set consists of a number of * similar slots, given by @n. Each slot is @size bytes, and aligned to - * @align bytes (which is at least @size). + * @align bytes (which is at least @size). For dynamically-sized + * regsets, @n must contain the maximum possible number of slots for the + * regset, and @get_size must point to a function that returns the
[PATCH 07/27] arm64/sve: Low-level SVE architectural state manipulation functions
Manipulating the SVE architectural state, including the vector and predicate registers, first-fault register and the vector length, requires the use of dedicated instructions added by SVE. This patch adds suitable assembly functions for saving and restoring the SVE registers and querying the vector length. Setting of the vector length is done as part of register restore. Since people building kernels may not all get an SVE-enabled toolchain for a while, this patch uses macros that generate explicit opcodes in place of assembler mnemonics. Signed-off-by: Dave Martin--- arch/arm64/include/asm/fpsimd.h | 5 ++ arch/arm64/include/asm/fpsimdmacros.h | 137 ++ arch/arm64/kernel/entry-fpsimd.S | 17 + 3 files changed, 159 insertions(+) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 410c481..026a7c7 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -67,6 +67,11 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state); extern void fpsimd_flush_task_state(struct task_struct *target); +extern void sve_save_state(void *state, u32 *pfpsr); +extern void sve_load_state(void const *state, u32 const *pfpsr, + unsigned long vq_minus_1); +extern unsigned int sve_get_vl(void); + /* For use by EFI runtime services calls only */ extern void __efi_fpsimd_begin(void); extern void __efi_fpsimd_end(void); diff --git a/arch/arm64/include/asm/fpsimdmacros.h b/arch/arm64/include/asm/fpsimdmacros.h index 0f5fdd3..5023064 100644 --- a/arch/arm64/include/asm/fpsimdmacros.h +++ b/arch/arm64/include/asm/fpsimdmacros.h @@ -75,3 +75,140 @@ ldr w\tmpnr, [\state, #16 * 2 + 4] fpsimd_restore_fpcr x\tmpnr, \state .endm + +.macro _check_general_reg nr + .if (\nr) < 0 || (\nr) > 30 + .error "Bad register number \nr." + .endif +.endm + +.macro _sve_check_zreg znr + .if (\znr) < 0 || (\znr) > 31 + .error "Bad Scalable Vector Extension vector register number \znr." + .endif +.endm + +.macro _sve_check_preg pnr + .if (\pnr) < 0 || (\pnr) > 15 + .error "Bad Scalable Vector Extension predicate register number \pnr." + .endif +.endm + +.macro _check_num n, min, max + .if (\n) < (\min) || (\n) > (\max) + .error "Number \n out of range [\min,\max]" + .endif +.endm + +.macro _sve_str_v nz, nxbase, offset=0 + _sve_check_zreg \nz + _check_general_reg \nxbase + _check_num (\offset), -0x100, 0xff + .inst 0xe5804000 \ + | (\nz) \ + | ((\nxbase) << 5) \ + | (((\offset) & 7) << 10) \ + | (((\offset) & 0x1f8) << 13) +.endm + +.macro _sve_ldr_v nz, nxbase, offset=0 + _sve_check_zreg \nz + _check_general_reg \nxbase + _check_num (\offset), -0x100, 0xff + .inst 0x85804000 \ + | (\nz) \ + | ((\nxbase) << 5) \ + | (((\offset) & 7) << 10) \ + | (((\offset) & 0x1f8) << 13) +.endm + +.macro _sve_str_p np, nxbase, offset=0 + _sve_check_preg \np + _check_general_reg \nxbase + _check_num (\offset), -0x100, 0xff + .inst 0xe580 \ + | (\np) \ + | ((\nxbase) << 5) \ + | (((\offset) & 7) << 10) \ + | (((\offset) & 0x1f8) << 13) +.endm + +.macro _sve_ldr_p np, nxbase, offset=0 + _sve_check_preg \np + _check_general_reg \nxbase + _check_num (\offset), -0x100, 0xff + .inst 0x8580 \ + | (\np) \ + | ((\nxbase) << 5) \ + | (((\offset) & 7) << 10) \ + | (((\offset) & 0x1f8) << 13) +.endm + +.macro _sve_rdvl nx, imm + _check_general_reg \nx + _check_num (\imm), -0x20, 0x1f + .inst 0x04bf5000 \ + | (\nx) \ + | (((\imm) & 0x3f) << 5) +.endm + +.macro _sve_rdffr np + _sve_check_preg \np + .inst 0x2519f000 \ + | (\np) +.endm + +.macro _sve_wrffr np + _sve_check_preg \np + .inst 0x25289000 \ + | ((\np) << 5) +.endm + +.macro __for from:req, to:req + .if (\from) == (\to) + _for__body \from + .else + __for \from, (\from) + ((\to) - (\from)) / 2 + __for (\from) + ((\to) - (\from)) / 2 + 1, \to + .endif +.endm + +.macro _for var:req, from:req, to:req, insn:vararg + .macro _for__body \var:req + \insn + .endm + + __for
[PATCH 08/27] arm64/sve: Kconfig update and conditional compilation support
This patch adds CONFIG_ARM64_SVE to control building of SVE support into the kernel, and adds a stub predicate system_supports_sve() to control conditional compilation and runtime SVE support. system_supports_sve() just returns false for now: it will be replaced with a non-trivial implementation in a later patch, once SVE support is complete enough to be enabled safely. Signed-off-by: Dave Martin--- arch/arm64/Kconfig | 11 +++ arch/arm64/include/asm/cpufeature.h | 5 + 2 files changed, 16 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fdc302e..fbf7a5c 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -962,6 +962,17 @@ config ARM64_UAO endmenu +config ARM64_SVE + bool "ARM Scalable Vector Extension support" + default y + help + The Scalable Vector Extension (SVE) is an extension to the AArch64 + execution state which complements and extends the SIMD functionality + of the base architecture to support much larger vectors and to enable + additional vectorisation opportunities. + + To enable use of this extension on CPUs that implement it, say Y. + config ARM64_MODULE_CMODEL_LARGE bool diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 428ee1f..4ea3441 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -262,6 +262,11 @@ static inline bool system_uses_ttbr0_pan(void) !cpus_have_const_cap(ARM64_HAS_PAN); } +static inline bool system_supports_sve(void) +{ + return false; +} + #endif /* __ASSEMBLY__ */ #endif -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 11/27] arm64/sve: Core task context handling
This patch adds the core support for switching and managing the SVE architectural state of user tasks. Calls to the existing FPSIMD low-level save/restore functions are factored out as new functions task_fpsimd_{save,load}(), since SVE now dynamically may or may not need to be handled at these points depending on the kernel configuration, hardware features discovered at boot, and the runtime state of the task. To make these decisions as fast as possible, const cpucaps are used where feasible, via the system_supports_sve() helper. The SVE registers are only tracked for threads that have explicitly used SVE, indicated by the new thread flag TIF_SVE. Otherwise, the FPSIMD view of the architectural state is stored in thread.fpsimd_state as usual. When in use, the SVE registers are not stored directly in thread_struct due to their potentially large and variable size. Because the task_struct slab allocator must be configured very early during kernel boot, it is also tricky to configure it correctly to match the maximum vector length provided by the hardware, since this depends on examining secondary CPUs as well as the primary. Instead, a pointer sve_state in thread_struct points to a dynamically allocated buffer containing the SVE register data, and code is added to allocate, duplicate and free this buffer at appropriate times. TIF_SVE is set when taking an SVE access trap from userspace, if suitable hardware support has been detected. This enables SVE for the thread: a subsequent return to userspace will disable the trap accordingly. If such a trap is taken without sufficient hardware support, SIGILL is sent to the thread instead as if an undefined instruction had been executed: this may happen if userspace tries to use SVE in a system where not all CPUs support it for example. The kernel may clear TIF_SVE and disable SVE for the thread whenever an explicit syscall is made by userspace, though this is considered an optimisation opportunity rather than a deterministic guarantee: the kernel may not do this on every syscall, but it is permitted to do so. For backwards compatibility reasons and conformance with the spirit of the base AArch64 procedure call standard, the subset of the SVE register state that aliases the FPSIMD registers is still preserved across a syscall even if this happens. TIF_SVE is also cleared, and SVE disabled, on exec: this is an obvious slow path and a hint that we are running a new binary that may not use SVE. Code is added to sync data between thread.fpsimd_state and thread.sve_state whenever enabling/disabling SVE, in a manner consistent with the SVE architectural programmer's model. Signed-off-by: Dave Martin--- arch/arm64/include/asm/fpsimd.h | 19 +++ arch/arm64/include/asm/processor.h | 2 + arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/include/asm/traps.h | 2 + arch/arm64/kernel/entry.S| 14 +- arch/arm64/kernel/fpsimd.c | 241 ++- arch/arm64/kernel/process.c | 6 +- arch/arm64/kernel/traps.c| 4 +- 8 files changed, 279 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 026a7c7..72090a1 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -20,6 +20,8 @@ #ifndef __ASSEMBLY__ +#include + /* * FP/SIMD storage area has: * - FPSR and FPCR @@ -72,6 +74,23 @@ extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); +#ifdef CONFIG_ARM64_SVE + +extern size_t sve_state_size(struct task_struct const *task); + +extern void sve_alloc(struct task_struct *task); +extern void fpsimd_release_thread(struct task_struct *task); +extern void fpsimd_dup_sve(struct task_struct *dst, + struct task_struct const *src); + +#else /* ! CONFIG_ARM64_SVE */ + +static void __maybe_unused sve_alloc(struct task_struct *task) { } +static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { } +static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst, + struct task_struct const *src) { } +#endif /* ! CONFIG_ARM64_SVE */ + /* For use by EFI runtime services calls only */ extern void __efi_fpsimd_begin(void); extern void __efi_fpsimd_end(void); diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index b7334f1..969feed 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -85,6 +85,8 @@ struct thread_struct { unsigned long tp2_value; #endif struct fpsimd_state fpsimd_state; + void*sve_state; /* SVE registers, if any */ + u16 sve_vl; /* SVE vector length */ unsigned long fault_address; /*
[PATCH 09/27] arm64/sve: Signal frame and context structure definition
This patch defines the representation that will be used for the SVE register state in the signal frame, and implements support for saving and restoring the SVE registers around signals. The same layout will also be used for the in-kernel task state. Due to the variability of the SVE vector length, it is not possible to define a fixed C struct to describe all the registers. Instead, Macros are defined in sigcontext.h to facilitate access to the parts of the structure. Signed-off-by: Dave Martin--- arch/arm64/include/uapi/asm/sigcontext.h | 113 ++- 1 file changed, 112 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/uapi/asm/sigcontext.h b/arch/arm64/include/uapi/asm/sigcontext.h index f0a76b9..0533bdf 100644 --- a/arch/arm64/include/uapi/asm/sigcontext.h +++ b/arch/arm64/include/uapi/asm/sigcontext.h @@ -16,6 +16,8 @@ #ifndef _UAPI__ASM_SIGCONTEXT_H #define _UAPI__ASM_SIGCONTEXT_H +#ifndef __ASSEMBLY__ + #include /* @@ -41,10 +43,11 @@ struct sigcontext { * * 0x210 fpsimd_context * 0x10 esr_context + * 0x8a0 sve_context (vl <= 64) (optional) * 0x20 extra_context (optional) * 0x10 terminator (null _aarch64_ctx) * - * 0xdb0 (reserved for future allocation) + * 0x510 (reserved for future allocation) * * New records that can exceed this space need to be opt-in for userspace, so * that an expanded signal frame is not generated unexpectedly. The mechanism @@ -116,4 +119,112 @@ struct extra_context { __u32 __reserved[3]; }; +#define SVE_MAGIC 0x53564501 + +struct sve_context { + struct _aarch64_ctx head; + __u16 vl; + __u16 __reserved[3]; +}; + +#endif /* !__ASSEMBLY__ */ + +/* + * The SVE architecture leaves space for future expansion of the + * vector length beyond its initial architectural limit of 2048 bits + * (16 quadwords). + */ +#define SVE_VQ_MIN 1 +#define SVE_VQ_MAX 0x200 + +#define SVE_VL_MIN (SVE_VQ_MIN * 0x10) +#define SVE_VL_MAX (SVE_VQ_MAX * 0x10) + +#define SVE_NUM_ZREGS 32 +#define SVE_NUM_PREGS 16 + +#define sve_vl_valid(vl) \ + ((vl) % 0x10 == 0 && (vl) >= SVE_VL_MIN && (vl) <= SVE_VL_MAX) +#define sve_vq_from_vl(vl) ((vl) / 0x10) +#define sve_vl_from_vq(vq) ((vq) * 0x10) + +/* + * If the SVE registers are currently live for the thread at signal delivery, + * sve_context.head.size >= + * SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)) + * and the register data may be accessed using the SVE_SIG_*() macros. + * + * If sve_context.head.size < + * SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)), + * the SVE registers were not live for the thread and no register data + * is included: in this case, the SVE_SIG_*() macros should not be + * used except for this check. + * + * The same convention applies when returning from a signal: a caller + * will need to remove or resize the sve_context block if it wants to + * make the SVE registers live when they were previously non-live or + * vice-versa. This may require the the caller to allocate fresh + * memory and/or move other context blocks in the signal frame. + * + * Changing the vector length during signal return is not permitted: + * sve_context.vl must equal the thread's current vector length when + * doing a sigreturn. + * + * + * Note: for all these macros, the "vq" argument denotes the SVE + * vector length in quadwords (i.e., units of 128 bits). + * + * The correct way to obtain vq is to use sve_vq_from_vl(vl). The + * result is valid if and only if sve_vl_valid(vl) is true. This is + * guaranteed for a struct sve_context written by the kernel. + * + * + * Additional macros describe the contents and layout of the payload. + * For each, SVE_SIG_x_OFFSET(args) is the start offset relative to + * the start of struct sve_context, and SVE_SIG_x_SIZE(args) is the + * size in bytes: + * + * x typedescription + * - --- + * REGSthe entire SVE context + * + * ZREGS __uint128_t[SVE_NUM_ZREGS][vq] all Z-registers + * ZREG__uint128_t[vq] individual Z-register Zn + * + * PREGS uint16_t[SVE_NUM_PREGS][vq] all P-registers + * PREGuint16_t[vq]individual P-register Pn + * + * FFR uint16_t[vq]first-fault status register + * + * Additional data might be appended in the future. + */ + +#define SVE_SIG_ZREG_SIZE(vq) ((__u32)(vq) * 16) +#define SVE_SIG_PREG_SIZE(vq) ((__u32)(vq) * 2) +#define SVE_SIG_FFR_SIZE(vq) SVE_SIG_PREG_SIZE(vq) + +#define SVE_SIG_REGS_OFFSET((sizeof(struct sve_context) + 15) / 16 * 16) + +#define SVE_SIG_ZREGS_OFFSET SVE_SIG_REGS_OFFSET +#define SVE_SIG_ZREG_OFFSET(vq, n) \ +
[PATCH 10/27] arm64/sve: Low-level CPU setup
To enable the kernel to use SVE, all SVE traps from EL1 must be disabled. To take maximum advantage of the hardware, the full available vector length also needs to be enabled for EL1 by programming ZCR_EL2.LEN. (The kernel will program ZCR_EL1.LEN as required, but this cannot override the limit set by ZCR_EL2.) In advance of full SVE support being implemented for userspace, it also necessary to ensure that SVE traps from EL0 are enabled. This patch makes the appropriate changes to the primary and secondary CPU initialisation code. Signed-off-by: Dave Martin--- arch/arm64/kernel/head.S | 13 - arch/arm64/mm/proc.S | 14 -- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 973df7d..0ae1713 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -514,8 +514,19 @@ CPU_LE(movkx0, #0x30d0, lsl #16) // Clear EE and E0E on LE systems mov x0, #0x33ff msr cptr_el2, x0// Disable copro. traps to EL2 + /* SVE register access */ + mrs x1, id_aa64pfr0_el1 + ubfxx1, x1, #ID_AA64PFR0_SVE_SHIFT, #4 + cbz x1, 7f + + bic x0, x0, #CPTR_EL2_TZ// Also disable SVE traps + msr cptr_el2, x0// Disable copro. traps to EL2 + isb + mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector + msr_s SYS_ZCR_EL2, x1 // length for EL1. + /* Hypervisor stub */ - adr_l x0, __hyp_stub_vectors +7: adr_l x0, __hyp_stub_vectors msr vbar_el2, x0 /* spsr */ diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 877d42f..dd22ef2 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -27,6 +27,7 @@ #include #include #include +#include #ifdef CONFIG_ARM64_64K_PAGES #define TCR_TG_FLAGS TCR_TG0_64K | TCR_TG1_64K @@ -186,8 +187,17 @@ ENTRY(__cpu_setup) tlbivmalle1 // Invalidate local TLB dsb nsh - mov x0, #3 << 20 - msr cpacr_el1, x0 // Enable FP/ASIMD + mov x0, #3 << 20// FEN + + /* SVE */ + mrs x5, id_aa64pfr0_el1 + ubfxx5, x5, #ID_AA64PFR0_SVE_SHIFT, #4 + cbz x5, 1f + + bic x0, x0, #CPACR_EL1_ZEN + orr x0, x0, #CPACR_EL1_ZEN_EL1EN// SVE: trap for EL0, not EL1 +1: msr cpacr_el1, x0 // Enable FP/ASIMD + mov x0, #1 << 12// Reset mdscr_el1 and disable msr mdscr_el1, x0 // access to the DCC from EL0 isb // Unmask debug exceptions now, -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 12/27] arm64/sve: Support vector length resetting for new processes
It's desirable to be able to reset the vector length to some sane default for new processes, since the new binary and its libraries processes may or may not be SVE-aware. This patch tracks the desired post-exec vector length (if any) in a new thread member sve_vl_onexec, and adds a new thread flag TIF_SVE_VL_INHERIT to control whether to inherit or reset the vector length. Currently these are inactive. Subsequent patches will provide the capability to configure them. Signed-off-by: Dave Martin--- arch/arm64/include/asm/processor.h | 1 + arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/fpsimd.c | 13 + 3 files changed, 15 insertions(+) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 969feed..da8802a 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -87,6 +87,7 @@ struct thread_struct { struct fpsimd_state fpsimd_state; void*sve_state; /* SVE registers, if any */ u16 sve_vl; /* SVE vector length */ + u16 sve_vl_onexec; /* SVE vl after next exec */ unsigned long fault_address; /* fault info */ unsigned long fault_code; /* ESR_EL1 value */ struct debug_info debug; /* debugging */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index 1a4b30b..bf9c552 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -97,6 +97,7 @@ struct thread_info { #define TIF_SINGLESTEP 21 #define TIF_32BIT 22 /* 32bit process */ #define TIF_SVE23 /* Scalable Vector Extension in use */ +#define TIF_SVE_VL_INHERIT 24 /* Inherit sve_vl_onexec across exec */ #define _TIF_SIGPENDING(1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 37dd1b2..80ecb2d 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -108,6 +108,9 @@ static void task_fpsimd_save(void); */ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state); +/* Default VL for tasks that don't set it explicitly: */ +static int sve_default_vl = -1; + static void sve_free(struct task_struct *task) { kfree(task->thread.sve_state); @@ -392,6 +395,9 @@ void fpsimd_flush_thread(void) clear_thread_flag(TIF_SVE); sve_free(current); + current->thread.sve_vl = current->thread.sve_vl_onexec ? + current->thread.sve_vl_onexec : sve_default_vl; + /* * User tasks must have a valid vector length set, but tasks * forked early (e.g., init) may not initially have one. @@ -401,6 +407,13 @@ void fpsimd_flush_thread(void) * If not, something went badly wrong. */ BUG_ON(!sve_vl_valid(current->thread.sve_vl)); + + /* +* If the task is not set to inherit, ensure that the vector +* length will be reset by a subsequent exec: +*/ + if (!test_thread_flag(TIF_SVE_VL_INHERIT)) + current->thread.sve_vl_onexec = 0; } set_thread_flag(TIF_FOREIGN_FPSTATE); -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 14/27] arm64/sve: Backend logic for setting the vector length
This patch implements the core logic for changing a task's vector length on request from userspace. This will be used by the ptrace and prctl frontends that are implemented in later patches. The SVE architecture permits, but does not require, implementations to support vector lengths that are not a power of two. To handle this, logic is added to check a requested vector length against a possibly sparse bitmap of available vector lengths at runtime, so that the best supported value can be chosen. Signed-off-by: Dave Martin--- arch/arm64/include/asm/fpsimd.h | 6 +++ arch/arm64/kernel/fpsimd.c | 116 include/uapi/linux/prctl.h | 5 ++ 3 files changed, 127 insertions(+) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 7efd04e..39b26d2 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -70,11 +70,15 @@ extern void fpsimd_update_current_state(struct fpsimd_state *state); extern void fpsimd_flush_task_state(struct task_struct *target); +#define SVE_VL_ARCH_MAX 0x100 + extern void sve_save_state(void *state, u32 *pfpsr); extern void sve_load_state(void const *state, u32 const *pfpsr, unsigned long vq_minus_1); extern unsigned int sve_get_vl(void); +extern int sve_max_vl; + #ifdef CONFIG_ARM64_SVE extern size_t sve_state_size(struct task_struct const *task); @@ -83,6 +87,8 @@ extern void sve_alloc(struct task_struct *task); extern void fpsimd_release_thread(struct task_struct *task); extern void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src); +extern int sve_set_vector_length(struct task_struct *task, +unsigned long vl, unsigned long flags); #else /* ! CONFIG_ARM64_SVE */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index e8674f6..bce95de 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -18,12 +18,14 @@ */ #include +#include #include #include #include #include #include #include +#include #include #include #include @@ -111,6 +113,20 @@ static DEFINE_PER_CPU(struct fpsimd_state *, fpsimd_last_state); /* Default VL for tasks that don't set it explicitly: */ static int sve_default_vl = -1; +#ifdef CONFIG_ARM64_SVE + +/* Maximum supported vector length across all CPUs (initially poisoned) */ +int sve_max_vl = -1; +/* Set of available vector lengths, as vq_to_bit(vq): */ +static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); + +#else /* ! CONFIG_ARM64_SVE */ + +/* Dummy declaration for code that will be optimised out: */ +extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); + +#endif /* ! CONFIG_ARM64_SVE */ + static void sve_free(struct task_struct *task) { kfree(task->thread.sve_state); @@ -148,6 +164,37 @@ static void change_cpacr(u64 old, u64 new) write_sysreg(new, CPACR_EL1); } +static unsigned int vq_to_bit(unsigned int vq) +{ + BUILD_BUG_ON(vq < 1 || vq > SVE_VQ_MAX); + + return SVE_VQ_MAX - vq; +} + +static unsigned int bit_to_vq(unsigned int bit) +{ + BUILD_BUG_ON(bit >= SVE_VQ_MAX); + + return SVE_VQ_MAX - bit; +} + +static unsigned int find_supported_vector_length(unsigned int vl) +{ + int bit; + + BUG_ON(!sve_vl_valid(vl)); + + BUG_ON(!sve_vl_valid(sve_max_vl)); + if (vl > sve_max_vl) + vl = sve_max_vl; + + bit = find_next_bit(sve_vq_map, SVE_VQ_MAX, + vq_to_bit(sve_vq_from_vl(vl))); + BUG_ON(bit < 0 || bit >= SVE_VQ_MAX); + + return 16 * bit_to_vq(bit); +} + #define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) @@ -235,6 +282,73 @@ void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src) } } +int sve_set_vector_length(struct task_struct *task, + unsigned long vl, unsigned long flags) +{ + BUG_ON(task == current && preemptible()); + + if (flags & ~(unsigned long)(PR_SVE_VL_INHERIT | +PR_SVE_SET_VL_ONEXEC)) + return -EINVAL; + + if (!sve_vl_valid(vl)) + return -EINVAL; + + /* +* Clamp to the maximum vector length that VL-agnostic SVE code can +* work with. A flag may be assigned in the future to allow setting +* of larger vector lengths without confusing older software. +*/ + if (vl > SVE_VL_ARCH_MAX) + vl = SVE_VL_ARCH_MAX; + + vl = find_supported_vector_length(vl); + + if (flags & (PR_SVE_VL_INHERIT | +PR_SVE_SET_VL_ONEXEC)) + task->thread.sve_vl_onexec = vl; + else + /* Reset VL to system default on next exec: */ + task->thread.sve_vl_onexec = 0; + + /* Only actually set the
[PATCH 13/27] arm64/sve: Signal handling support
This patch implements support for saving and restoring the SVE registers around signals. A fixed-size header struct sve_context is always included in the signal frame encoding the thread's vector length at the time of signal delivery, optionally followed by a variable-layout structure encoding the SVE registers. Because of the need to preserve backwards compatibility, the FPSIMD view of the SVE registers is always dumped as a struct fpsimd_context in the usual way, in addition to any sve_context. The SVE vector registers are dumped in full, including bits 127:0 of each register which alias the corresponding FPSIMD vector registers in the hardware. To avoid any ambiguity about which alias to restore during sigreturn, the kernel always restores bits 127:0 of each SVE vector register from the fpsimd_context in the signal frame (which must be present): userspace needs to take this into account if it wants to modify the SVE vector register contents on return from a signal. FPSR and FPCR, which are used by both FPSIMD and SVE, are not included in sve_context because they are always present in fpsimd_context anyway. For signal delivery, a new helper fpsimd_signal_preserve_current_state() is added to update _both_ the FPSIMD and SVE views in the task struct, to make it easier to populate this information into the signal frame. Because of the redundancy between the two views of the state, only one is updated otherwise. In order to avoid racing with a pending discard of the SVE state, this flush is hoisted before the sigframe layout phase, so that the layout and population phases see a consistent view of the thread. Signed-off-by: Dave Martin--- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/fpsimd.c | 23 -- arch/arm64/kernel/signal.c | 169 ++-- arch/arm64/kernel/signal32.c| 2 +- 4 files changed, 179 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 72090a1..7efd04e 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -63,6 +63,7 @@ extern void fpsimd_load_state(struct fpsimd_state *state); extern void fpsimd_thread_switch(struct task_struct *next); extern void fpsimd_flush_thread(void); +extern void fpsimd_signal_preserve_current_state(void); extern void fpsimd_preserve_current_state(void); extern void fpsimd_restore_current_state(void); extern void fpsimd_update_current_state(struct fpsimd_state *state); diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 80ecb2d..e8674f6 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -148,8 +148,6 @@ static void change_cpacr(u64 old, u64 new) write_sysreg(new, CPACR_EL1); } -#ifdef CONFIG_ARM64_SVE - #define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) @@ -191,6 +189,8 @@ static void fpsimd_to_sve(struct task_struct *task) sizeof(fst->vregs[i])); } +#ifdef CONFIG_ARM64_SVE + size_t sve_state_size(struct task_struct const *task) { unsigned int vl = task->thread.sve_vl; @@ -431,13 +431,17 @@ void fpsimd_preserve_current_state(void) return; local_bh_disable(); - - if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(>thread.fpsimd_state); - + task_fpsimd_save(); local_bh_enable(); } +void fpsimd_signal_preserve_current_state(void) +{ + fpsimd_preserve_current_state(); + if (system_supports_sve() && test_thread_flag(TIF_SVE)) + sve_to_fpsimd(current); +} + /* * Load the userland FPSIMD state of 'current' from memory, but only if the * FPSIMD state already held in the registers is /not/ the most recent FPSIMD @@ -473,7 +477,12 @@ void fpsimd_update_current_state(struct fpsimd_state *state) local_bh_disable(); - fpsimd_load_state(state); + if (system_supports_sve() && test_thread_flag(TIF_SVE)) { + current->thread.fpsimd_state = *state; + fpsimd_to_sve(current); + } + task_fpsimd_load(); + if (test_and_clear_thread_flag(TIF_FOREIGN_FPSTATE)) { struct fpsimd_state *st = >thread.fpsimd_state; diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 4991e87..2694143 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -62,6 +62,7 @@ struct rt_sigframe_user_layout { unsigned long fpsimd_offset; unsigned long esr_offset; + unsigned long sve_offset; unsigned long extra_offset; unsigned long end_offset; }; @@ -178,9 +179,6 @@ static int preserve_fpsimd_context(struct fpsimd_context __user *ctx) struct fpsimd_state *fpsimd = >thread.fpsimd_state; int err; - /* dump the hardware registers to the fpsimd_state
[PATCH 15/27] arm64/sve: Probe SVE capabilities and usable vector lengths
This patch uses the cpufeatures framework to determine common SVE capabilities and vector lengths, and configures the runtime SVE support code appropriately. ZCR_ELx is not really a feature register, but it is convenient to use it as a template for recording the maximum vector length supported by a CPU, using the LEN field. This field is similar to a feature field in that it is a contiguous bitfield for which we want to determine the minimum system-wide value. This patch adds ZCR as a pseudo-register in cpuinfo/cpufeatures, with appropriate custom code to populate it. Finding the minimum supported value of the LEN field is left to the cpufeatures framework in the usual way. The meaning of ID_AA64ZFR0_EL1 is not architecturally defined yet, so for now we just require it to be zero. Note that much of this code is dormant and SVE still won't be used yet, since system_supports_sve() remains hardwired to false. Signed-off-by: Dave Martin--- arch/arm64/include/asm/cpu.h| 4 ++ arch/arm64/include/asm/cpufeature.h | 28 ++ arch/arm64/include/asm/fpsimd.h | 10 arch/arm64/kernel/cpufeature.c | 48 arch/arm64/kernel/cpuinfo.c | 6 ++ arch/arm64/kernel/fpsimd.c | 108 6 files changed, 204 insertions(+) diff --git a/arch/arm64/include/asm/cpu.h b/arch/arm64/include/asm/cpu.h index 889226b..8839227 100644 --- a/arch/arm64/include/asm/cpu.h +++ b/arch/arm64/include/asm/cpu.h @@ -41,6 +41,7 @@ struct cpuinfo_arm64 { u64 reg_id_aa64mmfr2; u64 reg_id_aa64pfr0; u64 reg_id_aa64pfr1; + u64 reg_id_aa64zfr0; u32 reg_id_dfr0; u32 reg_id_isar0; @@ -59,6 +60,9 @@ struct cpuinfo_arm64 { u32 reg_mvfr0; u32 reg_mvfr1; u32 reg_mvfr2; + + /* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */ + u64 reg_zcr; }; DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data); diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 4ea3441..05eec27 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -10,6 +10,7 @@ #define __ASM_CPUFEATURE_H #include +#include #include #include @@ -223,6 +224,13 @@ static inline bool id_aa64pfr0_32bit_el0(u64 pfr0) return val == ID_AA64PFR0_EL0_32BIT_64BIT; } +static inline bool id_aa64pfr0_sve(u64 pfr0) +{ + u32 val = cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_SVE_SHIFT); + + return val > 0; +} + void __init setup_cpu_features(void); void update_cpu_capabilities(const struct arm64_cpu_capabilities *caps, @@ -267,6 +275,26 @@ static inline bool system_supports_sve(void) return false; } +/* + * Read the pseudo-ZCR used by cpufeatures to identify the supported SVE + * vector length. + * Use only if SVE is present. This function clobbers the SVE vector length. + */ +static u64 __maybe_unused read_zcr_features(void) +{ + u64 zcr; + unsigned int vq_max; + + write_sysreg_s(ZCR_ELx_LEN_MASK, SYS_ZCR_EL1); + + zcr = read_sysreg_s(SYS_ZCR_EL1); + zcr &= ~(u64)ZCR_ELx_LEN_MASK; + vq_max = sve_get_vl() / 16; + zcr |= vq_max - 1; + + return zcr; +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 39b26d2..f43f573 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -90,12 +90,22 @@ extern void fpsimd_dup_sve(struct task_struct *dst, extern int sve_set_vector_length(struct task_struct *task, unsigned long vl, unsigned long flags); +extern void __init sve_init_vq_map(void); +extern void sve_update_vq_map(void); +extern int sve_verify_vq_map(void); +extern void __init sve_setup(void); + #else /* ! CONFIG_ARM64_SVE */ static void __maybe_unused sve_alloc(struct task_struct *task) { } static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { } static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src) { } +static void __maybe_unused sve_init_vq_map(void) { } +static void __maybe_unused sve_update_vq_map(void) { } +static int __maybe_unused sve_verify_vq_map(void) { return 0; } +static void __maybe_unused sve_setup(void) { } + #endif /* ! CONFIG_ARM64_SVE */ /* For use by EFI runtime services calls only */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9f9e0064..f322c90 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -267,6 +268,12 @@ static const struct arm64_ftr_bits ftr_id_dfr0[] = {
[PATCH 16/27] arm64/sve: Preserve SVE registers around kernel-mode NEON use
Kernel-mode NEON will corrupt the SVE vector registers, due to the way they alias the FPSIMD vector registers in the hardware. This patch ensures that any live SVE register content for the task is saved by kernel_neon_begin(). The data will be restored in the usual way on return to userspace. Signed-off-by: Dave Martin--- arch/arm64/kernel/fpsimd.c | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 955c873..b7fb836 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -758,8 +758,10 @@ void kernel_neon_begin(void) __this_cpu_write(kernel_neon_busy, true); /* Save unsaved task fpsimd state, if any: */ - if (current->mm && !test_and_set_thread_flag(TIF_FOREIGN_FPSTATE)) - fpsimd_save_state(>thread.fpsimd_state); + if (current->mm) { + task_fpsimd_save(); + set_thread_flag(TIF_FOREIGN_FPSTATE); + } /* Invalidate any task state remaining in the fpsimd regs: */ __this_cpu_write(fpsimd_last_state, NULL); -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 19/27] arm64/sve: Add prctl controls for userspace vector length management
This patch adds two arm64-specific prctls, to permit userspace to control its vector length: * PR_SVE_SET_VL: set the thread's SVE vector length and vector length inheritance mode. * PR_SVE_GET_VL: get the same information. Although these calls shadow instruction set features in the SVE architecture, these prctls provide additional control: the vector length inheritance mode is Linux-specific and nothing to do with the architecture, and the architecture does not permit EL0 to set its own vector length directly. Both can be used in portable tools without requiring the use of SVE instructions. Signed-off-by: Dave Martin--- arch/arm64/include/asm/fpsimd.h| 14 arch/arm64/include/asm/processor.h | 4 arch/arm64/kernel/fpsimd.c | 46 ++ include/uapi/linux/prctl.h | 4 kernel/sys.c | 12 ++ 5 files changed, 80 insertions(+) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index ffb8a50..b58aa45 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -17,6 +17,7 @@ #define __ASM_FP_H #include +#include #ifndef __ASSEMBLY__ @@ -97,6 +98,9 @@ extern void sve_sync_from_fpsimd_zeropad(struct task_struct *task); extern int sve_set_vector_length(struct task_struct *task, unsigned long vl, unsigned long flags); +extern int sve_set_current_vl(unsigned long arg); +extern int sve_get_current_vl(void); + extern void __init sve_init_vq_map(void); extern void sve_update_vq_map(void); extern int sve_verify_vq_map(void); @@ -112,6 +116,16 @@ static void __maybe_unused sve_sync_to_fpsimd(struct task_struct *task) { } static void __maybe_unused sve_sync_from_fpsimd_zeropad( struct task_struct *task) { } +static int __maybe_unused sve_set_current_vl(unsigned long arg) +{ + return -EINVAL; +} + +static int __maybe_unused sve_get_current_vl(void) +{ + return -EINVAL; +} + static void __maybe_unused sve_init_vq_map(void) { } static void __maybe_unused sve_update_vq_map(void) { } static int __maybe_unused sve_verify_vq_map(void) { return 0; } diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index da8802a..70b03b4 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -197,4 +197,8 @@ static inline void spin_lock_prefetch(const void *ptr) int cpu_enable_pan(void *__unused); int cpu_enable_cache_maint_trap(void *__unused); +/* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */ +#define SVE_SET_VL(arg)sve_set_current_vl(arg) +#define SVE_GET_VL() sve_get_current_vl() + #endif /* __ASM_PROCESSOR_H */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 5a7c870..4fe1675 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -387,6 +388,51 @@ int sve_set_vector_length(struct task_struct *task, return 0; } +/* + * Encode the current vector length and flags for return. + * This is only required for prctl(): ptrace has separate fields + */ +static int sve_prctl_status(void) +{ + int ret = current->thread.sve_vl; + + if (test_thread_flag(TIF_SVE_VL_INHERIT)) + ret |= PR_SVE_VL_INHERIT; + + return ret; +} + +/* PR_SVE_SET_VL */ +int sve_set_current_vl(unsigned long arg) +{ + unsigned long vl, flags; + int ret; + + vl = arg & PR_SVE_VL_LEN_MASK; + flags = arg & ~vl; + + if (!system_supports_sve()) + return -EINVAL; + + preempt_disable(); + ret = sve_set_vector_length(current, vl, flags); + preempt_enable(); + + if (ret) + return ret; + + return sve_prctl_status(); +} + +/* PR_SVE_GET_VL */ +int sve_get_current_vl(void) +{ + if (!system_supports_sve()) + return -EINVAL; + + return sve_prctl_status(); +} + static unsigned long *sve_alloc_vq_map(void) { return kzalloc(BITS_TO_LONGS(SVE_VQ_MAX) * sizeof(unsigned long), diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 1b64901..1ef9370 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -198,7 +198,11 @@ struct prctl_mm_map { # define PR_CAP_AMBIENT_CLEAR_ALL 4 /* arm64 Scalable Vector Extension controls */ +/* Flag values must be kept in sync with ptrace NT_ARM_SVE interface */ +#define PR_SVE_SET_VL 48 /* set task vector length */ # define PR_SVE_SET_VL_ONEXEC (1 << 18) /* defer effect until exec */ +#define PR_SVE_GET_VL 49 /* get task vector length */ +/* Bits common to PR_SVE_SET_VL and PR_SVE_GET_VL */ # define PR_SVE_VL_LEN_MASK0x # define PR_SVE_VL_INHERIT (1 << 17) /* inherit
[PATCH 18/27] arm64/sve: ptrace and ELF coredump support
This patch defines and implements a new regset NT_ARM_SVE, which describes a thread's SVE register state. This allows a debugger to manipulate the SVE state, as well as being included in ELF coredumps for post-mortem debugging. Because the regset size and layout are dependent on the thread's current vector length, it is not possible to define a C struct to describe the regset contents as is done for existing regsets. Instead, and for the same reasons, NT_ARM_SVE is based on the freeform variable-layout approach used for the SVE signal frame. Additionally, to reduce debug overhead when debugging threads that might or might not have live SVE register state, NT_ARM_SVE may be presented in one of two different formats: the old struct user_fpsimd_state format is embedded for describing the state of a thread with no live SVE state, whereas a new variable-layout structure is embedded for describing live SVE state. This avoids a debugger needing to poll NT_PRFPREG in addition to NT_ARM_SVE, and allows existing userspace code to handle the non-SVE case without too much modification. For this to work, NT_ARM_SVE is defined with a fixed-format header of type struct user_sve_header, which the recipient can use to figure out the content, size and layout of the reset of the regset. Accessor macros are defined to allow the vector-length-dependent parts of the regset to be manipulated. Signed-off-by: Alan HaywardSigned-off-by: Dave Martin --- arch/arm64/include/asm/fpsimd.h | 13 +- arch/arm64/include/uapi/asm/ptrace.h | 130 arch/arm64/kernel/fpsimd.c | 34 + arch/arm64/kernel/ptrace.c | 288 ++- include/uapi/linux/elf.h | 1 + 5 files changed, 457 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index f43f573..ffb8a50 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -37,13 +37,16 @@ struct fpsimd_state { __uint128_t vregs[32]; u32 fpsr; u32 fpcr; + /* +* For ptrace compatibility, pad to next 128-bit +* boundary here if extending this struct. +*/ }; }; /* the id of the last cpu to have restored this state */ unsigned int cpu; }; - #if defined(__KERNEL__) && defined(CONFIG_COMPAT) /* Masks for extracting the FPSR and FPCR from the FPSCR */ #define VFP_FPSCR_STAT_MASK0xf89f @@ -87,6 +90,10 @@ extern void sve_alloc(struct task_struct *task); extern void fpsimd_release_thread(struct task_struct *task); extern void fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src); +extern void fpsimd_sync_to_sve(struct task_struct *task); +extern void sve_sync_to_fpsimd(struct task_struct *task); +extern void sve_sync_from_fpsimd_zeropad(struct task_struct *task); + extern int sve_set_vector_length(struct task_struct *task, unsigned long vl, unsigned long flags); @@ -101,6 +108,10 @@ static void __maybe_unused sve_alloc(struct task_struct *task) { } static void __maybe_unused fpsimd_release_thread(struct task_struct *task) { } static void __maybe_unused fpsimd_dup_sve(struct task_struct *dst, struct task_struct const *src) { } +static void __maybe_unused sve_sync_to_fpsimd(struct task_struct *task) { } +static void __maybe_unused sve_sync_from_fpsimd_zeropad( + struct task_struct *task) { } + static void __maybe_unused sve_init_vq_map(void) { } static void __maybe_unused sve_update_vq_map(void) { } static int __maybe_unused sve_verify_vq_map(void) { return 0; } diff --git a/arch/arm64/include/uapi/asm/ptrace.h b/arch/arm64/include/uapi/asm/ptrace.h index d1ff83d..16d9344 100644 --- a/arch/arm64/include/uapi/asm/ptrace.h +++ b/arch/arm64/include/uapi/asm/ptrace.h @@ -22,6 +22,7 @@ #include #include +#include /* @@ -63,6 +64,8 @@ #ifndef __ASSEMBLY__ +#include + /* * User structures for general purpose, floating point and debug registers. */ @@ -90,6 +93,133 @@ struct user_hwdebug_state { } dbg_regs[16]; }; +/* SVE/FP/SIMD state (NT_ARM_SVE) */ + +struct user_sve_header { + __u32 size; /* total meaningful regset content in bytes */ + __u32 max_size; /* maxmium possible size for this thread */ + __u16 vl; /* current vector length */ + __u16 max_vl; /* maximum possible vector length */ + __u16 flags; + __u16 __reserved; +}; + +/* Definitions for user_sve_header.flags: */ +#define SVE_PT_REGS_MASK (1 << 0) + +/* Flags: must be kept in sync with prctl interface in */ +#define SVE_PT_REGS_FPSIMD 0 +#define SVE_PT_REGS_SVE
[PATCH 23/27] arm64/sve: KVM: Hide SVE from CPU features exposed to guests
KVM guests cannot currently use SVE, because SVE is always configured to trap to EL2. However, a guest that sees SVE reported as present in ID_AA64PFR0_EL1 may legitimately expect that SVE works and try to use it. Instead of working, the guest will receive an injected undef exception, which may cause the guest to oops or go into a spin. To avoid misleading the guest into believing that SVE will work, this patch masks out the SVE field from ID_AA64PFR0_EL1 when a guest attempts to read this register. No support is explicitly added for ID_AA64ZFR0_EL1 either, so that is still emulated as reading as zero, which is consistent with SVE not being implemented. This is a temporary measure, and will be removed in a later series when full KVM support for SVE is implemented. Signed-off-by: Dave Martin--- arch/arm64/kvm/sys_regs.c | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 6583dd7..9e8c54e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -897,8 +897,20 @@ static u64 read_id_reg(struct sys_reg_desc const *r, bool raz) { u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); + u64 val = raz ? 0 : read_sanitised_ftr_reg(id); - return raz ? 0 : read_sanitised_ftr_reg(id); + if (id == SYS_ID_AA64PFR0_EL1) { + static bool printed; + + if ((val & (0xfUL << ID_AA64PFR0_SVE_SHIFT)) && !printed) { + kvm_info("SVE unsupported for guests, suppressing\n"); + printed = true; + } + + val &= ~(0xfUL << ID_AA64PFR0_SVE_SHIFT); + } + + return val; } /* cpufeature ID register access trap handlers */ -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 24/27] arm64/sve: Detect SVE and activate runtime support
This patch enables detection of hardware SVE support via the cpufeatures framework, and reports its presence to the kernel and userspace via the new ARM64_SVE cpucap and HWCAP_SVE hwcap respectively. Userspace can also detect SVE using ID_AA64PFR0_EL1, using the cpufeatures MRS emulation. When running on hardware that supports SVE, this enables runtime kernel support for SVE, and allows user tasks to execute SVE instructions and make of the of the SVE-specific user/kernel interface extensions implemented by this series. Signed-off-by: Dave Martin--- arch/arm64/include/asm/cpucaps.h| 3 ++- arch/arm64/include/asm/cpufeature.h | 3 ++- arch/arm64/include/uapi/asm/hwcap.h | 1 + arch/arm64/kernel/cpufeature.c | 16 arch/arm64/kernel/cpuinfo.c | 1 + 5 files changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 8d2272c..0b4ce6a 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -39,7 +39,8 @@ #define ARM64_WORKAROUND_QCOM_FALKOR_E1003 18 #define ARM64_WORKAROUND_85892119 #define ARM64_WORKAROUND_CAVIUM_30115 20 +#define ARM64_SVE 21 -#define ARM64_NCAPS21 +#define ARM64_NCAPS22 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 05eec27..27182bb 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -272,7 +272,8 @@ static inline bool system_uses_ttbr0_pan(void) static inline bool system_supports_sve(void) { - return false; + return IS_ENABLED(CONFIG_ARM64_SVE) && + cpus_have_const_cap(ARM64_SVE); } /* diff --git a/arch/arm64/include/uapi/asm/hwcap.h b/arch/arm64/include/uapi/asm/hwcap.h index 4e187ce..7d2ec07 100644 --- a/arch/arm64/include/uapi/asm/hwcap.h +++ b/arch/arm64/include/uapi/asm/hwcap.h @@ -35,5 +35,6 @@ #define HWCAP_JSCVT(1 << 13) #define HWCAP_FCMA (1 << 14) #define HWCAP_LRCPC(1 << 15) +#define HWCAP_SVE (1 << 16) #endif /* _UAPI__ASM_HWCAP_H */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f322c90..390cb73 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -125,6 +125,7 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = { }; static const struct arm64_ftr_bits ftr_id_aa64pfr0[] = { + ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_SVE_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_EXACT, ID_AA64PFR0_GIC_SHIFT, 4, 0), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_ASIMD_SHIFT, 4, ID_AA64PFR0_ASIMD_NI), S_ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64PFR0_FP_SHIFT, 4, ID_AA64PFR0_FP_NI), @@ -913,6 +914,18 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .min_field_value = 0, .matches = has_no_fpsimd, }, +#ifdef CONFIG_ARM64_SVE + { + .desc = "Scalable Vector Extension", + .capability = ARM64_SVE, + .def_scope = SCOPE_SYSTEM, + .sys_reg = SYS_ID_AA64PFR0_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64PFR0_SVE_SHIFT, + .min_field_value = ID_AA64PFR0_SVE, + .matches = has_cpuid_feature, + }, +#endif /* CONFIG_ARM64_SVE */ {}, }; @@ -944,6 +957,9 @@ static const struct arm64_cpu_capabilities arm64_elf_hwcaps[] = { HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_JSCVT_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_JSCVT), HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_FCMA_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_FCMA), HWCAP_CAP(SYS_ID_AA64ISAR1_EL1, ID_AA64ISAR1_LRCPC_SHIFT, FTR_UNSIGNED, 1, CAP_HWCAP, HWCAP_LRCPC), +#ifdef CONFIG_ARM64_SVE + HWCAP_CAP(SYS_ID_AA64PFR0_EL1, ID_AA64PFR0_SVE_SHIFT, FTR_UNSIGNED, ID_AA64PFR0_SVE, CAP_HWCAP, HWCAP_SVE), +#endif {}, }; diff --git a/arch/arm64/kernel/cpuinfo.c b/arch/arm64/kernel/cpuinfo.c index 0e087d7..943aad0 100644 --- a/arch/arm64/kernel/cpuinfo.c +++ b/arch/arm64/kernel/cpuinfo.c @@ -69,6 +69,7 @@ static const char *const hwcap_str[] = { "jscvt", "fcma", "lrcpc", + "sve", NULL }; -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[RFC PATCH 26/27] arm64: signal: Report signal frame size to userspace via auxv
Stateful CPU architecture extensions may require the signal frame to grow to a size that exceeds the arch's MINSIGSTKSZ #define. However, changing this #define is an ABI break. To allow userspace the option of determining the signal frame size in a more forwards-compatible way, this patch adds a new auxv entry tagged with AT_MINSIGSTKSZ, which provides the maximum signal frame size that the process can observe during its lifetime. If AT_MINSIGSTKSZ is absent from the aux vector, the caller can assume that the MINSIGSTKSZ #define is sufficient. This allows for a consistent interface with older kernels that do not provide AT_MINSIGSTKSZ. The idea is that libc could expose this via sysconf() or some similar mechanism. There is deliberately no AT_SIGSTKSZ. The kernel knows nothing about userspace's own stack overheads and should not pretend to know. For arm64: The primary motivation for this interface is the Scalable Vector Extension, which can require at least 4KB or so of extra space in the signal frame for the largest hardware implementations. To determine the correct value, a "Christmas tree" mode (via the add_all argument) is added to setup_sigframe_layout(), to simulate addition of all possible records to the signal frame at maximum possible size. If this procedure goes wrong somehow, resulting in a stupidly large frame layout and hence failure of sigframe_alloc() to allocate a record to the frame, then this is indicative of a kernel bug: the kernel's internal SIGFRAME_MAXSZ is supposed to sanity-check against generting frames that we consider _impossibly_ large. In this case, SIGSTKSZ is returned as a "reasonable guess that is at least bigger than MINSIGSTKSZ" and we WARN(). Signed-off-by: Dave Martin--- arch/arm64/include/asm/elf.h | 5 + arch/arm64/include/asm/processor.h | 3 +++ arch/arm64/include/uapi/asm/auxvec.h | 3 ++- arch/arm64/kernel/signal.c | 38 +++- 4 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h index acae781..7cee254 100644 --- a/arch/arm64/include/asm/elf.h +++ b/arch/arm64/include/asm/elf.h @@ -24,6 +24,10 @@ #include #include +#ifndef __ASSEMBLY__ +#include /* for get_minsigstksz(), used by ARCH_DLINFO */ +#endif + /* * AArch64 static relocation types. */ @@ -149,6 +153,7 @@ typedef struct user_fpsimd_state elf_fpregset_t; do { \ NEW_AUX_ENT(AT_SYSINFO_EHDR,\ (elf_addr_t)current->mm->context.vdso); \ + NEW_AUX_ENT(AT_MINSIGSTKSZ, get_minsigstksz()); \ } while (0) #define ARCH_HAS_SETUP_ADDITIONAL_PAGES diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 70b03b4..121d011 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -197,6 +197,9 @@ static inline void spin_lock_prefetch(const void *ptr) int cpu_enable_pan(void *__unused); int cpu_enable_cache_maint_trap(void *__unused); +/* User signal frame size discovery: */ +int get_minsigstksz(void); + /* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */ #define SVE_SET_VL(arg)sve_set_current_vl(arg) #define SVE_GET_VL() sve_get_current_vl() diff --git a/arch/arm64/include/uapi/asm/auxvec.h b/arch/arm64/include/uapi/asm/auxvec.h index 4cf0c17..1d45b28 100644 --- a/arch/arm64/include/uapi/asm/auxvec.h +++ b/arch/arm64/include/uapi/asm/auxvec.h @@ -18,7 +18,8 @@ /* vDSO location */ #define AT_SYSINFO_EHDR33 +#define AT_MINSIGSTKSZ 34 /* stack needed for signal delivery */ -#define AT_VECTOR_SIZE_ARCH 1 /* entries in ARCH_DLINFO */ +#define AT_VECTOR_SIZE_ARCH 2 /* entries in ARCH_DLINFO */ #endif diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 2694143..221bcf2 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -560,8 +560,15 @@ asmlinkage long sys_rt_sigreturn(struct pt_regs *regs) return 0; } -/* Determine the layout of optional records in the signal frame */ -static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) +/* + * Determine the layout of optional records in the signal frame + * + * add_all: if true, lays out the biggest possible signal frame for + * this task; otherwise, generates a layout for the current state + * of the task. + */ +static int setup_sigframe_layout(struct rt_sigframe_user_layout *user, +bool add_all) { int err; @@ -571,7 +578,7 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user) return err; /* fault information, if valid */ - if (current->thread.fault_code) { + if (add_all || current->thread.fault_code) { err = sigframe_alloc(user,
[PATCH 25/27] arm64/sve: Add documentation
This patch adds basic documentation of the user/kernel interface provided by the for SVE. Signed-off-by: Dave Martin--- Documentation/arm64/sve.txt | 454 1 file changed, 454 insertions(+) create mode 100644 Documentation/arm64/sve.txt diff --git a/Documentation/arm64/sve.txt b/Documentation/arm64/sve.txt new file mode 100644 index 000..df25abf0 --- /dev/null +++ b/Documentation/arm64/sve.txt @@ -0,0 +1,454 @@ +Scalable Vector Extension support for AArch64 Linux +=== + +Author: Dave Martin +Date: 4 August 2017 + +This document outlines briefly the interface provided to userspace by Linux in +order to support use of the ARM Scalable Vector Extension (SVE). + +This is an outline of the most important features and issues only and not +intended to be exhaustive. + +This document does not aim to describe the SVE architecture or programmer's +model. To aid understanding, a minimal description of relevant programmer's +model features for SVE is included in Appendix A. + + +1. General +--- + +* SVE registers Z0..Z31, P0..P15 and FFR and the current vector length VL, are + tracked per-thread. + +* The presence of SVE is reported to userspace via HWCAP_SVE in the aux vector + AT_HWCAP entry. SVE is reported in /proc/cpuinfo as "sve". + + +2. Vector length terminology +- + +The size of an SVE vector (Z) register is referred to as the "vector length". + +To avoid confusion about the units used to express vector length, the kernel +adopts the following conventions: + +* Vector length (VL) = size of a Z-register in bytes + +* Vector quadwords (VQ) = size of a Z-register in units of 128 bits + +(So, VL = 16 * VQ.) + +The VQ convention is used where the underlying granularity is important, such +as in data structure definitions. In most other situations, the VL convention +is used. This is consistent with the meaning of the "VL" pseudo-register in +the SVE instruction set architecture. + + +3. System call behaviour +- + +* On syscall, V0..V31 are preserved (as without SVE). Thus, bits [127:0] of + Z0..Z31 are preserved. All other bits of Z0..Z31, and all of P0..P15 and FFR + become unspecified on return from a syscall. + +* The SVE registers are not used to pass arguments to or receive results from + any syscall. + +* In practice the affected registers/bits will be preserved or will be replaced + with zeros on return from a syscall, but userspace should not make + assumptions about this. The kernel behaviour may vary on a case-by-case + basis. + + +4. Signal handling +--- + +* A new signal frame record sve_context encodes the SVE registers on signal + delivery. [1] + +* This record is supplementary to fpsimd_context. The FPSR and FPCR registers + are only present in fpsimd_context. For convenience, the content of V0..V31 + is duplicated between sve_context and fpsimd_context. + +* The signal frame record for SVE always contains basic metadata, in particular + the thread's vector length (in sve_context.vl). + +* The SVE registers may or may not be included in the record, depending on + whether the registers are live for the thread. The registers are present if + and only if: + sve_context.head.size >= SVE_SIG_CONTEXT_SIZE(sve_vq_from_vl(sve_context.vl)). + +* If the registers are present, the remainder of the record has a vl-dependent + size and layout. Macros SIG_SVE_* are defined [1] to facilitate access to + the members. + +* If the SVE context is too big to fit in sigcontext.__reserved[], then extra + space is allocated on the stack, an extra_context record is written in + __reserved[] referencing this space. sve_context is then written in the + extra space. Refer to [1] for further details about this mechanism. + + +5. Signal return +- + +When returning from a signal handler: + +* If there is no sve_context record in the signal frame, or if the record is + present but contains no register data as desribed in the previous section, + then the SVE registers/bits become non-live and take unspecified values. + +* If sve_context is present in the signal frame and contains full register + data, the SVE registers become live and are populated with the specified + data. However, for backward compatibility reasons, bits [127:0] of Z0..Z31 + are always restored from the corresponding members of fpsimd_context.vregs[] + and not from sve_context. The remaining bits are restored from sve_context. + +* Inclusion of fpsimd_context in the signal frame remains mandatory, + irrespective of whether sve_context is present or not. + +* The vector length cannot be changed via signal return. If sve_context.vl in + the signal frame does not match the current vector length, the signal return + attempt is treated as illegal, resulting
[PATCH 20/27] arm64/sve: Add sysctl to set the default vector length for new processes
Because of the effect of SVE on the size of the signal frame, the default vector length used for new processes involves a tradeoff between performance of SVE-enabled software on the one hand, and reliability of non-SVE-aware software on the other hand. For this reason, the best choice depends on the repertoire of userspace software in use and is thus best left up to distro maintainers, sysadmins and developers. If CONFIG_SYSCTL is enabled, this patch exposes the default vector length in /proc/sys/abi/sve_default_vector_length, where boot scripts or the adventurous can poke it. In common with other arm64 ABI sysctls, this control is currently global: setting it requires CAP_SYS_ADMIN in the root user namespace, but the value set is effective for subsequent execs in all namespaces. The control only affects _new_ processes, however: changing it does not affect the vector length of any existing process. The intended usage model is that if userspace is known to be fully SVE-tolerant (or a developer is curious to find out) then init scripts can crank this up during startup. Signed-off-by: Dave Martin--- arch/arm64/kernel/fpsimd.c | 57 +- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 4fe1675..d45fcfb 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include @@ -200,6 +201,60 @@ static unsigned int find_supported_vector_length(unsigned int vl) return 16 * bit_to_vq(bit); } +#ifdef CONFIG_SYSCTL + +static int sve_proc_do_default_vl(struct ctl_table *table, int write, + void __user *buffer, size_t *lenp, + loff_t *ppos) +{ + int ret; + int vl = sve_default_vl; + struct ctl_table tmp_table = { + .data = , + .maxlen = sizeof(vl), + }; + + ret = proc_dointvec(_table, write, buffer, lenp, ppos); + if (ret || !write) + return ret; + + /* Writing -1 has the special meaning "set to max": */ + if (vl == -1) { + BUG_ON(find_supported_vector_length(sve_max_vl) != sve_max_vl); + sve_default_vl = sve_max_vl; + + return 0; + } + + if (!sve_vl_valid(vl)) + return -EINVAL; + + sve_default_vl = find_supported_vector_length(vl); + return 0; +} + +static struct ctl_table sve_default_vl_table[] = { + { + .procname = "sve_default_vector_length", + .mode = 0644, + .proc_handler = sve_proc_do_default_vl, + }, + { } +}; + +static int __init sve_sysctl_init(void) +{ + if (system_supports_sve()) + if (!register_sysctl("abi", sve_default_vl_table)) + return -EINVAL; + + return 0; +} + +#else /* ! CONFIG_SYSCTL */ +static int __init sve_sysctl_init(void) { return 0; } +#endif /* ! CONFIG_SYSCTL */ + #define ZREG(sve_state, vq, n) ((char *)(sve_state) + \ (SVE_SIG_ZREG_OFFSET(vq, n) - SVE_SIG_REGS_OFFSET)) @@ -1037,6 +1092,6 @@ static int __init fpsimd_init(void) if (!(elf_hwcap & HWCAP_ASIMD)) pr_notice("Advanced SIMD is not implemented\n"); - return 0; + return sve_sysctl_init(); } late_initcall(fpsimd_init); -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 21/27] arm64/sve: KVM: Prevent guests from using SVE
Until KVM has full SVE support, guests must not be allowed to execute SVE instructions. This patch enables the necessary traps, and also ensures that the traps are disabled again on exit from the guest so that the host can still use SVE if it wants to. Signed-off-by: Dave Martin--- arch/arm64/include/asm/kvm_arm.h | 3 ++- arch/arm64/kvm/hyp/switch.c | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index dbf0537..8a19651 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -186,7 +186,7 @@ #define CPTR_EL2_TTA (1 << 20) #define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) #define CPTR_EL2_TZ(1 << 8) -#define CPTR_EL2_DEFAULT 0x33ff +#define CPTR_EL2_DEFAULT (0x33ff & ~CPTR_EL2_TZ) /* Hyp Debug Configuration Register bits */ #define MDCR_EL2_TPMS (1 << 14) @@ -237,5 +237,6 @@ #define CPACR_EL1_FPEN (3 << 20) #define CPACR_EL1_TTA (1 << 28) +#define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN) #endif /* __ARM64_KVM_ARM_H__ */ diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 35a90b8..951f3eb 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -48,7 +48,7 @@ static void __hyp_text __activate_traps_vhe(void) val = read_sysreg(cpacr_el1); val |= CPACR_EL1_TTA; - val &= ~CPACR_EL1_FPEN; + val &= ~(CPACR_EL1_FPEN | CPACR_EL1_ZEN); write_sysreg(val, cpacr_el1); write_sysreg(__kvm_hyp_vector, vbar_el1); @@ -59,7 +59,7 @@ static void __hyp_text __activate_traps_nvhe(void) u64 val; val = CPTR_EL2_DEFAULT; - val |= CPTR_EL2_TTA | CPTR_EL2_TFP; + val |= CPTR_EL2_TTA | CPTR_EL2_TFP | CPTR_EL2_TZ; write_sysreg(val, cptr_el2); } @@ -117,7 +117,7 @@ static void __hyp_text __deactivate_traps_vhe(void) write_sysreg(mdcr_el2, mdcr_el2); write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2); - write_sysreg(CPACR_EL1_FPEN, cpacr_el1); + write_sysreg(CPACR_EL1_DEFAULT, cpacr_el1); write_sysreg(vectors, vbar_el1); } -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[PATCH 17/27] arm64/sve: Preserve SVE registers around EFI runtime service calls
The EFI runtime services ABI allows EFI to make free use of the FPSIMD registers during EFI runtime service calls, subject to the callee-save requirements of the AArch64 procedure call standard. However, the SVE architecture allows upper bits of the SVE vector registers to be zeroed as a side-effect of FPSIMD V-register writes. This means that the SVE vector registers must be saved in their entirety in order to avoid data loss: non-SVE-aware EFI implementations cannot restore them correctly. The non-IRQ case is already handled gracefully by kernel_neon_begin(). For the IRQ case, this patch allocates a suitable per-CPU stash buffer for the full SVE register state and uses it to preserve the affected registers around EFI calls. It is currently unclear how the EFI runtime services ABI will be clarified with respect to SVE, so it safest to assume that the predicate registers and FFR must be saved and restored too. No attempt is made to restore the restore the vector length after a call, for now. It is deemed rather insane for EFI to change it, and contemporary EFI implementations certainly won't. Signed-off-by: Dave Martin--- arch/arm64/kernel/fpsimd.c | 53 ++ 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index b7fb836..c727b47 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -120,12 +120,14 @@ int sve_max_vl = -1; /* Set of available vector lengths, as vq_to_bit(vq): */ static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); static bool sve_vq_map_finalised; +static void __percpu *efi_sve_state; #else /* ! CONFIG_ARM64_SVE */ /* Dummy declaration for code that will be optimised out: */ extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); extern bool sve_vq_map_finalised; +extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ @@ -416,6 +418,23 @@ int sve_verify_vq_map(void) return ret; } +static void __init sve_kernel_mode_neon_setup(void) +{ + if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) + return; + + /* +* alloc_percpu() warns and prints a backtrace if this goes wrong. +* This is evidence of a crippled system and we are returning void, +* so no attempt is made to handle this situation here. +*/ + BUG_ON(!sve_vl_valid(sve_max_vl)); + efi_sve_state = __alloc_percpu( + SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), 16); + if (!efi_sve_state) + panic("Cannot allocate percpu memory for EFI SVE save/restore"); +} + void __init sve_setup(void) { u64 zcr; @@ -455,6 +474,8 @@ void __init sve_setup(void) sve_max_vl); pr_info("SVE: default vector length %u bytes per vector\n", sve_default_vl); + + sve_kernel_mode_neon_setup(); } void fpsimd_release_thread(struct task_struct *dead_task) @@ -797,6 +818,7 @@ EXPORT_SYMBOL(kernel_neon_end); DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state); DEFINE_PER_CPU(bool, efi_fpsimd_state_used); +DEFINE_PER_CPU(bool, efi_sve_state_used); /* * EFI runtime services support functions @@ -825,7 +847,20 @@ void __efi_fpsimd_begin(void) if (may_use_simd()) kernel_neon_begin(); else { - fpsimd_save_state(this_cpu_ptr(_fpsimd_state)); + /* +* If !efi_sve_state, SVE can't be in use yet and doesn't need +* preserving: +*/ + if (system_supports_sve() && likely(efi_sve_state)) { + char *sve_state = this_cpu_ptr(efi_sve_state); + + __this_cpu_write(efi_sve_state_used, true); + + sve_save_state(sve_state + sve_ffr_offset(sve_max_vl), + _cpu_ptr(_fpsimd_state)->fpsr); + } else + fpsimd_save_state(this_cpu_ptr(_fpsimd_state)); + __this_cpu_write(efi_fpsimd_state_used, true); } } @@ -838,10 +873,20 @@ void __efi_fpsimd_end(void) if (!system_supports_fpsimd()) return; - if (__this_cpu_xchg(efi_fpsimd_state_used, false)) - fpsimd_load_state(this_cpu_ptr(_fpsimd_state)); - else + if (!__this_cpu_xchg(efi_fpsimd_state_used, false)) kernel_neon_end(); + else + if (system_supports_sve() && + likely(__this_cpu_read(efi_sve_state_used))) { + char const *sve_state = this_cpu_ptr(efi_sve_state); + + sve_load_state(sve_state + sve_ffr_offset(sve_max_vl), + _cpu_ptr(_fpsimd_state)->fpsr, + sve_vq_from_vl(sve_get_vl()) - 1); + + __this_cpu_write(efi_sve_state_used, false); + } else +
[PATCH 22/27] arm64/sve: KVM: Treat guest SVE use as undefined instruction execution
When trapping forbidden attempts by a guest to use SVE, we want the guest to see a trap consistent with SVE not being implemented. This patch injects an undefined instruction exception into the guest in response to such an exception. Signed-off-by: Dave Martin--- arch/arm64/kvm/handle_exit.c | 8 1 file changed, 8 insertions(+) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 17d8a16..e3e42d0 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -147,6 +147,13 @@ static int kvm_handle_unknown_ec(struct kvm_vcpu *vcpu, struct kvm_run *run) return 1; } +static int handle_sve(struct kvm_vcpu *vcpu, struct kvm_run *run) +{ + /* Until SVE is supported for guests: */ + kvm_inject_undefined(vcpu); + return 1; +} + static exit_handle_fn arm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec, [ESR_ELx_EC_WFx]= kvm_handle_wfx, @@ -160,6 +167,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_HVC64] = handle_hvc, [ESR_ELx_EC_SMC64] = handle_smc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, + [ESR_ELx_EC_SVE]= handle_sve, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug, -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
[RFC PATCH 27/27] arm64/sve: signal: Include SVE when computing AT_MINSIGSTKSZ
The SVE context block in the signal frame needs to be considered too when computing the maximum possible signal frame size. Because the size of this block depends on the vector length, this patch computes the size based not on the thread's current vector length but instead on the maximum possible vector length: this determines the maximum size of SVE context block that can be observed in any signal frame for the lifetime of the process. Signed-off-by: Dave Martin--- arch/arm64/kernel/signal.c | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 221bcf2..cb01353 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -586,11 +586,18 @@ static int setup_sigframe_layout(struct rt_sigframe_user_layout *user, } if (system_supports_sve()) { - unsigned int vq = 0; + unsigned int vq; - if (test_thread_flag(TIF_SVE)) { - BUG_ON(!sve_vl_valid(current->thread.sve_vl)); - vq = sve_vq_from_vl(current->thread.sve_vl); + if (!add_all && !test_thread_flag(TIF_SVE)) + vq = 0; + else { + unsigned int vl = sve_max_vl; + + if (!add_all) + vl = current->thread.sve_vl; + + BUG_ON(!sve_vl_valid(vl)); + vq = sve_vq_from_vl(vl); } err = sigframe_alloc(user, >sve_offset, -- 2.1.4 ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 12/15] arm-soc: omap: replace open coded VA->PA calculations
On 9 August 2017 at 20:05, Tony Lindgrenwrote: > * Ard Biesheuvel [170805 13:54]: >> This replaces a couple of open coded calculations to obtain the >> physical address of a far symbol with calls to the new adr_l etc >> macros. > > I gave this series a quick test and omap3 no longer boots it seems. > Thanks Tony. I will investigate ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH 12/15] arm-soc: omap: replace open coded VA->PA calculations
* Ard Biesheuvel[170809 12:24]: > On 9 August 2017 at 20:05, Tony Lindgren wrote: > > * Ard Biesheuvel [170805 13:54]: > >> This replaces a couple of open coded calculations to obtain the > >> physical address of a far symbol with calls to the new adr_l etc > >> macros. > > > > I gave this series a quick test and omap3 no longer boots it seems. > > > > Thanks Tony. I will investigate Thanks. Looks like omap4 still boots with all your patches, but omap3 won't boot even with patch 12 left out. Regards, Tony ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
kvm-unit-tests arm ptimer test
Hello, I have a general question on the ptimer test for KVM unit tests for the ARM/ARM64 platform, as the guest is only allowed to access to the physical counter and access to the physical timer is not allowed for the guest (CNTHCTL.EL1PCTEN is set and CNTHCTL.EL1PCEN is clear), so how is this ptimer test supposed to pass : For example, in this test, reading CNTP_{CTL/CVAL}_EL0 will always cause exceptions while running in guest in the following function : static void print_ptimer_info(void) { printf("CNTPCT_EL0 : 0x%016lx\n", read_sysreg(cntpct_el0)); printf("CNTP_CTL_EL0 : 0x%016lx\n", read_sysreg(cntp_ctl_el0)); printf("CNTP_CVAL_EL0: 0x%016lx\n", read_sysreg(cntp_cval_el0)); } Thanks, Ashish ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm