Hi Jack,
On Fri, Jun 12, 2026 at 05:23:50PM +0100, Jack Thomson wrote:
> From: Jack Thomson <[email protected]>
>
> Add arm64 support for KVM_PRE_FAULT_MEMORY by synthesizing a read data
> abort and routing it through the existing stage-2 fault handlers. Treat
> the requested GPA as an IPA in the userspace-owned VM's memslot space
> and always target the canonical stage-2, even if the vCPU last ran with
> a nested/shadow MMU selected.
>
> If the vCPU last ran in a nested context, switch to the canonical
> stage-2 with the vCPU put/load helpers so VMID, VNCR and shadow-MMU
> refcount state stay consistent. Leave the switch in place for the ioctl;
> vcpu_put() at ioctl exit drops the hw_mmu and the next vcpu_load()
> reselects the correct MMU from vCPU state.
>
> Check existing mappings with a shared page-table walk under the MMU read
> lock, and use the resulting walk level when constructing the synthetic
> fault. Report poisoned pages through the ioctl return path with
> -EHWPOISON instead of also queueing SIGBUS, and use the installed
> mapping size to advance the prefault range.
>
> Advertise KVM_CAP_PRE_FAULT_MEMORY on arm64. Protected VMs remain
> unsupported: pKVM filters the capability, and the ioctl returns
> -EOPNOTSUPP if invoked anyway.
>
> Signed-off-by: Jack Thomson <[email protected]>
> ---
> Documentation/virt/kvm/api.rst | 18 +++-
> arch/arm64/kvm/Kconfig | 1 +
> arch/arm64/kvm/arm.c | 1 +
> arch/arm64/kvm/mmu.c | 162 +++++++++++++++++++++++++++++++++
> 4 files changed, 178 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
> index 52bbbb553ce1..657e05656fa6 100644
> --- a/Documentation/virt/kvm/api.rst
> +++ b/Documentation/virt/kvm/api.rst
> @@ -6462,7 +6462,7 @@ See KVM_SET_USER_MEMORY_REGION2 for additional details.
> ---------------------------
>
> :Capability: KVM_CAP_PRE_FAULT_MEMORY
> -:Architectures: none
> +:Architectures: x86, arm64
> :Type: vcpu ioctl
> :Parameters: struct kvm_pre_fault_memory (in/out)
> :Returns: 0 if at least one page is processed, < 0 on error
> @@ -6470,11 +6470,14 @@ See KVM_SET_USER_MEMORY_REGION2 for additional
> details.
> Errors:
>
> ========== ===============================================================
> + EAGAIN A memslot update raced with the ioctl before any page was
> + processed.
> EINVAL The specified `gpa` and `size` were invalid (e.g. not
> page aligned, causes an overflow, or size is zero).
> ENOENT The specified `gpa` is outside defined memslots.
> EINTR An unmasked signal is pending and no page was processed.
> EFAULT The parameter address was invalid.
> + EHWPOISON A poisoned host page was encountered.
> EOPNOTSUPP Mapping memory for a GPA is unsupported by the
> hypervisor, and/or for the current vCPU state/mode.
> EIO unexpected error conditions (also causes a WARN)
> @@ -6494,7 +6497,14 @@ Errors:
> KVM_PRE_FAULT_MEMORY populates KVM's stage-2 page tables used to map memory
> for the current vCPU state. KVM maps memory as if the vCPU generated a
> stage-2 read page fault, e.g. faults in memory as needed, but doesn't break
> -CoW. However, KVM does not mark any newly created stage-2 PTE as Accessed.
> +CoW. However, on x86, KVM does not mark any newly created stage-2 PTE as
> +Accessed. On arm64, newly created stage-2 PTEs are marked Accessed.
> +
> +On arm64, `gpa` is interpreted as an IPA in the userspace-owned VM's
> +memslot address space. If the vCPU most recently ran a nested guest, KVM
> +still targets the VM's canonical stage-2, and does not interpret `gpa` as
> +a nested guest IPA or target the nested/shadow stage-2 selected by the
> +vCPU's last run state.
>
> In the case of confidential VM types where there is an initial set up of
> private guest memory before the guest is 'finalized'/measured, this ioctl
> @@ -6507,9 +6517,9 @@ case, the ioctl can be called in parallel.
>
> When the ioctl returns, the input values are updated to point to the
> remaining range. If `size` > 0 on return, the caller can just issue
> -the ioctl again with the same `struct kvm_map_memory` argument.
> +the ioctl again with the same `struct kvm_pre_fault_memory` argument.
>
> -Shadow page tables cannot support this ioctl because they
> +On x86, shadow page tables cannot support this ioctl because they
> are indexed by virtual address or nested guest physical address.
> Calling this ioctl when the guest is using shadow page tables (for
> example because it is running a nested guest with nested page tables)
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 449154f9a485..6b89262e8ba7 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -24,6 +24,7 @@ menuconfig KVM
> select HAVE_KVM_CPU_RELAX_INTERCEPT
> select KVM_MMIO
> select KVM_GENERIC_DIRTYLOG_READ_PROTECT
> + select KVM_GENERIC_PRE_FAULT_MEMORY
> select VIRT_XFER_TO_GUEST_WORK
> select KVM_VFIO
> select HAVE_KVM_DIRTY_RING_ACQ_REL
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 9453321ef8c6..dcb92bee13af 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -392,6 +392,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long
> ext)
> case KVM_CAP_COUNTER_OFFSET:
> case KVM_CAP_ARM_WRITABLE_IMP_ID_REGS:
> case KVM_CAP_ARM_SEA_TO_USER:
> + case KVM_CAP_PRE_FAULT_MEMORY:
> r = 1;
> break;
> case KVM_CAP_SET_GUEST_DEBUG2:
> diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
> index c720f07cb82e..4bf048bbcf8b 100644
> --- a/arch/arm64/kvm/mmu.c
> +++ b/arch/arm64/kvm/mmu.c
> @@ -1571,6 +1571,8 @@ struct kvm_s2_fault_desc {
> struct kvm_s2_trans *nested;
> struct kvm_memory_slot *memslot;
> unsigned long hva;
> + unsigned long *page_size;
As far as I know, fault handling was reworked to use struct
kvm_s2_fault_desc to store the fault information that user_mem_abort()
needs to handle the fault. Adding a 'page_size' field, that represents the
result of the gpa mapping process, might not be desirable.
> + bool prefault;
> };
>
> static int gmem_abort(const struct kvm_s2_fault_desc *s2fd)
> @@ -1882,6 +1884,13 @@ static int kvm_s2_fault_pin_pfn(const struct
> kvm_s2_fault_desc *s2fd,
> &s2vi->map_writable, &s2vi->page);
> if (unlikely(is_error_noslot_pfn(s2vi->pfn))) {
> if (s2vi->pfn == KVM_PFN_ERR_HWPOISON) {
> + /*
> + * When prefaulting, report the poison via -EHWPOISON
> + * only; don't also queue a SIGBUS as the run path
> + * does for the faulting vCPU thread.
> + */
> + if (s2fd->prefault)
> + return -EHWPOISON;
> kvm_send_hwpoison_signal(s2fd->hva,
> __ffs(s2vi->vma_pagesize));
> return 0;
> }
> @@ -2053,6 +2062,9 @@ static int kvm_s2_fault_map(const struct
> kvm_s2_fault_desc *s2fd,
> kvm_release_faultin_page(kvm, s2vi->page, !!ret, writable);
> kvm_fault_unlock(kvm);
>
> + if (s2fd->page_size && !ret)
> + *s2fd->page_size = mapping_size;
> +
> /*
> * Mark the page dirty only if the fault is handled successfully,
> * making sure we adjust the canonical IPA if the mapping size has
> @@ -2757,3 +2769,153 @@ void kvm_toggle_cache(struct kvm_vcpu *vcpu, bool
> was_enabled)
>
> trace_kvm_toggle_cache(*vcpu_pc(vcpu), was_enabled, now_enabled);
> }
> +
> +/*
> + * Prefaulting always targets the canonical stage-2. If the vCPU last ran
> + * in a nested context, swap in the canonical MMU via the vCPU put/load
> + * helpers so that preemption, VMID, VNCR fixmap and shadow-MMU refcount
> + * state stay consistent.
> + *
> + * The swap is deliberately not undone: nothing runs in between the
> + * per-page invocations of kvm_arch_vcpu_pre_fault_memory() except the
> + * generic prefault loop, and the vcpu_put() at ioctl exit discards
> + * vcpu->arch.hw_mmu anyway (see kvm_vcpu_put_hw_mmu()), so the next
> + * vcpu_load() re-derives the correct MMU from the vCPU's context. If the
> + * prefault task is preempted in the meantime, kvm_vcpu_put_hw_mmu()
> + * keeps the canonical MMU in place for the reload. Leaving the swap in
> + * place also bounds the cost to at most one put/load pair per ioctl,
> + * rather than two pairs per prefaulted page.
> + */
> +static void kvm_pre_fault_load_canonical_mmu(struct kvm_vcpu *vcpu)
> +{
> + if (!vcpu_has_nv(vcpu) || vcpu->arch.hw_mmu == &vcpu->kvm->arch.mmu)
> + return;
> +
> + preempt_disable();
> + kvm_arch_vcpu_put(vcpu);
> + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
> + kvm_arch_vcpu_load(vcpu, smp_processor_id());
> + preempt_enable();
> +}
> +
> +long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
> + struct kvm_pre_fault_memory *range)
> +{
> + struct kvm_vcpu_fault_info *fault_info = &vcpu->arch.fault;
> + struct kvm_vcpu_fault_info fault_backup = *fault_info;
I'm not sure you need to make a backup here. vcpu->arch.fault is populated
each time the CPU takes a fault.
> + s8 walk_level = KVM_PGTABLE_LAST_LEVEL;
> + unsigned long page_size = PAGE_SIZE;
> + struct kvm_memory_slot *memslot;
> + phys_addr_t gpa = range->gpa;
> + struct kvm_pgtable *pgt;
> + phys_addr_t end;
> + kvm_pte_t pte;
> + hva_t hva;
> + gfn_t gfn;
> + long ret;
> +
> + if (vcpu_is_protected(vcpu))
> + return -EOPNOTSUPP;
> +
> + /*
> + * Interpret range->gpa in the userspace-owned VM's IPA space, not in
> + * any nested guest IPA space that may have been active on the vCPU's
> + * last run. Always target the canonical stage-2.
> + */
> + kvm_pre_fault_load_canonical_mmu(vcpu);
> +
> + if (gpa >= kvm_phys_size(vcpu->arch.hw_mmu)) {
> + ret = -ENOENT;
> + goto out;
> + }
> +
> + gfn = gpa_to_gfn(gpa);
> + memslot = gfn_to_memslot(vcpu->kvm, gfn);
> + if (!memslot) {
> + ret = -ENOENT;
> + goto out;
> + }
> +
> + /*
> + * A racing memslot deletion or move installs an invalid slot before
> + * zapping stage-2. Ask userspace to retry once the update settles.
> + */
> + if (memslot->flags & KVM_MEMSLOT_INVALID) {
I don't think that's something we should care about, the flag can be set
immediately after the check as the function doesn't take kvm->slots_lock.
kvm_vcpu_prefault_memory() takes the srcu lock in read mode, so the
function is safe to run even if userspace does something silly like
deleting a memslot at the same time that it's prefaulting the guest memory
it represents.
> + ret = -EAGAIN;
> + goto out;
> + }
> +
> + /*
> + * pKVM stage-2 mappings aren't directly walkable from the host; let
> + * the fault path handle both new and existing mappings.
> + */
> + if (!is_protected_kvm_enabled()) {
> + pgt = vcpu->arch.hw_mmu->pgt;
> + scoped_guard(read_lock, &vcpu->kvm->mmu_lock) {
> + ret = kvm_pgtable_get_leaf(pgt, gpa, &pte, &walk_level,
> + KVM_PGTABLE_WALK_SHARED);
It might not be obvious, but taking kvm->mmu_lock in read mode does not
guarantee that gpa will be mapped when kvm_pgtable_get_leaf() returns.
That's because a concurrent kvm_pgtable_stage2_map() for a different gpa
can destroy the mapping of the current gpa.
> + }
The kvm->mmu_lock is dropped here, which means that it is possible for a
MMU notifier callback to have just unmapped the entire stage 2 for the VM.
> + if (ret)
> + goto out;
> +
> + if (kvm_pte_valid(pte)) {
> + page_size = kvm_granule_size(walk_level);
> + if (!(pte & KVM_PTE_LEAF_ATTR_LO_S2_AF))
> + handle_access_fault(vcpu, gpa);
handle_access_fault() will fail if the mapping is gone. But I guess that's
fine if the ioctl does not guarantee that memory is still mapped after it
completes.
Also, the documentation that this patch adds says: 'On arm64, newly created
stage-2 PTEs are marked Accessed'. Does not say anything about marking
**existing** ptes as accessed. Would be useful to explain the code does it.
> + goto out_success;
> + }
> + }
> +
> + /*
> + * Synthesize a read translation fault for the canonical IPA, at the
> + * level where the stage-2 walk currently ends (the last level under
> + * pKVM, where stage-2 isn't walkable from the host).
> + */
> + fault_info->esr_el2 = (ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT) |
> + ESR_ELx_IL | ESR_ELx_FSC_FAULT_L(walk_level);
> + fault_info->hpfar_el2 = HPFAR_EL2_NS |
> + FIELD_PREP(HPFAR_EL2_FIPA, gpa >> 12);
> +
> + struct kvm_s2_fault_desc s2fd = {
> + .vcpu = vcpu,
> + .fault_ipa = gpa,
> + .nested = NULL,
> + .memslot = memslot,
> + .page_size = &page_size,
> + .prefault = true,
> + };
> +
> + /*
> + * As in the run path, -EAGAIN from the abort handlers is treated as
> + * progress: either a parallel fault installed the mapping, or a racing
> + * invalidation is in flight and the next access will refault.
> + */
> + if (kvm_slot_has_gmem(memslot)) {
> + ret = gmem_abort(&s2fd);
> + } else {
> + hva = gfn_to_hva_memslot_prot(memslot, gfn, NULL);
There's gfn_to_hva_memslot(memslot, gfn), is that what you are looking for?
> + if (kvm_is_error_hva(hva)) {
> + ret = -EFAULT;
> + goto out;
> + }
> +
> + s2fd.hva = hva;
> + ret = user_mem_abort(&s2fd);
If is_protected_kvm_enabled(), this function doesn't check if the mapping
already exists. user_mem_abort() will call **p**kvm_pgtable_stage2_map()
(emphasis is mine) to create the mapping, and pkvm_pgtable_stage2_map() will
return -EAGAIN, and keep returning -EAGAIN until the mapping is destroyed, which
might be some time away.
Same thing happens **without** pKVM, kvm_pgtable_stage2_map() will return
-EAGAIN if the mapping exists. But that's not such a big issue, when the
user calls KVM_PRE_FAULT_MEMORY on the remaining range
kvm_pgtable_get_leaf() will detect the existing mapping. Which you have
already taken into consideration judging by the comment, so all good here.
Thanks,
Alex
> + }
> +
> + if (ret < 0)
> + goto out;
> +
> +out_success:
> + end = ALIGN_DOWN(gpa, page_size) + page_size;
> + ret = min_t(u64, range->size, end - gpa);
> +out:
> + /*
> + * Restore the synthetic fault state so a subsequent KVM_RUN does not
> + * observe it. kvm_handle_mmio_return() runs before guest entry can
> + * refresh fault.esr_el2 from hardware, so leaving the synthetic ESR
> + * in place would corrupt the completion of a pending MMIO exit.
> + */
> + *fault_info = fault_backup;
> + return ret;
> +}
> --
> 2.43.0
>
>
