From: David Woodhouse <[email protected]> Windows VSM boot stalls and PatchGuard-induced triple faults traced to missing host behaviours, all confirmed against the reference VSM host:
- A pending interrupt for a suspended higher VTL must preempt the lower VTL. In userspace-switch mode, report it as a new KVM_EXIT_HYPERV_VTL_INTERRUPT exit — from the halted/block path too, since a halted lower VTL never reaches vcpu_enter_guest() — so userspace enters the higher VTL itself and its view of the active VTL stays in sync. kvm_higher_plane_with_event() tests raw pending-ness (kvm_apic_has_interrupt), because a suspended plane carries a stale apicv_active that makes kvm_cpu_has_injectable_intr() report false. - The in-kernel forced entry, like the in-kernel VTL switch, writes the entered VTL's control area through the *entered* plane's context, since the assist page may be protected against the lower VTL; on failure a VTL call forwards to userspace and a forced entry stays pending. The refusal conditions live in kvm_hv_vtl_entry_blocked(), which also gates kvm_higher_plane_with_event() in in-kernel mode: an armed event on a plane that cannot be entered (never run, stopped by userspace, parked on a non-hypercall completion owning the shared run union) must not keep waking a halted lower VTL that has no way to consume it. A plane which has never entered the guest holds reset state and userspace owns the HvCallEnableVpVtl tracking (the reference host fails such calls with HV_STATUS_INVALID_VP_STATE), so hand it the switch and leave the event pending. The resume value is guest-derived in every switch flavour, so KVM_HV_HCALL_RESULT_RAISE_UD is masked out of it, and a target with no completion armed receives it directly in RAX, as the TLFS register-restore contract on VTL return requires. - RAX/RCX of the lower VTL are captured at in-kernel VTL entry and restored on an assist-less, non-fast return; vtl_entry_regs_valid tracks whether the capture is live. A return finding it clear (the entry was performed by userspace, or predates a migration — the capture has no GET/SET exposure) is forwarded to userspace, which owns restoring the lower VTL's registers. - plane_user_stopped is a pure userspace-policy latch: only KVM_SET_PLANE_RUN_STATE writes it, and no runnable plane can carry it (it is cleared at the single RUNNABLE choke point). In-kernel suspensions get their own plane_kernel_stopped flag, reported and restorable as KVM_PLANE_RUN_STATE_KERNEL_STOPPED, with the same plane-0 event gating. The in-kernel VTL switch and the forced entry refuse a user-stopped target (forwarding the switch to userspace or leaving the event pending), and the kvm_vcpu_select_plane() fallback refuses a user-stopped plane 0: KVM_RUN fails with -EINVAL, surfacing the VMM's ordering bug instead of resuming a plane the VMM parked mid-hypercall. - The secure kernel arms HvRegisterCrInterceptControl descriptor-table write intercepts for VTL0, then PatchGuard runs its protected patching window under an intercepted lidt. The write must *not* be applied — the window would execute with a genuinely null IDT and the first event triple-faults the guest. Add a per-vCPU KVM_CAP_HYPERV_REG_INTERCEPT that arms the control word, trap the descriptor-table instructions (SECONDARY_EXEC_DESC, dropped again when the intercepts disarm), and divert armed writes into a KVM_EXIT_HYPERV_REG_INTERCEPT exit with RIP rewound to the intercepted instruction: the higher VTL owns completion, and Windows' patch-protection probe matches its reflected #GP(0) against the probe RIP. - HvCallSendSyntheticClusterIpi(Ex) honours the input's in_vtl field (both the memory and fast forms carry it after the vector), delivering to the resolved plane's twins; reserved bits 7:5 are rejected with HV_STATUS_INVALID_HYPERCALL_INPUT before anything is delivered, as on the reference host. - hv_inkernel_vtl_switch is written under kvm->lock but read locklessly from vCPU context; the accesses carry WRITE_ONCE/READ_ONCE so each individual check reads a coherent value. A VMM that flips the cap while vCPUs run owns the consequences of catching a transition in flight. Also includes debug instrumentation, to be stripped before posting. Signed-off-by: David Woodhouse <[email protected]> Assisted-by: LLM --- Documentation/virt/kvm/api.rst | 31 ++-- arch/x86/hyperv/hv_apic.c | 3 +- arch/x86/include/asm/kvm_host.h | 21 ++- arch/x86/kvm/hyperv.c | 257 +++++++++++++++++++++++++++++--- arch/x86/kvm/hyperv.h | 3 + arch/x86/kvm/vmx/vmx.c | 23 ++- arch/x86/kvm/x86.c | 210 +++++++++++++++++++++++++- include/hyperv/hvgdk_mini.h | 3 +- include/hyperv/hvhdk_mini.h | 3 +- include/linux/kvm_host.h | 12 ++ include/uapi/linux/kvm.h | 13 ++ virt/kvm/kvm_main.c | 35 ++++- 12 files changed, 571 insertions(+), 43 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index c9283994014a..3ef57b3b050b 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -6628,19 +6628,32 @@ a vCPU fd of every plane, not just plane 0. :Returns: 0 on success, <0 on error Reads or sets the per-plane run state of a vCPU: -KVM_PLANE_RUN_STATE_STOPPED (0) or KVM_PLANE_RUN_STATE_RUNNABLE (1). +KVM_PLANE_RUN_STATE_STOPPED (0), KVM_PLANE_RUN_STATE_RUNNABLE (1) or +KVM_PLANE_RUN_STATE_KERNEL_STOPPED (2). When a vCPU thread calls KVM_RUN, KVM executes the lowest-numbered plane -whose vCPU is runnable, falling back to plane 0. A VMM implementing a -software privilege model (such as Hyper-V VSM) switches planes by -marking the target plane's vCPU runnable and the current one stopped -before re-entering KVM_RUN. Pending events for plane 0 make it runnable -again on their own only as long as userspace has not explicitly stopped -it with this ioctl; a VMM-stopped plane 0 keeps its events pending until -the VMM makes it runnable, so a software privilege model can keep a -higher plane running undisturbed. The ioctl is accepted on any plane's +whose vCPU is runnable, falling back to plane 0. If no plane is +runnable and userspace has stopped plane 0, KVM_RUN fails with -EINVAL +instead of overriding the stop. A VMM implementing a software privilege +model (such as Hyper-V VSM) switches planes by marking the target +plane's vCPU runnable and the current one stopped before re-entering +KVM_RUN. Pending events for plane 0 make it runnable again on their own +only as long as userspace has not explicitly stopped it with this ioctl; +a VMM-stopped plane 0 keeps its events pending until the VMM makes it +runnable, so a software privilege model can keep a higher plane running +undisturbed. In-kernel plane transitions never override a userspace +stop: a guest-initiated VTL switch targeting a stopped plane is +forwarded to userspace instead. The ioctl is accepted on any plane's vCPU fd. +KVM_PLANE_RUN_STATE_KERNEL_STOPPED is reported for a plane that KVM +itself suspended (such as the lower plane of an in-kernel VTL switch); +such a plane becomes runnable again on the matching in-kernel transition +without userspace involvement. Setting it stops the plane without +recording a userspace stop, so restoring previously read run states +round-trips a kernel suspension without converting it into userspace +stop policy. + The run state is distinct from KVM_GET/SET_MP_STATE, which describes the architectural state of a vCPU within its plane. diff --git a/arch/x86/hyperv/hv_apic.c b/arch/x86/hyperv/hv_apic.c index a8de503def37..533304c29603 100644 --- a/arch/x86/hyperv/hv_apic.c +++ b/arch/x86/hyperv/hv_apic.c @@ -127,7 +127,8 @@ static bool __send_ipi_mask_ex(const struct cpumask *mask, int vector, goto ipi_mask_ex_done; ipi_arg->vector = vector; - ipi_arg->reserved = 0; + ipi_arg->in_vtl.as_uint8 = 0; + memset(ipi_arg->reserved, 0, sizeof(ipi_arg->reserved)); ipi_arg->vp_set.valid_bank_mask = 0; /* diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 537ea6cded7a..74810e0e9ed7 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -732,13 +732,32 @@ struct kvm_vcpu_hv { u32 vp_id; } nested; + /* + * HvRegisterCrInterceptControl value the higher VTL armed for the + * lower VTL (userspace VSM): descriptor-table write bits divert + * the emulated write into a KVM_EXIT_HYPERV_REG_INTERCEPT exit. + */ + u64 reg_intercept_control; + struct { + u32 reg; + u8 insn_len; + bool pending; + u64 base; + u32 limit; + u64 rip; + } reg_intercept; + /* * RAX/RCX of the lower VTL captured at VTL entry, restored on a * non-fast VTL return when this plane has no VP assist page to - * carry them in its VTL control area. + * carry them in its VTL control area. Only valid between an + * in-kernel VTL entry and the matching return: a plane entered by + * userspace, or restored from migration, has no captured values + * and its assist-less return is forwarded to userspace. */ u64 vtl_entry_rax; u64 vtl_entry_rcx; + bool vtl_entry_regs_valid; }; struct kvm_hypervisor_cpuid { diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 2f57f89cd8ea..ca31054ed065 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -2364,7 +2364,8 @@ static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) } static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector, - u64 *sparse_banks, u64 valid_bank_mask) + u64 *sparse_banks, u64 valid_bank_mask, + int target_plane) { struct kvm_lapic_irq irq = { .delivery_mode = APIC_DM_FIXED, @@ -2374,13 +2375,25 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector, unsigned long i; kvm_for_each_vcpu(i, vcpu, kvm) { + struct kvm_vcpu *target = vcpu; + if (sparse_banks && !hv_is_vp_in_sparse_set(kvm_hv_get_vpindex(vcpu), valid_bank_mask, sparse_banks)) continue; + /* + * TLFS: the IPI targets the VPs' in_vtl (or the sender's + * VTL): deliver into that plane's APIC. + */ + if (target_plane) { + target = vcpu->common->vcpus[target_plane]; + if (!target) + continue; + } + /* We fail only when APIC is disabled */ - kvm_apic_set_irq(vcpu, &irq, NULL); + kvm_apic_set_irq(target, &irq, NULL); } } @@ -2393,6 +2406,8 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) struct hv_send_ipi send_ipi; u64 valid_bank_mask; u32 vector; + u8 in_vtl = 0; + int target_plane; bool all_cpus; int r; @@ -2410,13 +2425,25 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) return HV_STATUS_INVALID_HYPERCALL_INPUT; sparse_banks[0] = send_ipi.cpu_mask; vector = send_ipi.vector; + in_vtl = send_ipi.in_vtl.as_uint8; } else { /* 'reserved' part of hv_send_ipi should be 0 */ - if (unlikely(hc->ingpa >> 32 != 0)) + if (unlikely(hc->ingpa >> 40 != 0)) return HV_STATUS_INVALID_HYPERCALL_INPUT; sparse_banks[0] = hc->outgpa; vector = (u32)hc->ingpa; + in_vtl = (hc->ingpa >> 32) & 0xff; } + + /* + * Only hv_input_vtl.target_vtl (bits 3:0) and + * use_target_vtl (bit 4) are defined; reserved_z (7:5) + * must be zero in every form of the hypercall, before + * anything is delivered. + */ + if (unlikely(in_vtl & 0xe0)) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + all_cpus = false; valid_bank_mask = BIT_ULL(0); @@ -2426,12 +2453,18 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) if (unlikely(kvm_read_guest(kvm, hc->ingpa, &send_ipi_ex, sizeof(send_ipi_ex)))) return HV_STATUS_INVALID_HYPERCALL_INPUT; + in_vtl = send_ipi_ex.in_vtl.as_uint8; } else { send_ipi_ex.vector = (u32)hc->ingpa; send_ipi_ex.vp_set.format = hc->outgpa; send_ipi_ex.vp_set.valid_bank_mask = sse128_lo(hc->xmm[0]); + in_vtl = (hc->ingpa >> 32) & 0xff; } + /* As above: hv_input_vtl bits 7:5 are reserved. */ + if (unlikely(in_vtl & 0xe0)) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + trace_kvm_hv_send_ipi_ex(send_ipi_ex.vector, send_ipi_ex.vp_set.format, send_ipi_ex.vp_set.valid_bank_mask); @@ -2463,10 +2496,19 @@ static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) if ((vector < HV_IPI_LOW_VECTOR) || (vector > HV_IPI_HIGH_VECTOR)) return HV_STATUS_INVALID_HYPERCALL_INPUT; + /* TLFS: in_vtl bit 4 selects an explicit target VTL. */ + if (in_vtl & 0x10) + target_plane = in_vtl & 0xf; + else + target_plane = vcpu->plane_level; + if (target_plane >= KVM_MAX_PLANES) + return HV_STATUS_INVALID_HYPERCALL_INPUT; + if (all_cpus) - kvm_hv_send_ipi_to_many(kvm, vector, NULL, 0); + kvm_hv_send_ipi_to_many(kvm, vector, NULL, 0, target_plane); else - kvm_hv_send_ipi_to_many(kvm, vector, sparse_banks, valid_bank_mask); + kvm_hv_send_ipi_to_many(kvm, vector, sparse_banks, + valid_bank_mask, target_plane); ret_success: return HV_STATUS_SUCCESS; @@ -2772,6 +2814,116 @@ static u64 hv_vtl_assist_page_gpa(struct kvm_vcpu *vcpu) * Returns -ENOENT when the transition cannot be handled here (no twin * plane); the caller forwards the hypercall to userspace unchanged. */ +/* + * Enter a higher VTL because it has a pending interrupt (TLFS: higher + * VTL interrupts preempt lower VTL execution). The lower VTL is + * suspended mid-instruction, not in a hypercall, so no completion is + * pended for it: it resumes at the preempted RIP when the higher VTL + * returns, and the entered VTL's control area carries the INTERRUPT + * entry reason with the preempted RAX/RCX. + */ +#define HV_VTL_ENTRY_REASON_INTERRUPT 2 + +/* + * States in which the in-kernel paths must not enter a plane: + * + * - never entered the guest: it still holds reset state (userspace owns + * HvCallEnableVpVtl and runs a plane's first entry itself); + * - stopped by userspace with KVM_SET_PLANE_RUN_STATE: the VMM owns + * restarting it; + * - parked on a non-hypercall completion (MMIO, PIO): the completion + * owns the shared run union, which the resume protocol would clobber. + * + * The forced interrupt entry refuses such a target (the event stays + * pending), and kvm_higher_plane_with_event() must not report its + * pending event as a wakeup either, or a halted lower VTL spins through + * hlt/wake with nothing able to consume the event. + */ +bool kvm_hv_vtl_entry_blocked(struct kvm_vcpu *target) +{ + if (target->arch.last_vmentry_cpu == -1) + return true; + + if (target->plane_user_stopped) + return true; + + if (target->arch.complete_userspace_io && + target->arch.complete_userspace_io != kvm_hv_hypercall_complete_userspace) + return true; + + return false; +} + +bool kvm_hv_vtl_interrupt_entry(struct kvm_vcpu *vcpu, + struct kvm_vcpu *target) +{ + u64 rax = kvm_rax_read(vcpu), rcx = kvm_rcx_read(vcpu); + struct hv_vtl_control ctl; + u64 assist; + int i; + + static const int shared_gprs[] = { + VCPU_REGS_RAX, VCPU_REGS_RBX, VCPU_REGS_RCX, VCPU_REGS_RDX, + VCPU_REGS_RSI, VCPU_REGS_RDI, VCPU_REGS_RBP, + VCPU_REGS_R8, VCPU_REGS_R9, VCPU_REGS_R10, VCPU_REGS_R11, + VCPU_REGS_R12, VCPU_REGS_R13, VCPU_REGS_R14, VCPU_REGS_R15, + }; + + /* The event stays pending until the plane can be entered. */ + if (kvm_hv_vtl_entry_blocked(target)) + return false; + + /* + * Write the entered plane's control area through the entered + * plane's vCPU: the write must pass that plane's own memory + * attributes and memslot view, not the suspending plane's (which + * typically has no access to the higher VTL's assist page). On + * failure leave both planes untouched; the event stays pending in + * the target and the entry is retried. + */ + assist = hv_vtl_assist_page_gpa(target); + if (assist) { + ctl = (struct hv_vtl_control) { + .vtl_entry_reason = HV_VTL_ENTRY_REASON_INTERRUPT, + .vtl_ret_x64rax = rax, + .vtl_ret_x64rcx = rcx, + }; + if (kvm_vcpu_write_guest(target, assist + HV_VTL_CONTROL_OFFSET, + &ctl, sizeof(ctl))) + return false; + } + + for (i = 0; i < ARRAY_SIZE(shared_gprs); i++) + kvm_register_write_raw(target, shared_gprs[i], + kvm_register_read_raw(vcpu, + shared_gprs[i])); + target->arch.cr2 = vcpu->arch.cr2; + vcpu->common->vtl_fpu_from = vcpu->plane_level; + + if (to_hv_vcpu(target)) { + to_hv_vcpu(target)->vtl_entry_rax = rax; + to_hv_vcpu(target)->vtl_entry_rcx = rcx; + to_hv_vcpu(target)->vtl_entry_regs_valid = true; + } + + /* + * The target resumes its suspended hypercall, if it has one. The + * resume value is the shared RAX as copied from the entering plane + * just above: GPRs are VTL-shared, so the retired hypercall's RAX + * must match what every other shared register already shows. The + * value is guest-derived; strip the VMM-only RAISE_UD control bit + * so the completion cannot be made to inject #UD by the guest. + */ + if (target->arch.complete_userspace_io) + vcpu->run->hyperv.u.hcall.result = kvm_rax_read(target) & + ~KVM_HV_HCALL_RESULT_RAISE_UD; + + vcpu->plane_kernel_stopped = true; + kvm_vcpu_set_plane_stopped(vcpu); + kvm_vcpu_set_plane_runnable(target); + return true; +} + static int hv_vtl_switch_limit = -1; module_param(hv_vtl_switch_limit, int, 0644); @@ -2797,6 +2949,35 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return) return -ENOENT; target = vcpu->common->vcpus[to]; + /* + * A VMM-stopped plane stays stopped until the VMM restarts it. + * Forward the switch so userspace can apply its own policy. + */ + if (target->plane_user_stopped) + return -ENOENT; + + /* + * The resume protocol below delivers the result through the + * target's armed hypercall completion. A target parked on any + * other completion (MMIO, PIO) owns the shared run union; writing + * hcall.result over it would feed the pending completion garbage + * and lose the resume value. Forward such a switch to userspace. + */ + if (target->arch.complete_userspace_io && + target->arch.complete_userspace_io != kvm_hv_hypercall_complete_userspace) + return -ENOENT; + + /* + * Userspace owns HvCallEnableVpVtl state; the kernel only knows a + * plane vCPU exists, and VMMs create those at setup time. Refuse + * to call into a plane that has never entered the guest: it still + * holds reset state, and the reference host fails such a call with + * HV_STATUS_INVALID_VP_STATE. Forwarding lets userspace apply its + * enablement tracking (and run the plane's first entry itself). + */ + if (!is_return && target->arch.last_vmentry_cpu == -1) + return -ENOENT; + /* * Debug budget on in-kernel switches, counted per VM and only for * switches that pass the eligibility checks above (a forwarded @@ -2816,9 +2997,39 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return) &ctl, sizeof(ctl))) return -ENOENT; } else { + /* + * No captured entry registers to restore: this plane + * was last entered by userspace (or the capture was + * lost across migration). Forward the return so + * userspace restores the lower VTL's RAX/RCX itself. + */ + if (!to_hv_vcpu(vcpu)->vtl_entry_regs_valid) + return -ENOENT; + ctl.vtl_ret_x64rax = to_hv_vcpu(vcpu)->vtl_entry_rax; ctl.vtl_ret_x64rcx = to_hv_vcpu(vcpu)->vtl_entry_rcx; } + } else if (!is_return) { + /* + * Write the entered VTL's control area through the entered + * plane's vCPU: the write must pass that plane's memory + * attributes and memslot view, and the higher VTL normally + * protects its assist page against the caller's plane. + * Fail before any state moves (mirroring the read above); + * userspace then owns the switch. + */ + assist = hv_vtl_assist_page_gpa(target); + if (assist) { + ctl = (struct hv_vtl_control) { + .vtl_entry_reason = HV_VTL_ENTRY_REASON_VTL_CALL, + .vtl_ret_x64rax = rax, + .vtl_ret_x64rcx = rcx, + }; + if (kvm_vcpu_write_guest(target, + assist + HV_VTL_CONTROL_OFFSET, + &ctl, sizeof(ctl))) + return -ENOENT; + } } /* VTL-shared state: GPRs except RSP (RIP/RFLAGS stay private). */ @@ -2830,26 +3041,17 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return) vcpu->common->vtl_fpu_from = cur; if (!is_return) { - assist = hv_vtl_assist_page_gpa(target); - if (assist) { - ctl = (struct hv_vtl_control) { - .vtl_entry_reason = HV_VTL_ENTRY_REASON_VTL_CALL, - .vtl_ret_x64rax = rax, - .vtl_ret_x64rcx = rcx, - }; - kvm_vcpu_write_guest(vcpu, - assist + HV_VTL_CONTROL_OFFSET, - &ctl, sizeof(ctl)); - } if (to_hv_vcpu(target)) { to_hv_vcpu(target)->vtl_entry_rax = rax; to_hv_vcpu(target)->vtl_entry_rcx = rcx; + to_hv_vcpu(target)->vtl_entry_regs_valid = true; } resume = rax; } else { if (!fast) kvm_rcx_write(target, ctl.vtl_ret_x64rcx); resume = fast ? rax : ctl.vtl_ret_x64rax; + to_hv_vcpu(vcpu)->vtl_entry_regs_valid = false; } /* @@ -2859,9 +3061,27 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return) * exactly as when userspace drives the switch; until then its * suspended RIP stays at the vmcall, which the higher VTL can * observe (and rewrite) through HvCallGetVpRegisters. + * + * The resume value is guest-derived in all three cases (the + * caller's RAX on a call or a fast return, the control area on a + * non-fast return): strip the VMM-only RAISE_UD control bit so + * the completion cannot be made to inject #UD into the suspended + * plane. */ + resume &= ~KVM_HV_HCALL_RESULT_RAISE_UD; if (target->arch.complete_userspace_io) vcpu->run->hyperv.u.hcall.result = resume; + else + /* + * No completion is armed when the target's suspension + * already retired its vmcall (forwarded switch) or was a + * forced interrupt entry: RIP is already where the plane + * resumes, so deliver the resume RAX directly. Without + * this the GPR copy above leaves the switching plane's raw + * RAX in place and a non-fast return never restores the + * control-area value. + */ + kvm_rax_write(target, resume); /* * Suspend the current plane in its uncompleted hypercall; the @@ -2869,9 +3089,8 @@ static int kvm_hv_vtl_switch(struct kvm_vcpu *vcpu, bool is_return) */ vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace; - vcpu->plane_user_stopped = true; + vcpu->plane_kernel_stopped = true; kvm_vcpu_set_plane_stopped(vcpu); - target->plane_user_stopped = false; kvm_vcpu_set_plane_runnable(target); kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu); return 0; @@ -3050,7 +3269,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) ret = HV_STATUS_INVALID_HYPERCALL_INPUT; break; } - if (to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch && + if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch) && !hc.fast && !kvm_hv_vtl_switch(vcpu, hc.code == HVCALL_VTL_RETURN)) { /* diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 0593fe4333d3..0b81ba482201 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -102,6 +102,9 @@ static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu) } int kvm_hv_hypercall(struct kvm_vcpu *vcpu); +bool kvm_hv_vtl_interrupt_entry(struct kvm_vcpu *vcpu, + struct kvm_vcpu *target); +bool kvm_hv_vtl_entry_blocked(struct kvm_vcpu *target); void kvm_hv_irq_routing_update(struct kvm *kvm); int kvm_hv_synic_set_irq(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm, diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d10aa5f60cad..e1c5f1608b09 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5640,7 +5640,14 @@ static int handle_desc(struct kvm_vcpu *vcpu) */ BUILD_BUG_ON(KVM_POSSIBLE_CR4_GUEST_BITS & X86_CR4_UMIP); - WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)); + /* + * Descriptor-table exits are also armed when a higher VTL registers + * DT intercepts (reg_intercept_control); only an exit with neither + * UMIP emulation nor an armed intercept indicates a stale + * SECONDARY_EXEC_DESC control. + */ + WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP) && + !(to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control)); return kvm_emulate_instruction(vcpu, 0); } @@ -8044,6 +8051,20 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) /* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap(vcpu); + + /* + * Trap descriptor-table instructions when a higher VTL armed + * register intercepts on them, and stop trapping when the + * intercepts are disarmed. Preserve the control when UMIP + * emulation needs it (CR4.UMIP owns it then). + */ + if (vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_DESC) { + if (to_hv_vcpu(vcpu) && to_hv_vcpu(vcpu)->reg_intercept_control) + secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_DESC); + else if (!(vmx_umip_emulated() && + kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP))) + secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_DESC); + } } static __init u64 vmx_get_perf_capabilities(void) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index aa8e14a45568..374330f61eb2 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4885,6 +4885,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_SYS_HYPERV_CPUID: case KVM_CAP_HYPERV_FORWARD_HCALL: case KVM_CAP_HYPERV_VTL_SWITCH: + case KVM_CAP_HYPERV_REG_INTERCEPT: #endif case KVM_CAP_HCALL_FAULT_EXIT: case KVM_CAP_PCI_SEGMENT: @@ -6144,6 +6145,19 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, } return r; } + case KVM_CAP_HYPERV_REG_INTERCEPT: { + struct kvm_vcpu_hv *hv_vcpu; + int r; + + r = kvm_hv_vcpu_init(vcpu); + if (r) + return r; + hv_vcpu = to_hv_vcpu(vcpu); + hv_vcpu->reg_intercept_control = cap->args[0]; + /* Refresh the descriptor-table exiting controls. */ + kvm_x86_call(vcpu_after_set_cpuid)(vcpu); + return 0; + } case KVM_CAP_HYPERV_DIRECT_TLBFLUSH: if (!kvm_x86_ops.enable_l2_tlb_flush) return -ENOTTY; @@ -6931,7 +6945,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, r = -EINVAL; if (cap->args[0] & ~1ULL) break; - to_kvm_hv(kvm)->hv_inkernel_vtl_switch = !!cap->args[0]; + WRITE_ONCE(to_kvm_hv(kvm)->hv_inkernel_vtl_switch, !!cap->args[0]); r = 0; break; #endif @@ -8814,14 +8828,55 @@ static void emulator_get_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) kvm_x86_call(get_idt)(emul_to_vcpu(ctxt), dt); } +/* + * Divert a descriptor-table write into a register-intercept exit when the + * higher VTL armed one: the write is not applied, and userspace delivers + * the TLFS register-intercept message to the higher VTL. + */ +static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg, + const struct desc_ptr *dt, u64 ctl_bit) +{ +#ifdef CONFIG_KVM_HYPERV + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + if (!hv_vcpu || vcpu->plane_level || + !(hv_vcpu->reg_intercept_control & ctl_bit)) + return false; + + hv_vcpu->reg_intercept.reg = hv_reg; + hv_vcpu->reg_intercept.base = dt->address; + hv_vcpu->reg_intercept.limit = dt->size; + hv_vcpu->reg_intercept.rip = ctxt->eip; + hv_vcpu->reg_intercept.insn_len = ctxt->_eip - ctxt->eip; + hv_vcpu->reg_intercept.pending = true; + return true; +#else + return false; +#endif +} + +#define HV_CR_INTERCEPT_GDTR_WRITE BIT_ULL(15) +#define HV_CR_INTERCEPT_IDTR_WRITE BIT_ULL(16) +#define HV_X64_REGISTER_IDTR_ID 0x00070000 +#define HV_X64_REGISTER_GDTR_ID 0x00070001 + static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) { + if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_GDTR_ID, dt, + HV_CR_INTERCEPT_GDTR_WRITE)) + return; kvm_x86_call(set_gdt)(emul_to_vcpu(ctxt), dt); } static void emulator_set_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) { - kvm_x86_call(set_idt)(emul_to_vcpu(ctxt), dt); + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + + if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_IDTR_ID, dt, + HV_CR_INTERCEPT_IDTR_WRITE)) + return; + kvm_x86_call(set_idt)(vcpu, dt); } static unsigned long emulator_get_cached_segment_base( @@ -9607,6 +9662,44 @@ int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type, } EXPORT_SYMBOL_FOR_KVM_INTERNAL(x86_decode_emulated_instruction); +#ifdef CONFIG_KVM_HYPERV +/* + * Complete a descriptor-table write that hv_divert_dt_write() diverted + * during the emulation that just finished: rewind RIP and build the + * KVM_EXIT_HYPERV_REG_INTERCEPT exit. Consuming the record before + * x86_emulate_instruction() returns means it never outlives the emulation + * call: no stale rewind can fire on a later KVM_RUN, and disarming the cap + * or migrating the vCPU can never see a half-delivered intercept. + */ +static int complete_hv_reg_intercept(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + hv_vcpu->reg_intercept.pending = false; + + /* + * The intercepted instruction must not retire: the emulator advanced + * RIP when it diverted the write, but the higher VTL owns + * completion. It either applies the write and advances RIP itself, + * disarms the intercept so re-execution succeeds, or queues an + * exception that must fault AT the instruction (the + * kernel-patch-protection LIDT probe matches the faulting RIP + * against the probe site). + */ + kvm_rip_write(vcpu, hv_vcpu->reg_intercept.rip); + vcpu->run->exit_reason = KVM_EXIT_HYPERV; + vcpu->run->hyperv.type = KVM_EXIT_HYPERV_REG_INTERCEPT; + vcpu->run->hyperv.plane = vcpu->plane_level; + vcpu->run->hyperv.u.reg_intercept.reg = hv_vcpu->reg_intercept.reg; + vcpu->run->hyperv.u.reg_intercept.insn_len = + hv_vcpu->reg_intercept.insn_len; + vcpu->run->hyperv.u.reg_intercept.base = hv_vcpu->reg_intercept.base; + vcpu->run->hyperv.u.reg_intercept.limit = hv_vcpu->reg_intercept.limit; + vcpu->run->hyperv.u.reg_intercept.rip = hv_vcpu->reg_intercept.rip; + return 0; +} +#endif + int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int emulation_type, void *insn, int insn_len) { @@ -9826,6 +9919,15 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, } else vcpu->arch.emulate_regs_need_sync_to_vcpu = true; +#ifdef CONFIG_KVM_HYPERV + { + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + if (hv_vcpu && hv_vcpu->reg_intercept.pending) + r = complete_hv_reg_intercept(vcpu); + } +#endif + return r; } @@ -11083,6 +11185,77 @@ static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu, return r; } +/* + * A pending interrupt for a higher plane preempts this one: the higher + * VTL's interrupts take priority over lower-VTL execution (TLFS 15.2.4), + * and the secure kernel relies on it, e.g. to freeze the other processors' + * VTL1 while starting a new one. + */ +static struct kvm_vcpu *kvm_higher_plane_with_event(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu *higher; + int i; + + for (i = vcpu->plane_level + 1; i < KVM_MAX_PLANES; i++) { + higher = vcpu->common->vcpus[i]; + if (!higher) + continue; +#ifdef CONFIG_KVM_HYPERV + /* + * With in-kernel switching, an event on a plane the forced + * entry would refuse (never run, user-stopped, parked on a + * foreign completion) has no consumer until userspace + * starts the plane; reporting it would wake a halted lower + * VTL into an endless hlt/wake loop. With userspace + * switching the event is consumable via the interrupt exit, + * so it does count. + */ + if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch) && + kvm_hv_vtl_entry_blocked(higher)) + continue; +#endif + /* + * Test raw pending-ness, not injectability: + * kvm_cpu_has_injectable_intr() reports false on an + * APICv-active vCPU, and a suspended plane may carry a + * stale apicv_active until it next runs. A pending + * synthetic-timer request counts as well: it can only be + * processed (and its interrupt delivered) once the higher + * plane runs, so it would otherwise never fire while the + * plane is suspended. + */ + if (kvm_cpu_has_extint(higher) || + kvm_apic_has_interrupt(higher) != -1 || + kvm_test_request(KVM_REQ_HV_STIMER, higher)) + return higher; + } + return NULL; +} + +#ifdef CONFIG_KVM_HYPERV +/* + * With userspace owning VTL transitions, report a pending higher-plane + * event as an exit so userspace can enter the higher VTL. Returns true + * when the exit was prepared. + */ +static bool kvm_hv_vtl_interrupt_exit(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu *higher; + + if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch)) + return false; + + higher = kvm_higher_plane_with_event(vcpu); + if (!higher) + return false; + + vcpu->run->exit_reason = KVM_EXIT_HYPERV; + vcpu->run->hyperv.type = KVM_EXIT_HYPERV_VTL_INTERRUPT; + vcpu->run->hyperv.plane = higher->plane_level; + return true; +} +#endif + static inline bool kvm_check_plane0_events(struct kvm_vcpu *vcpu) { struct kvm_vcpu *vcpu_plane0; @@ -11548,13 +11721,30 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } if (kvm_check_plane0_events(vcpu) && - !vcpu->common->vcpus[0]->plane_user_stopped) { + !vcpu->common->vcpus[0]->plane_user_stopped && + !vcpu->common->vcpus[0]->plane_kernel_stopped) { kvm_vcpu_set_plane_runnable(vcpu->common->vcpus[0]); kvm_make_request(KVM_REQ_EVENT, vcpu); kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu); } +#ifdef CONFIG_KVM_HYPERV + if (kvm_hv_vtl_interrupt_exit(vcpu)) { + r = 0; + goto out; + } + + if (READ_ONCE(to_kvm_hv(vcpu->kvm)->hv_inkernel_vtl_switch)) { + struct kvm_vcpu *higher = kvm_higher_plane_with_event(vcpu); + + if (higher && kvm_hv_vtl_interrupt_entry(vcpu, higher)) { + kvm_make_request(KVM_REQ_EVENT, higher); + kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu); + } + } +#endif + if (kvm_check_request(KVM_REQ_PLANE_RESCHED, vcpu)) { vcpu->common->plane_switch = true; r = 0; @@ -11904,7 +12094,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_has_events); int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) { return kvm_vcpu_running(vcpu) || vcpu->arch.pv.pv_unhalted || - kvm_vcpu_has_events(vcpu); + kvm_vcpu_has_events(vcpu) || + kvm_higher_plane_with_event(vcpu); } /* Called within kvm->srcu read side. */ @@ -11999,6 +12190,17 @@ static int vcpu_run(struct kvm_vcpu *vcpu) if (r <= 0) break; +#ifdef CONFIG_KVM_HYPERV + /* + * A halted lower VTL never reaches vcpu_enter_guest, but a + * pending higher-VTL event must still preempt it. + */ + if (!kvm_vcpu_running(vcpu) && kvm_hv_vtl_interrupt_exit(vcpu)) { + r = 0; + break; + } +#endif + kvm_clear_request(KVM_REQ_UNBLOCK, vcpu); if (kvm_xen_has_pending_events(vcpu)) kvm_xen_inject_pending_events(vcpu); diff --git a/include/hyperv/hvgdk_mini.h b/include/hyperv/hvgdk_mini.h index e1b6ee863777..956e9fb7ec33 100644 --- a/include/hyperv/hvgdk_mini.h +++ b/include/hyperv/hvgdk_mini.h @@ -1346,7 +1346,8 @@ struct hv_input_set_vp_registers { /* HvCallSendSyntheticClusterIpi hypercall */ struct hv_send_ipi { /* HV_INPUT_SEND_SYNTHETIC_CLUSTER_IPI */ u32 vector; - u32 reserved; + union hv_input_vtl in_vtl; + u8 reserved[3]; u64 cpu_mask; } __packed; diff --git a/include/hyperv/hvhdk_mini.h b/include/hyperv/hvhdk_mini.h index b4cb2fa26e9b..a2180b35f646 100644 --- a/include/hyperv/hvhdk_mini.h +++ b/include/hyperv/hvhdk_mini.h @@ -464,7 +464,8 @@ struct hv_input_unmap_device_interrupt { struct hv_send_ipi_ex { /* HV_INPUT_SEND_SYNTHETIC_CLUSTER_IPI_EX */ u32 vector; - u32 reserved; + union hv_input_vtl in_vtl; + u8 reserved[3]; struct hv_vpset vp_set; } __packed; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 79b3ec2b95b8..a76615f2c3dc 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -427,8 +427,20 @@ struct kvm_vcpu { * Set when userspace stopped this plane with * KVM_SET_PLANE_RUN_STATE: pending plane-0 events then no longer * make plane 0 runnable on their own, the VMM owns the switch. + * Only the ioctl sets it; in-kernel plane transitions never + * override it, and kvm_vcpu_set_plane_runnable() clears it so a + * runnable plane cannot carry a stale latch. */ bool plane_user_stopped; + /* + * Set when an in-kernel VTL switch suspends this plane: plane-0 + * events are gated exactly as for a userspace stop, but + * KVM_GET_PLANE_RUN_STATE reports the state as + * KVM_PLANE_RUN_STATE_KERNEL_STOPPED so that a save/restore + * round trip does not convert a transient kernel suspension + * into userspace stop policy. + */ + bool plane_kernel_stopped; /* S390 only */ bool valid_wakeup; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index b50f92135acd..9e453f215e68 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -99,6 +99,8 @@ struct kvm_hyperv_exit { #define KVM_EXIT_HYPERV_SYNIC 1 #define KVM_EXIT_HYPERV_HCALL 2 #define KVM_EXIT_HYPERV_SYNDBG 3 +#define KVM_EXIT_HYPERV_VTL_INTERRUPT 4 +#define KVM_EXIT_HYPERV_REG_INTERCEPT 5 __u32 type; __u32 plane; union { @@ -114,6 +116,15 @@ struct kvm_hyperv_exit { __u64 result; __u64 params[2]; } hcall; + struct { + __u32 reg; + __u8 insn_len; + __u8 pad[3]; + __u64 base; + __u32 limit; + __u32 pad2; + __u64 rip; + } reg_intercept; struct { __u32 msr; __u32 pad2; @@ -1026,6 +1037,7 @@ struct kvm_enable_cap { #define KVM_HV_HCALL_RESULT_RAISE_UD (1ULL << 63) #define KVM_CAP_HCALL_FAULT_EXIT 252 #define KVM_CAP_HYPERV_VTL_SWITCH 253 +#define KVM_CAP_HYPERV_REG_INTERCEPT 254 struct kvm_irq_routing_irqchip { __u32 irqchip; @@ -1384,6 +1396,7 @@ struct kvm_s390_keyop { #define KVM_SET_PLANE_RUN_STATE _IOW(KVMIO, 0xe6, __u32) #define KVM_PLANE_RUN_STATE_STOPPED 0 #define KVM_PLANE_RUN_STATE_RUNNABLE 1 +#define KVM_PLANE_RUN_STATE_KERNEL_STOPPED 2 /* * ioctls for vcpu fds diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ef6e59a15846..4197496f7176 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5023,9 +5023,14 @@ static long kvm_vcpu_ioctl(struct file *filp, break; } case KVM_GET_PLANE_RUN_STATE: { - u32 state = vcpu->plane_state == RUNNABLE ? - KVM_PLANE_RUN_STATE_RUNNABLE : - KVM_PLANE_RUN_STATE_STOPPED; + u32 state; + + if (vcpu->plane_state == RUNNABLE) + state = KVM_PLANE_RUN_STATE_RUNNABLE; + else if (vcpu->plane_user_stopped) + state = KVM_PLANE_RUN_STATE_STOPPED; + else + state = KVM_PLANE_RUN_STATE_KERNEL_STOPPED; r = -EINVAL; if (kvm_arch_max_planes(vcpu->kvm) <= 1) @@ -5052,13 +5057,15 @@ static long kvm_vcpu_ioctl(struct file *filp, if (copy_from_user(&state, argp, sizeof(state))) goto out; r = -EINVAL; - if (state > KVM_PLANE_RUN_STATE_RUNNABLE) + if (state > KVM_PLANE_RUN_STATE_KERNEL_STOPPED) goto out; if (state == KVM_PLANE_RUN_STATE_RUNNABLE) { - vcpu->plane_user_stopped = false; kvm_vcpu_set_plane_runnable(vcpu); } else { - vcpu->plane_user_stopped = true; + vcpu->plane_user_stopped = + state == KVM_PLANE_RUN_STATE_STOPPED; + vcpu->plane_kernel_stopped = + state == KVM_PLANE_RUN_STATE_KERNEL_STOPPED; kvm_vcpu_set_plane_stopped(vcpu); } r = 0; @@ -5319,6 +5326,13 @@ static struct file_operations kvm_plane_fops = { void kvm_vcpu_set_plane_runnable(struct kvm_vcpu *vcpu) { + /* + * A runnable plane must not carry a stale stop latch: the latches + * gate the plane-0 event wakeup, and KVM_GET_PLANE_RUN_STATE could + * not report the inconsistency. + */ + vcpu->plane_user_stopped = false; + vcpu->plane_kernel_stopped = false; vcpu->plane_state = RUNNABLE; vcpu->common->plane_switch = true; kvm_make_request(KVM_REQ_PLANE_RESCHED, vcpu); @@ -5349,7 +5363,16 @@ struct kvm_vcpu *kvm_vcpu_select_plane(struct kvm_vcpu *vcpu) } if (ret == NULL) { + /* + * Never force a userspace-stopped plane 0 to run: it may be + * suspended mid-hypercall, and the VMM owns restarting it. + * KVM_RUN fails instead, surfacing the VMM's ordering bug. + */ ret = common->vcpus[0]; + if (ret->plane_user_stopped) + return NULL; + + ret->plane_kernel_stopped = false; ret->plane_state = RUNNABLE; } -- 2.50.1
