From: David Woodhouse <[email protected]> The secure kernel arms descriptor-table write intercepts for VTL0 with HvRegisterCrInterceptControl, and PatchGuard then runs its protected patching window under an intercepted LIDT. The write must not be applied: the window would run with a genuinely null IDT and the first event triple-faults the guest. KVM has no notion of the intercept, so it loads the IDT.
Add a per-vCPU KVM_CAP_HYPERV_REG_INTERCEPT through which the VMM sets the control word. While a descriptor-table write bit is armed, the vendor code traps descriptor-table loads, and the emulator diverts an armed LGDT or LIDT executed in plane 0 into a KVM_EXIT_HYPERV_REG_INTERCEPT exit instead of applying it. The exit carries the HV_X64_REGISTER_* id, the base and limit the guest tried to load, and the RIP and length of the instruction. RIP is rewound to the instruction: the higher VTL owns completion, and Windows' patch-protection probe matches its reflected #GP(0) against the probe RIP. The diverted write is consumed before x86_emulate_instruction() returns, so it never outlives the emulation. api.rst documents the exit and its reg_intercept payload. VMX traps with SECONDARY_EXEC_DESC; SVM arms the four VMCB descriptor-table load intercepts as a block to match the all-or-nothing VMX control, and routes the exits into kvm_emulate_instruction(). A load the divert check declines is applied by the emulator as before. Store intercepts stay untrapped, as the control has no bits for them. The (dis)arming gets its own kvm_x86_ops.update_hv_reg_intercepts(), touching only the L1 controls, rather than riding on hooks with lifecycles of their own: - vmx_set_cr4() owns SECONDARY_EXEC_DESC for UMIP emulation, so a VTL0 set-then-clear of CR4.UMIP would disarm the very lidt/lgdt trap this exists for. The CR4 path preserves the control while the intercepts are armed. - On VMX the op defers while vmcs02 is loaded, and nested_vmx_vmexit() refreshes vmcs01 after switching back, rather than writing vmcs02 state that the next nested VM-exit would throw away. - On SVM it arms vmcb01, and nested_vmcb02_recalc_intercepts() strips the LOAD_* bits from vmcb02 unless vmcb12 wants them, matching the VMX DESC strip. hv_divert_dt_write() bails in guest mode, so an L2 descriptor exit that still reaches the emulator (UMIP emulation, for example) applies the load normally. - init_vmcb() re-applies the SVM intercepts, since reg_intercept_control survives a vCPU reset: a guest SHUTDOWN clears the VMCB and resets the vCPU, and must not leave them disarmed behind the higher VTL's back. Arming is refused with -EINVAL where the diverted load cannot be emulated, before anything is armed; disarming always succeeds: - TDX does not allow descriptor-table loads to be intercepted. - On SEV-ES, desc_interception() hits svm_check_emulate_instruction() returning X86EMUL_RETRY_INSTR, which resumes at the same RIP with the intercept still armed, so one descriptor-table load would spin a vCPU (and a host CPU) forever. - On plain SEV, the emulator has no instruction bytes and would inject #UD on an architecturally valid lgdt. For the same reason KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM refuses to turn a VM into an SEV guest while any of its vCPUs has the intercepts armed; the VMM can disarm first. A failure to re-apply the intercepts after that is a broken invariant, so svm_vcpu_after_set_cpuid() and init_vmcb() warn and disarm rather than leave reg_intercept_control claiming intercepts the VMCB does not have. The cap stays refused on plane fds, like the other unaudited vCPU caps: the divert only ever applies to plane 0. Signed-off-by: David Woodhouse <[email protected]> --- Documentation/virt/kvm/api.rst | 19 +++++ arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 7 ++ arch/x86/kvm/hyperv.h | 40 ++++++++++ arch/x86/kvm/svm/nested.c | 16 ++++ arch/x86/kvm/svm/sev.c | 24 ++++++ arch/x86/kvm/svm/svm.c | 90 ++++++++++++++++++++++ arch/x86/kvm/vmx/main.c | 13 ++++ arch/x86/kvm/vmx/nested.c | 1 + arch/x86/kvm/vmx/vmx.c | 65 +++++++++++++++- arch/x86/kvm/vmx/x86_ops.h | 1 + arch/x86/kvm/x86.c | 120 ++++++++++++++++++++++++++++- include/uapi/linux/kvm.h | 11 +++ 13 files changed, 401 insertions(+), 7 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index b76c46cc5000..83daee450284 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -7448,6 +7448,7 @@ EOI was received. #define KVM_EXIT_HYPERV_HCALL 2 #define KVM_EXIT_HYPERV_SYNDBG 3 #define KVM_EXIT_HYPERV_VTL_INTERRUPT 4 + #define KVM_EXIT_HYPERV_REG_INTERCEPT 5 __u32 type; __u32 plane; union { @@ -7465,6 +7466,15 @@ EOI was received. #define KVM_HYPERV_HCALL_UD (1ULL << 0) __u64 flags; } hcall; + struct { + __u32 reg; + __u8 insn_len; + __u8 pad[3]; + __u64 base; + __u32 limit; + __u32 pad2; + __u64 rip; + } reg_intercept; struct { __u32 msr; __u32 pad2; @@ -7534,6 +7544,15 @@ single-shot: if the VMM declines the entry, the event waits in the target plane for a cooperative entry rather than re-triggering the exit. + - KVM_EXIT_HYPERV_REG_INTERCEPT -- a descriptor-table write that a + +higher plane intercepted (KVM_CAP_HYPERV_REG_INTERCEPT) has been +diverted to userspace instead of being applied. 'reg_intercept' +carries the HV_X64_REGISTER_* id of the written register, the +descriptor-table base and limit the guest tried to load, and the RIP +and length of the writing instruction. The vCPU's RIP has been +rewound to that instruction; the higher plane owns its completion. + :: /* KVM_EXIT_ARM_NISV / KVM_EXIT_ARM_LDST64B */ diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index c50d7ba07ff4..1b56c290c412 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -143,6 +143,7 @@ KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked) KVM_X86_OP_OPTIONAL(enable_l2_tlb_flush) +KVM_X86_OP(update_hv_reg_intercepts) KVM_X86_OP_OPTIONAL(migrate_timers) KVM_X86_OP(recalc_intercepts) KVM_X86_OP(complete_emulated_msr) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 74899a27e29e..fe8fddab6039 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1803,6 +1803,13 @@ struct kvm_x86_ops { bool (*apic_init_signal_blocked)(struct kvm_vcpu *vcpu); int (*enable_l2_tlb_flush)(struct kvm_vcpu *vcpu); + /* + * Bring the L1 (vmcs01/vmcb01) descriptor-table intercepts in line + * with hv_vcpu->reg_intercept_control. Never touches L2 controls; + * safe to call at any point, including with a vCPU in guest mode. + */ + int (*update_hv_reg_intercepts)(struct kvm_vcpu *vcpu); + void (*migrate_timers)(struct kvm_vcpu *vcpu); void (*recalc_intercepts)(struct kvm_vcpu *vcpu); int (*complete_emulated_msr)(struct kvm_vcpu *vcpu, int err); diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index fbfbe0d52f9c..069c59788bf6 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -116,6 +116,21 @@ struct kvm_vcpu_hv { u32 vp_id; } nested; + /* + * HvRegisterCrInterceptControl value the higher VTL armed for the + * lower VTL (userspace VSM): descriptor-table write bits divert + * the emulated write into a KVM_EXIT_HYPERV_REG_INTERCEPT exit. + */ + u64 reg_intercept_control; + struct { + u32 reg; + u8 insn_len; + bool pending; + u64 base; + u32 limit; + u64 rip; + } reg_intercept; + /* * RAX/RCX of the lower VTL captured at VTL entry, restored on a * non-fast VTL return when this plane has no VP assist page to @@ -197,6 +212,26 @@ static inline struct kvm_vcpu_hv *to_hv_vcpu(struct kvm_vcpu *vcpu) return vcpu->arch.hyperv; } +/* + * True when a higher VTL armed descriptor-table write intercepts + * (HvRegisterCrInterceptControl) on this vCPU, i.e. when the vendor code + * must trap the L1 guest's descriptor-table loads for hv_divert_dt_write(). + */ +static inline bool kvm_hv_dt_intercepts_armed(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + return hv_vcpu && hv_vcpu->reg_intercept_control; +} + +static inline void kvm_hv_disarm_dt_intercepts(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + if (hv_vcpu) + hv_vcpu->reg_intercept_control = 0; +} + static inline struct kvm_vcpu_hv_synic *to_hv_synic(struct kvm_vcpu *vcpu) { struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); @@ -434,6 +469,11 @@ static inline bool kvm_hv_hcall_feature(struct kvm *kvm, u64 feature) { return false; } +static inline bool kvm_hv_dt_intercepts_armed(struct kvm_vcpu *vcpu) +{ + return false; +} +static inline void kvm_hv_disarm_dt_intercepts(struct kvm_vcpu *vcpu) {} static inline void kvm_hv_setup_tsc_page(struct kvm *kvm, struct pvclock_vcpu_time_info *hv_clock) {} static inline void kvm_hv_request_tsc_page_update(struct kvm *kvm) {} diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 75b02a84f2df..1836828436a9 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -172,6 +172,22 @@ void nested_vmcb02_recalc_intercepts(struct vcpu_svm *svm) for (i = 0; i < MAX_INTERCEPT; i++) vmcb02->control.intercepts[i] |= vmcb12_ctrl->intercepts[i]; + /* + * KVM arms the descriptor-table load intercepts in vmcb01 to divert + * L1's loads into Hyper-V register-intercept exits (see + * hv_divert_dt_write()); they do not apply to L2. Drop them from + * vmcb02 unless L1 wants the intercept itself, matching the + * SECONDARY_EXEC_DESC strip in VMX's prepare_vmcs02(). + */ + if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_IDTR)) + vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_IDTR); + if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_GDTR)) + vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_GDTR); + if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_LDTR)) + vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_LDTR); + if (!vmcb12_is_intercept(vmcb12_ctrl, INTERCEPT_LOAD_TR)) + vmcb_clr_intercept(&vmcb02->control, INTERCEPT_LOAD_TR); + /* If SMI is not intercepted, ignore guest SMI intercept as well */ if (!intercept_smi) vmcb_clr_intercept(&vmcb02->control, INTERCEPT_SMI); diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index a63822defc3b..79e78c7f56b3 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -35,6 +35,7 @@ #include "svm.h" #include "svm_ops.h" #include "cpuid.h" +#include "hyperv.h" #include "trace.h" #include "lapic.h" @@ -2161,6 +2162,25 @@ static int sev_check_source_vcpus(struct kvm *dst, struct kvm *src) return 0; } +/* + * The destination becomes an SEV guest, which cannot have the Hyper-V + * descriptor-table register intercepts (svm_update_hv_reg_intercepts() + * refuses to arm them). Refuse the move while any destination vCPU has + * them armed; the VMM can disarm first. + */ +static int sev_check_dest_reg_intercepts(struct kvm *dst) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + kvm_for_each_vcpu(i, vcpu, dst) { + if (kvm_hv_dt_intercepts_armed(vcpu)) + return -EINVAL; + } + + return 0; +} + int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) { struct kvm_sev_info *dst_sev = to_kvm_sev_info(kvm); @@ -2210,6 +2230,10 @@ int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd) if (ret) goto out_source_vcpu; + ret = sev_check_dest_reg_intercepts(kvm); + if (ret) + goto out_source_vcpu; + /* * Allocate a new have_run_cpus for the destination, i.e. don't copy * the set of CPUs from the source. If a CPU was used to run a vCPU in diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 5f07ddfafca4..b87c12944108 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1142,6 +1142,65 @@ static void svm_recalc_intercepts(struct kvm_vcpu *vcpu) svm_recalc_msr_intercepts(vcpu); } +static int svm_update_hv_reg_intercepts(struct kvm_vcpu *vcpu) +{ + struct vcpu_svm *svm = to_svm(vcpu); + + /* + * Trap descriptor-table loads while a higher VTL has register + * intercepts armed on them (reg_intercept_control), and stop + * trapping when they are disarmed; the emulator diverts a trapped + * load into a register-intercept exit (hv_divert_dt_write()). + * VMX arms the all-or-nothing SECONDARY_EXEC_DESC; the SVM + * intercepts are per-register but armed as a block to match, and + * an exit the emulator does not divert applies the load normally. + * + * The intercepts are armed in vmcb01 only: an L2 descriptor load + * is never diverted, and nested_vmcb02_recalc_intercepts() keeps + * these bits out of vmcb02 unless L1 asked for them itself. + */ + if (kvm_hv_dt_intercepts_armed(vcpu)) { + /* + * The diverted load needs full emulation, which encrypted + * guests cannot have: on SEV-ES and SNP the emulator cannot + * read the registers, so svm_check_emulate_instruction() + * retries at the same RIP with the intercept still armed and + * every descriptor-table load livelocks the vCPU; on plain + * SEV there are no instruction bytes to decode and a valid + * lgdt takes a spurious #UD. Refuse to arm. + */ + if (is_sev_guest(vcpu)) + return -EINVAL; + + svm_set_intercept(svm, INTERCEPT_LOAD_IDTR); + svm_set_intercept(svm, INTERCEPT_LOAD_GDTR); + svm_set_intercept(svm, INTERCEPT_LOAD_LDTR); + svm_set_intercept(svm, INTERCEPT_LOAD_TR); + } else { + svm_clr_intercept(svm, INTERCEPT_LOAD_IDTR); + svm_clr_intercept(svm, INTERCEPT_LOAD_GDTR); + svm_clr_intercept(svm, INTERCEPT_LOAD_LDTR); + svm_clr_intercept(svm, INTERCEPT_LOAD_TR); + } + return 0; +} + +/* + * Re-apply the descriptor-table intercepts from reg_intercept_control. + * Arming is refused up front for guests that cannot have it (the + * enable-cap and sev_vm_move_enc_context_from() both check), so a failure + * here is a broken invariant: disarm, rather than leave the controls and + * reg_intercept_control disagreeing. + */ +static void svm_refresh_hv_reg_intercepts(struct kvm_vcpu *vcpu) +{ + if (!WARN_ON_ONCE(svm_update_hv_reg_intercepts(vcpu))) + return; + + kvm_hv_disarm_dt_intercepts(vcpu); + svm_update_hv_reg_intercepts(vcpu); +} + static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) { struct vcpu_svm *svm = to_svm(vcpu); @@ -1308,6 +1367,14 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) svm_hv_init_vmcb(vmcb); + /* + * The descriptor-table intercepts follow reg_intercept_control, + * which a reset does not touch: re-apply them, or a SHUTDOWN (which + * clears the VMCB and resets the vCPU) leaves them disarmed while + * the higher VTL still believes them armed. + */ + svm_refresh_hv_reg_intercepts(vcpu); + kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); vmcb_mark_all_dirty(vmcb); @@ -2087,6 +2154,17 @@ static int npf_interception(struct kvm_vcpu *vcpu) return rc; } +static int desc_interception(struct kvm_vcpu *vcpu) +{ + /* + * Descriptor-table write exits are armed only while a higher VTL + * has register intercepts on them (reg_intercept_control); the + * emulator diverts the write into a register-intercept exit via + * hv_divert_dt_write() instead of applying it. + */ + return kvm_emulate_instruction(vcpu, 0); +} + static int db_interception(struct kvm_vcpu *vcpu) { struct kvm_run *kvm_run = vcpu->run; @@ -3461,6 +3539,10 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = { [SVM_EXIT_WRITE_CR3] = cr_interception, [SVM_EXIT_WRITE_CR4] = cr_interception, [SVM_EXIT_WRITE_CR8] = cr8_write_interception, + [SVM_EXIT_IDTR_WRITE] = desc_interception, + [SVM_EXIT_GDTR_WRITE] = desc_interception, + [SVM_EXIT_LDTR_WRITE] = desc_interception, + [SVM_EXIT_TR_WRITE] = desc_interception, [SVM_EXIT_READ_DR0] = dr_interception, [SVM_EXIT_READ_DR1] = dr_interception, [SVM_EXIT_READ_DR2] = dr_interception, @@ -4931,6 +5013,13 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) if (guest_cpuid_is_intel_compatible(vcpu)) guest_cpu_cap_clear(vcpu->common, X86_FEATURE_V_VMSAVE_VMLOAD); + /* + * Re-arm the descriptor-table intercepts; setting guest CPUID does + * not change reg_intercept_control, but keep the controls in sync + * the same way vmx_vcpu_after_set_cpuid() does. + */ + svm_refresh_hv_reg_intercepts(vcpu); + if (is_sev_guest(vcpu)) sev_vcpu_after_set_cpuid(svm); } @@ -5681,6 +5770,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .apic_init_signal_blocked = svm_apic_init_signal_blocked, .recalc_intercepts = svm_recalc_intercepts, + .update_hv_reg_intercepts = svm_update_hv_reg_intercepts, .complete_emulated_msr = svm_complete_emulated_msr, .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 14d21d02bff4..6efbb7062201 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -211,6 +211,18 @@ static void vt_recalc_intercepts(struct kvm_vcpu *vcpu) vmx_recalc_intercepts(vcpu); } +static int vt_update_hv_reg_intercepts(struct kvm_vcpu *vcpu) +{ + /* + * TDX doesn't allow descriptor-table loads to be intercepted, so + * arming fails; disarming has nothing to undo and succeeds. + */ + if (is_td_vcpu(vcpu)) + return kvm_hv_dt_intercepts_armed(vcpu) ? -EINVAL : 0; + + return vmx_update_hv_reg_intercepts(vcpu); +} + static int vt_complete_emulated_msr(struct kvm_vcpu *vcpu, int err) { if (is_td_vcpu(vcpu)) @@ -1147,6 +1159,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .check_emulate_instruction = vt_op(check_emulate_instruction), .apic_init_signal_blocked = vt_op(apic_init_signal_blocked), + .update_hv_reg_intercepts = vt_op(update_hv_reg_intercepts), .migrate_timers = vmx_migrate_timers, .recalc_intercepts = vt_op(recalc_intercepts), diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index c980ecf9202a..a33b6bd0e8a4 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -5177,6 +5177,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, * Update any VMCS fields that might have changed while vmcs02 was the * active VMCS. The tracking is per-vCPU, not per-VMCS. */ + vmx_update_hv_reg_intercepts(vcpu); vmcs_write32(VM_EXIT_MSR_STORE_COUNT, vmx->msr_autostore.nr); vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr); vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 2d85205cebb4..4e7eec9a0625 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -3595,6 +3595,42 @@ bool vmx_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) return true; } +/* + * SECONDARY_EXEC_DESC has two consumers: UMIP emulation (descriptor-table + * reads must #GP when CR4.UMIP is set and the CPU cannot do it) and the + * Hyper-V VTL register intercepts (descriptor-table writes divert into + * KVM_EXIT_HYPERV_REG_INTERCEPT, see hv_divert_dt_write()). Every arm or + * disarm of the control for one consumer must preserve the other's claim. + */ +static bool vmx_need_desc_exiting(struct kvm_vcpu *vcpu) +{ + return (vmx_umip_emulated() && + kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)) || + kvm_hv_dt_intercepts_armed(vcpu); +} + +int vmx_update_hv_reg_intercepts(struct kvm_vcpu *vcpu) +{ + struct vcpu_vmx *vmx = to_vmx(vcpu); + + if (!(vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_DESC)) + return 0; + + /* + * Only vmcs01 carries the register intercepts: prepare_vmcs02() + * strips SECONDARY_EXEC_DESC unless vmcs12 or L2's UMIP emulation + * wants it, so an L2 descriptor load is never diverted. While + * vmcs02 is loaded, leave it alone; nested_vmx_vmexit() refreshes + * vmcs01 after switching back. + */ + if (vmx->loaded_vmcs != &vmx->vmcs01) + return 0; + + secondary_exec_controls_changebit(vmx, SECONDARY_EXEC_DESC, + vmx_need_desc_exiting(vcpu)); + return 0; +} + void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) { unsigned long old_cr4 = kvm_read_cr4(vcpu); @@ -3618,8 +3654,16 @@ void vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) if (cr4 & X86_CR4_UMIP) { secondary_exec_controls_setbit(vmx, SECONDARY_EXEC_DESC); hw_cr4 &= ~X86_CR4_UMIP; - } else if (!is_guest_mode(vcpu) || - !nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC)) { + } else if (is_guest_mode(vcpu)) { + if (!nested_cpu_has2(get_vmcs12(vcpu), SECONDARY_EXEC_DESC)) + secondary_exec_controls_clearbit(vmx, + SECONDARY_EXEC_DESC); + } else if (!kvm_hv_dt_intercepts_armed(vcpu)) { + /* + * Clearing CR4.UMIP must not disarm the control + * while a higher VTL holds register intercepts on + * the descriptor tables. + */ secondary_exec_controls_clearbit(vmx, SECONDARY_EXEC_DESC); } } @@ -5698,7 +5742,14 @@ static int handle_desc(struct kvm_vcpu *vcpu) */ BUILD_BUG_ON(KVM_POSSIBLE_CR4_GUEST_BITS & X86_CR4_UMIP); - WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP)); + /* + * Descriptor-table exits are also armed when a higher VTL registers + * DT intercepts (reg_intercept_control); only an exit with neither + * UMIP emulation nor an armed intercept indicates a stale + * SECONDARY_EXEC_DESC control. + */ + WARN_ON_ONCE(!kvm_is_cr4_bit_set(vcpu, X86_CR4_UMIP) && + !kvm_hv_dt_intercepts_armed(vcpu)); return kvm_emulate_instruction(vcpu, 0); } @@ -7916,6 +7967,14 @@ void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) /* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap(vcpu); + + /* + * Bring descriptor-table exiting in line with the VTL register + * intercepts and UMIP emulation. The secondary controls set above + * leave SECONDARY_EXEC_DESC alone, so this is a consistency refresh + * rather than a repair. + */ + vmx_update_hv_reg_intercepts(vcpu); } static __init u64 vmx_get_perf_capabilities(void) diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 63e7b9256152..2e1dc101a5a6 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -53,6 +53,7 @@ void vmx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu); bool vmx_has_emulated_msr(struct kvm *kvm, u32 index); void vmx_recalc_intercepts(struct kvm_vcpu *vcpu); +int vmx_update_hv_reg_intercepts(struct kvm_vcpu *vcpu); void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void vmx_update_exception_bitmap(struct kvm_vcpu *vcpu); int vmx_get_feature_msr(u32 msr, u64 *data); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 0933291298e0..0fda04f6114a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2368,6 +2368,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_HYPERV_CPUID: case KVM_CAP_HYPERV_ENFORCE_CPUID: case KVM_CAP_SYS_HYPERV_CPUID: + case KVM_CAP_HYPERV_REG_INTERCEPT: #endif case KVM_CAP_HCALL_FAULT_EXIT: case KVM_CAP_PCI_SEGMENT: @@ -3501,9 +3502,10 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, /* * KVM_ENABLE_CAP is accepted on plane vCPU fds, but only the caps * below have been audited for plane_level > 0. The nested paths - * behind eVMCS and direct TLB flush have not; refuse those rather - * than implicitly extending every current and future vCPU cap to - * planes. + * behind eVMCS and direct TLB flush have not, and the register + * intercept only ever diverts plane 0 (hv_divert_dt_write() skips + * higher planes); refuse those rather than implicitly extending + * every current and future vCPU cap to planes. */ if (vcpu->plane_level) { switch (cap->cap) { @@ -3547,6 +3549,26 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, } return r; } + case KVM_CAP_HYPERV_REG_INTERCEPT: { + struct kvm_vcpu_hv *hv_vcpu; + int r; + + r = kvm_hv_vcpu_init(vcpu); + if (r) + return r; + hv_vcpu = to_hv_vcpu(vcpu); + hv_vcpu->reg_intercept_control = cap->args[0]; + /* + * Refresh the descriptor-table exiting controls. The vendor + * op touches only the L1 controls, so it is safe here even + * when the vCPU last exited from a nested guest and + * vcpu_load() restored vmcs02/vmcb02. + */ + r = kvm_x86_call(update_hv_reg_intercepts)(vcpu); + if (r) + hv_vcpu->reg_intercept_control = 0; + return r; + } case KVM_CAP_HYPERV_DIRECT_TLBFLUSH: if (!kvm_x86_ops.enable_l2_tlb_flush) return -ENOTTY; @@ -5879,14 +5901,57 @@ static void emulator_get_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) kvm_x86_call(get_idt)(emul_to_vcpu(ctxt), dt); } +/* + * Divert a descriptor-table write into a register-intercept exit when the + * higher VTL armed one: the write is not applied, and userspace delivers + * the TLFS register-intercept message to the higher VTL. The intercept + * contract covers only the L1 guest; a descriptor load emulated on behalf + * of a nested (L2) guest, however its exit arrived, is applied normally. + */ +static bool hv_divert_dt_write(struct x86_emulate_ctxt *ctxt, u32 hv_reg, + const struct desc_ptr *dt, u64 ctl_bit) +{ +#ifdef CONFIG_KVM_HYPERV + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + if (!hv_vcpu || vcpu->plane_level || is_guest_mode(vcpu) || + !(hv_vcpu->reg_intercept_control & ctl_bit)) + return false; + + hv_vcpu->reg_intercept.reg = hv_reg; + hv_vcpu->reg_intercept.base = dt->address; + hv_vcpu->reg_intercept.limit = dt->size; + hv_vcpu->reg_intercept.rip = ctxt->eip; + hv_vcpu->reg_intercept.insn_len = ctxt->_eip - ctxt->eip; + hv_vcpu->reg_intercept.pending = true; + return true; +#else + return false; +#endif +} + +#define HV_CR_INTERCEPT_GDTR_WRITE BIT_ULL(15) +#define HV_CR_INTERCEPT_IDTR_WRITE BIT_ULL(16) +#define HV_X64_REGISTER_IDTR_ID 0x00070000 +#define HV_X64_REGISTER_GDTR_ID 0x00070001 + static void emulator_set_gdt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) { + if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_GDTR_ID, dt, + HV_CR_INTERCEPT_GDTR_WRITE)) + return; kvm_x86_call(set_gdt)(emul_to_vcpu(ctxt), dt); } static void emulator_set_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) { - kvm_x86_call(set_idt)(emul_to_vcpu(ctxt), dt); + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + + if (hv_divert_dt_write(ctxt, HV_X64_REGISTER_IDTR_ID, dt, + HV_CR_INTERCEPT_IDTR_WRITE)) + return; + kvm_x86_call(set_idt)(vcpu, dt); } static unsigned long emulator_get_cached_segment_base( @@ -6630,6 +6695,44 @@ int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type, } EXPORT_SYMBOL_FOR_KVM_INTERNAL(x86_decode_emulated_instruction); +#ifdef CONFIG_KVM_HYPERV +/* + * Complete a descriptor-table write that hv_divert_dt_write() diverted + * during the emulation that just finished: rewind RIP and build the + * KVM_EXIT_HYPERV_REG_INTERCEPT exit. Consuming the record before + * x86_emulate_instruction() returns means it never outlives the emulation + * call: no stale rewind can fire on a later KVM_RUN, and disarming the cap + * or migrating the vCPU can never see a half-delivered intercept. + */ +static int complete_hv_reg_intercept(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + hv_vcpu->reg_intercept.pending = false; + + /* + * The intercepted instruction must not retire: the emulator advanced + * RIP when it diverted the write, but the higher VTL owns + * completion. It either applies the write and advances RIP itself, + * disarms the intercept so re-execution succeeds, or queues an + * exception that must fault AT the instruction (the + * kernel-patch-protection LIDT probe matches the faulting RIP + * against the probe site). + */ + kvm_rip_write(vcpu, hv_vcpu->reg_intercept.rip); + vcpu->run->exit_reason = KVM_EXIT_HYPERV; + vcpu->run->hyperv.type = KVM_EXIT_HYPERV_REG_INTERCEPT; + vcpu->run->hyperv.plane = vcpu->plane_level; + vcpu->run->hyperv.u.reg_intercept.reg = hv_vcpu->reg_intercept.reg; + vcpu->run->hyperv.u.reg_intercept.insn_len = + hv_vcpu->reg_intercept.insn_len; + vcpu->run->hyperv.u.reg_intercept.base = hv_vcpu->reg_intercept.base; + vcpu->run->hyperv.u.reg_intercept.limit = hv_vcpu->reg_intercept.limit; + vcpu->run->hyperv.u.reg_intercept.rip = hv_vcpu->reg_intercept.rip; + return 0; +} +#endif + int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int emulation_type, void *insn, int insn_len) { @@ -6861,6 +6964,15 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, } else vcpu->arch.emulate_regs_need_sync_to_vcpu = true; +#ifdef CONFIG_KVM_HYPERV + { + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + + if (hv_vcpu && hv_vcpu->reg_intercept.pending) + r = complete_hv_reg_intercept(vcpu); + } +#endif + return r; } diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 694409459cb8..b260d4d8aa22 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -100,6 +100,7 @@ struct kvm_hyperv_exit { #define KVM_EXIT_HYPERV_HCALL 2 #define KVM_EXIT_HYPERV_SYNDBG 3 #define KVM_EXIT_HYPERV_VTL_INTERRUPT 4 +#define KVM_EXIT_HYPERV_REG_INTERCEPT 5 __u32 type; __u32 plane; union { @@ -117,6 +118,15 @@ struct kvm_hyperv_exit { #define KVM_HYPERV_HCALL_UD (1ULL << 0) __u64 flags; } hcall; + struct { + __u32 reg; + __u8 insn_len; + __u8 pad[3]; + __u64 base; + __u32 limit; + __u32 pad2; + __u64 rip; + } reg_intercept; struct { __u32 msr; __u32 pad2; @@ -1032,6 +1042,7 @@ struct kvm_enable_cap { #define KVM_CAP_PLANES 254 #define KVM_CAP_HYPERV_FORWARD_HCALL 255 #define KVM_CAP_PLANES_SHARE 256 +#define KVM_CAP_HYPERV_REG_INTERCEPT 257 struct kvm_irq_routing_irqchip { __u32 irqchip; -- 2.50.1
