I've recently discovered that if you attempt to use introspection to capture CR3 changes with the new KPTI enabled kernels, the guest dies shortly after the start of introspection with failed VM entry due to invalid guest state.
I believe the invalid state here is the high bit being set in CR3 - while this is how one indicates that PCID should not invalidate the various page table caches, introspection leads to this being set in the VMCS, which appears to be wrong. With the XenServer 4.7.1 code base (which is my working code base at the moment), I have not found a way around this, as the vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set the CR3 value, and vm_event_register_write_resume only allows inhibiting the write, not writing a modified value. I've attempted several ways to work around this with a livepatch, and have not (yet) been successful. Masking at the top of hvm_set_cr3 allows the guest to continue, but appears to do the wrong thing with regards to the guest (tasks begin dying quickly from invalid opcode errors). In any case, Andrew mentions that this appears to still be an issue in staging, so this likely needs addressing. At this point in time, I believe guests with KPTI enabled cannot be introspected if that introspection involves capturing CR3 changes. Please let me know if you need any more details on this issue! -Bit (XEN) [19458.318035] Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0). (XEN) [19458.318042] ************* VMCS Area ************** (XEN) [19458.318050] *** Guest State *** (XEN) [19458.318056] CR0: actual=0x000000008005003b, shadow=0x0000000080050033, gh_mask=ffffffffffffffff (XEN) [19458.318062] CR4: actual=0x0000000000362670, shadow=0x0000000000360670, gh_mask=ffffffffffffffff (XEN) [19458.318069] CR3 = 0x800000001ded7080 (XEN) [19458.318076] PDPTE0 = 0x0000000000020000 PDPTE1 = 0x000006f800150018 (XEN) [19458.318082] PDPTE2 = 0x0000000000000000 PDPTE3 = 0x000006f800150018 (XEN) [19458.318089] RSP = 0xffff880015b87f50 (0xffff880015b87f50) RIP = 0xffffffff81845857 (0xffffffff81845857) (XEN) [19458.318095] RFLAGS=0x00000082 (0x00000082) DR7 = 0x0000000000000400 (XEN) [19458.318101] Sysenter RSP=0000000000000000 CS:RIP=0010:ffffffff8184a220 (XEN) [19458.318105] sel attr limit base (XEN) [19458.318112] CS: 0010 0a09b ffffffff 0000000000000000 (XEN) [19458.318119] DS: 0000 1c000 ffffffff 0000000000000000 (XEN) [19458.318126] SS: 0018 0c093 ffffffff 0000000000000000 (XEN) [19458.318133] ES: 0000 1c000 ffffffff 0000000000000000 (XEN) [19458.318140] FS: 0000 1c000 ffffffff 00007fde038ba700 (XEN) [19458.318147] GS: 0000 1c000 ffffffff ffff88001ba00000 (XEN) [19458.318152] GDTR: 0000007f ffff88001ba0c000 (XEN) [19458.318158] LDTR: 0000 1c000 ffffffff 0000000000000000 (XEN) [19458.318164] IDTR: 00000fff ffffffffff574000 (XEN) [19458.318169] TR: 0040 0008b 00002087 ffff88001ba048c0 (XEN) [19458.318175] EFER = 0x0000000000000000 PAT = 0x0407010600070106 (XEN) [19458.318179] PreemptionTimer = 0x00000000 SM Base = 0x00000000 (XEN) [19458.318185] DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 (XEN) [19458.318233] PerfGlobCtl = 0x0000000000000000 BndCfgS = 0x0000000000000000 (XEN) [19458.318297] Interruptibility = 00000000 ActivityState = 00000000 (XEN) [19458.318324] *** Host State *** (XEN) [19458.318329] RIP = 0xffff82d0801ee100 (vmx_asm_vmexit_handler) RSP = 0xffff8300bfcfff90 (XEN) [19458.318333] CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000 TR=e040 (XEN) [19458.318335] FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff82d08035e780 (XEN) [19458.318337] GDTBase=ffff82d0802d9000 IDTBase=ffff82d080357ce0 (XEN) [19458.318339] CR0=000000008005003b CR3=000000010f001000 CR4=00000000003526e0 (XEN) [19458.318341] Sysenter RSP=ffff8300bfcfffc0 CS:RIP=e008:ffff82d08022bb30 (XEN) [19458.318343] EFER = 0x0000000000000000 PAT = 0x0000050100070406 (XEN) [19458.318344] *** Control State *** (XEN) [19458.318347] PinBased=0000003f CPUBased=b6a0e5fa SecondaryExec=001014ea (XEN) [19458.318348] EntryControls=000153ff ExitControls=008fefff (XEN) [19458.318350] ExceptionBitmap=00060082 PFECmask=00000000 PFECmatch=00000000 (XEN) [19458.318352] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000 (XEN) [19458.318353] VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 (XEN) [19458.318355] reason=80000021 qualification=0000000000000000 (XEN) [19458.318357] IDTVectoring: info=00000000 errcode=00000000 (XEN) [19458.318359] TSC Offset = 0xffffd23bbd8772ac TSC Multiplier = 0x0000000000000000 (XEN) [19458.318361] TPR Threshold = 0x00 PostedIntrVec = 0x00 (XEN) [19458.318365] EPT pointer = 0x000000010ee9501e EPTP index = 0x0000 (XEN) [19458.318396] PLE Gap=00000080 Window=00001000 (XEN) [19458.318402] Virtual processor ID = 0xccd3 VMfunc controls = 0000000000000000 (XEN) [19458.318406] ************************************** (XEN) [19458.318412] domain_crash called from vmx_vmexit_handler+0x4ab/0x19f5 (XEN) [19458.318417] Domain 15 (vcpu#0) crashed on cpu#0: (XEN) [19458.318443] ----[ Xen-4.7.1-1.0 x86_64 debug=n Not tainted ]---- (XEN) [19458.318448] CPU: 0 (XEN) [19458.318453] RIP: 0010:[<ffffffff81845857>] (XEN) [19458.318458] RFLAGS: 0000000000000082 CONTEXT: hvm guest (d15v0) (XEN) [19458.318466] rax: 800000001ded7080 rbx: 0000000000000000 rcx: 00007fde033ce730 (XEN) [19458.318470] rdx: 00000000000000fa rsi: 0000000000000002 rdi: 00007ffd8ee85250 (XEN) [19458.318484] rbp: 00007ffd8ee85410 rsp: ffff880015b87f50 r8: 0000000000000000 (XEN) [19458.318498] r9: 0000000000000017 r10: 0000000000000000 r11: 0000000000000246 (XEN) [19458.318502] r12: 00007ffd8ee85250 r13: 0000000000000000 r14: 0000000000000004 (XEN) [19458.318525] r15: 000055a8503b3828 cr0: 0000000080050033 cr4: 0000000000360670 (XEN) [19458.318538] cr3: 800000001ded7080 cr2: 00007ffef290a090 (XEN) [19458.318552] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0018 cs: 0010
_______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
