On 10/29/15 02:32, Laszlo Ersek wrote: > On 10/29/15 01:31, Yao, Jiewen wrote: >> Hi Ersek >> I think S3ResumePei supports Ia32 and Ia32X64. It does not support X64. So I >> believe Ia32X64 crash is a bug somewhere. >> >> Since you already run into SmmRestoreCpu(), would you please help to check >> where is last instruction causing crash? > > Sure. The crash occurs on the following call path (starting with > SmmRestoreCpu()): > > SmmRestoreCpu() > [UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c] > EarlyInitializeCpu() [UefiCpuPkg/PiSmmCpuDxeSmm/CpuS3.c] > SendInitSipiSipiAllExcludingSelf() > [UefiCpuPkg/Library/BaseXApicX2ApicLib/BaseXApicX2ApicLib.c] > > I have a debug print right after the > SendInitSipiSipiAllExcludingSelf(), and the BSP starts printing it, > but before the message is completely printed, one of the APs (just > started) seems to crash the VM. > > It is perfectly possible that the ACPI_CPU_DATA structure that > OvmfPkg/QuarkPort/CpuS3DataDxe collects earlier causes this. Because, > before EarlyInitializeCpu() calls SendInitSipiSipiAllExcludingSelf(), > the startup vector is prepared from ACPI_CPU_DATA, in the > PrepareApStartupVector() function. No clue what goes wrong there. > > Let me check the KVM trace though... Okay, I massaged it a little bit > for easier readability. Here's when VCPU#0 runs > SendInitSipiSipiAllExcludingSelf(): > >> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d7f info 1300 0 >> kvm_emulate_insn: 0:7ffc9d7f: 89 10 >> kvm_mmio: mmio write len 4 gpa 0xfee00300 val 0xc4697 >> kvm_apic: apic_write APIC_ICR = 0xc4697 >> kvm_apic_ipi: dst 0 vec 151 (SIPI|physical|assert|edge|all-but-self) >> kvm_apic_accept_irq: apicid 1 vec 151 (SIPI|edge) >> kvm_apic_accept_irq: apicid 2 vec 151 (SIPI|edge) >> kvm_apic_accept_irq: apicid 3 vec 151 (SIPI|edge) >> kvm_entry: vcpu 0 > >> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d06 info 300 0 >> kvm_emulate_insn: 0:7ffc9d06: 8b 00 >> kvm_apic: apic_read APIC_ICR = 0xc4697 >> kvm_mmio: mmio read len 4 gpa 0xfee00300 val 0xc4697 >> kvm_entry: vcpu 0 > >> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d7f info 1310 0 >> kvm_emulate_insn: 0:7ffc9d7f: 89 10 >> kvm_mmio: mmio write len 4 gpa 0xfee00310 val 0x0 >> kvm_apic: apic_write APIC_ICR2 = 0x0 >> kvm_entry: vcpu 0 > > And in response, this is how VCPU#1 behaves (I guess this matches > "UefiCpuPkg/PiSmmCpuDxeSmm/X64/MpFuncs.asm"?): > >> kvm_exit: reason CR_ACCESS rip 0x35 info 0 0 >> kvm_cr: cr_write 0 = 0x60000011 >> kvm_entry: vcpu 1 > >> kvm_exit: reason CR_ACCESS rip 0x9705b info 4 0 >> kvm_cr: cr_write 4 = 0x20 >> kvm_entry: vcpu 1 > >> kvm_exit: reason CR_ACCESS rip 0x9705e info 103 0 >> kvm_cr: cr_write 3 = 0x7ff83000 >> kvm_entry: vcpu 1 > >> kvm_exit: reason MSR_READ rip 0x97068 info 0 0 >> kvm_msr: msr_read c0000080 = 0x0 >> kvm_entry: vcpu 1 > >> kvm_exit: reason MSR_WRITE rip 0x9706e info 0 0 >> kvm_msr: msr_write c0000080 = 0x100 >> kvm_entry: vcpu 1 > >> kvm_exit: reason CR_ACCESS rip 0x97077 info 0 0 >> kvm_cr: cr_write 0 = 0xe0000011 >> kvm_entry: vcpu 1 > >> kvm_exit: reason TRIPLE_FAULT rip 0x9707a info 0 0 >> kvm_userspace_exit: reason KVM_EXIT_SHUTDOWN (8) > > From counting the bytes in "MpFuncs.asm", and correlating them with > the RIP values in the trace, I *think* that the triple fault happens > right here: > > LONG_JUMP:: > > db 67h, 0EAh ; far jump > dd 0h ; 32-bit offset > dw 38h ; 16-bit selector > > Do you want me to log some values from from PrepareApStartupVector()? > Like the offset that gets patched in here, or the GDT (to see that > 0x38 makes sense)... I might have to do that tomorrow ^W after > getting some sleep however, it's 02:32 AM here.
FWIW I retested with TCG too (i.e., emulation, not hw virtualization), and the VM crashes the same way. Does it matter that I have this patch in my build, temporarily: UefiCpuPkg: PiSmmCpuDxeSmm: do not execute RSM from 64-bit mode http://thread.gmane.org/gmane.comp.bios.edk2.devel/3020/focus=3023 (I don't think it matters at this stage in "MpFuncs.asm", but I should be clear about my environment...) Thanks Laszlo >> -----Original Message----- >> From: Laszlo Ersek [mailto:[email protected]] >> Sent: Thursday, October 29, 2015 7:59 AM >> To: Yao, Jiewen; Kinney, Michael D; Fan, Jeff >> Cc: edk2-devel-01 >> Subject: Re: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei >> >> On 10/28/15 23:41, Laszlo Ersek wrote: >>> On 10/28/15 23:26, Yao, Jiewen wrote: >>>> Right. It seems S3Resume2Pei does not consider X64 mode. I found at least >>>> 3 functions need enhancement on mode transition: >>>> 1) S3RestoreConfig2() - S3Resume <-> SmmCpu (DXE mode); >>>> 2) S3ResumeExecuteBootScript() - S3Resume <-> BootScriptExecutor (DXE >>>> mode) >>>> 3) S3ResumeBootOs() - S3Resume -> OS WakingVector (OS decide). >>> >>> In practice at least, these problems appear specific to SMM / SMRAM >>> usage. When we use OVMF's custom (insecure) LockBoxLib instance, the >>> X64 build of S3Resume2Pei (actually, a fully X64 build of OVMF) >>> provides a working S3 feature, including Windows 7 and later guests, >>> and Linux guests. Even a minimal boot script is executed correctly (it >>> has just an INFO opcode). >>> >>> If I remember correctly, quite a few code paths are possible through >>> S3Resume2Pei. I don't exactly recall which one is taken in the above >>> case, but I thought I'd point out that it works very well in practice. >>> (The fact notwithstanding that the lockbox is not protected from the >>> runtime guest OS.) >>> >>> The pure Ia32 case works well both with and without OVMF's SMM feature. >>> >>> I don't recall ever testing S3 with the Ia32X64 build; I plan to do >>> that soonish. >> >> Ia32X64 crashes (with SMM enabled) with the following messages leading up to >> it: >> >> -------- >> SmmLockBoxPeiLib RestoreAllLockBoxInPlace - Exit (Success) >> S3NvsPageTableAddress - 7DFDE000 (1) >> SMM S3 Signature = 534D4D53 >> SMM S3 Stack Base = 7FF8A000 >> SMM S3 Stack Size = 8000 >> SMM S3 Resume Entry Point = 7FFB5617 >> SMM S3 CR0 = 80000033 >> SMM S3 CR3 = 7FF84000 >> SMM S3 CR4 = 668 >> SMM S3 Return CS = 10 >> SMM S3 Return Entry Point = 846C69 >> SMM S3 Return Context1 = 7F6FA000 >> SMM S3 Return Context2 = 7E039000 >> SMM S3 Return Stack Pointer = 81730C >> SMM S3 Smst = 7FFFDE00 >> SmmRestoreCpu() >> <CRASH> >> -------- >> >> If I build without SMM, then Ia32X64 works fine as well. >> >> Summary: >> - without SMM: S3 works in all three of the Ia32, Ia32X64, and X64 >> OVMF builds >> - with SMM: Ia32 works, the other two crash. >> >> I guess this just confirms what you've already determined from the code. >> But, at least, it confirms it. :) >> >> Thank you all for looking into it! >> Laszlo >> >> >>> >>> Thanks, >>> Laszlo >>> >>>> Thank you >>>> Yao Jiewen >>>> >>>> -----Original Message----- >>>> From: Laszlo Ersek [mailto:[email protected]] >>>> Sent: Thursday, October 29, 2015 1:34 AM >>>> To: Kinney, Michael D; Fan, Jeff; Yao, Jiewen >>>> Cc: edk2-devel-01 >>>> Subject: Re: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei >>>> >>>> On 10/28/15 17:54, Kinney, Michael D wrote: >>>>> Laszlo, >>>>> >>>>> I do not believe any X64 PEI testing has not been performed with this >>>>> module. We will investigate a fix. >>>> >>>> Thank you. >>>> >>>> In any case, in OVMF we might be able to use this module nonetheless, with >>>> the OvmfPkgIa32X64.dsc build (== 32-bit PEI, 64-bit DXE). >>>> >>>> Thanks! >>>> Laszlo >>>> >>>>> >>>>> Thanks, >>>>> >>>>> Mike >>>>> >>>>>> -----Original Message----- >>>>>> From: Laszlo Ersek [mailto:[email protected]] >>>>>> Sent: Wednesday, October 28, 2015 8:57 AM >>>>>> To: Fan, Jeff; Yao, Jiewen >>>>>> Cc: edk2-devel-01; Kinney, Michael D >>>>>> Subject: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei >>>>>> >>>>>> Hi, >>>>>> >>>>>> I have a question about the following code in >>>>>> "UefiCpuPkg/Universal/Acpi/S3Resume2Pei/S3Resume.c", function >>>>>> S3RestoreConfig2(): >>>>>> >>>>>>> if (SmmS3ResumeState->Signature == SMM_S3_RESUME_SMM_64) { >>>>>>> // >>>>>>> // Switch to long mode to complete resume. >>>>>>> // >>>>>>> >>>>>>> InterruptStatus = SaveAndDisableInterrupts (); >>>>>>> // >>>>>>> // Need to make sure the GDT is loaded with values that >>>>>>> support long >>>>>> mode and real mode. >>>>>>> // >>>>>>> AsmWriteGdtr (&mGdt); >>>>>>> // >>>>>>> // update segment selectors per the new GDT. >>>>>>> // >>>>>>> AsmSetDataSelectors (DATA_SEGEMENT_SELECTOR); >>>>>>> // >>>>>>> // Restore interrupt state. >>>>>>> // >>>>>>> SetInterruptState (InterruptStatus); >>>>>>> >>>>>>> AsmWriteCr3 ((UINTN)SmmS3ResumeState->SmmS3Cr3); >>>>>>> >>>>>>> // >>>>>>> // Disable interrupt of Debug timer, since IDT table cannot >>>>>>> work in long >>>>>> mode. >>>>>>> // NOTE: On x64 platforms, because DisablePaging64() will >>>>>>> disable >>>>>> interrupts, >>>>>>> // the code in S3ResumeExecuteBootScript() cannot be halted >>>>>>> by soft >>>>>> debugger. >>>>>>> // >>>>>>> SaveAndSetDebugTimerInterrupt (FALSE); >>>>>>> >>>>>>> AsmEnablePaging64 ( >>>>>>> 0x38, >>>>>>> SmmS3ResumeState->SmmS3ResumeEntryPoint, >>>>>>> (UINT64)(UINTN)AcpiS3Context, >>>>>>> 0, >>>>>>> SmmS3ResumeState->SmmS3StackBase + SmmS3ResumeState- >>>>>>> SmmS3StackSize >>>>>>> ); >>>>>>> } >>>>>> >>>>>> At the end of this block, the AsmEnablePaging64() function is called. >>>>>> That call results in the following call tree, *if* the module was built >>>>>> for X64: >>>>>> >>>>>> AsmEnablePaging64() >>>>>> [MdePkg/Library/BaseLib/X86EnablePaging64.c] >>>>>> InternalX86EnablePaging64() [MdePkg/Library/BaseLib/X64/Non-existing.c] >>>>>> ASSERT (FALSE) >>>>>> >>>>>> This is because the InternalX86EnablePaging64() functionality is >>>>>> unavailable in BaseLib on X64. >>>>>> >>>>>> My question: how is this branch in S3RestoreConfig2() supposed to >>>>>> work *at >>>>>> all* in an X64 PEI build? >>>>>> >>>>>> Thank you, >>>>>> Laszlo >>>> >>> >> > _______________________________________________ edk2-devel mailing list [email protected] https://lists.01.org/mailman/listinfo/edk2-devel
