On 10/29/15 01:31, Yao, Jiewen wrote:
> Hi Ersek
> I think S3ResumePei supports Ia32 and Ia32X64. It does not support X64. So I
> believe Ia32X64 crash is a bug somewhere.
>
> Since you already run into SmmRestoreCpu(), would you please help to check
> where is last instruction causing crash?
Sure. The crash occurs on the following call path (starting with
SmmRestoreCpu()):
SmmRestoreCpu()
[UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c]
EarlyInitializeCpu() [UefiCpuPkg/PiSmmCpuDxeSmm/CpuS3.c]
SendInitSipiSipiAllExcludingSelf()
[UefiCpuPkg/Library/BaseXApicX2ApicLib/BaseXApicX2ApicLib.c]
I have a debug print right after the SendInitSipiSipiAllExcludingSelf(), and
the BSP starts printing it, but before the message is completely printed, one
of the APs (just started) seems to crash the VM.
It is perfectly possible that the ACPI_CPU_DATA structure that
OvmfPkg/QuarkPort/CpuS3DataDxe collects earlier causes this. Because, before
EarlyInitializeCpu() calls SendInitSipiSipiAllExcludingSelf(), the startup
vector is prepared from ACPI_CPU_DATA, in the PrepareApStartupVector()
function. No clue what goes wrong there.
Let me check the KVM trace though... Okay, I massaged it a little bit for
easier readability. Here's when VCPU#0 runs SendInitSipiSipiAllExcludingSelf():
> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d7f info 1300 0
> kvm_emulate_insn: 0:7ffc9d7f: 89 10
> kvm_mmio: mmio write len 4 gpa 0xfee00300 val 0xc4697
> kvm_apic: apic_write APIC_ICR = 0xc4697
> kvm_apic_ipi: dst 0 vec 151 (SIPI|physical|assert|edge|all-but-self)
> kvm_apic_accept_irq: apicid 1 vec 151 (SIPI|edge)
> kvm_apic_accept_irq: apicid 2 vec 151 (SIPI|edge)
> kvm_apic_accept_irq: apicid 3 vec 151 (SIPI|edge)
> kvm_entry: vcpu 0
> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d06 info 300 0
> kvm_emulate_insn: 0:7ffc9d06: 8b 00
> kvm_apic: apic_read APIC_ICR = 0xc4697
> kvm_mmio: mmio read len 4 gpa 0xfee00300 val 0xc4697
> kvm_entry: vcpu 0
> kvm_exit: reason APIC_ACCESS rip 0x7ffc9d7f info 1310 0
> kvm_emulate_insn: 0:7ffc9d7f: 89 10
> kvm_mmio: mmio write len 4 gpa 0xfee00310 val 0x0
> kvm_apic: apic_write APIC_ICR2 = 0x0
> kvm_entry: vcpu 0
And in response, this is how VCPU#1 behaves (I guess this matches
"UefiCpuPkg/PiSmmCpuDxeSmm/X64/MpFuncs.asm"?):
> kvm_exit: reason CR_ACCESS rip 0x35 info 0 0
> kvm_cr: cr_write 0 = 0x60000011
> kvm_entry: vcpu 1
> kvm_exit: reason CR_ACCESS rip 0x9705b info 4 0
> kvm_cr: cr_write 4 = 0x20
> kvm_entry: vcpu 1
> kvm_exit: reason CR_ACCESS rip 0x9705e info 103 0
> kvm_cr: cr_write 3 = 0x7ff83000
> kvm_entry: vcpu 1
> kvm_exit: reason MSR_READ rip 0x97068 info 0 0
> kvm_msr: msr_read c0000080 = 0x0
> kvm_entry: vcpu 1
> kvm_exit: reason MSR_WRITE rip 0x9706e info 0 0
> kvm_msr: msr_write c0000080 = 0x100
> kvm_entry: vcpu 1
> kvm_exit: reason CR_ACCESS rip 0x97077 info 0 0
> kvm_cr: cr_write 0 = 0xe0000011
> kvm_entry: vcpu 1
> kvm_exit: reason TRIPLE_FAULT rip 0x9707a info 0 0
> kvm_userspace_exit: reason KVM_EXIT_SHUTDOWN (8)
>From counting the bytes in "MpFuncs.asm", and correlating them with the RIP
>values in the trace, I *think* that the triple fault happens right here:
LONG_JUMP::
db 67h, 0EAh ; far jump
dd 0h ; 32-bit offset
dw 38h ; 16-bit selector
Do you want me to log some values from from PrepareApStartupVector()? Like the
offset that gets patched in here, or the GDT (to see that 0x38 makes sense)...
I might have to do that tomorrow ^W after getting some sleep however, it's
02:32 AM here.
Thanks!
Laszlo
>
> Thank you
> Yao Jiewen
>
> -----Original Message-----
> From: Laszlo Ersek [mailto:[email protected]]
> Sent: Thursday, October 29, 2015 7:59 AM
> To: Yao, Jiewen; Kinney, Michael D; Fan, Jeff
> Cc: edk2-devel-01
> Subject: Re: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei
>
> On 10/28/15 23:41, Laszlo Ersek wrote:
>> On 10/28/15 23:26, Yao, Jiewen wrote:
>>> Right. It seems S3Resume2Pei does not consider X64 mode. I found at least 3
>>> functions need enhancement on mode transition:
>>> 1) S3RestoreConfig2() - S3Resume <-> SmmCpu (DXE mode);
>>> 2) S3ResumeExecuteBootScript() - S3Resume <-> BootScriptExecutor (DXE
>>> mode)
>>> 3) S3ResumeBootOs() - S3Resume -> OS WakingVector (OS decide).
>>
>> In practice at least, these problems appear specific to SMM / SMRAM
>> usage. When we use OVMF's custom (insecure) LockBoxLib instance, the
>> X64 build of S3Resume2Pei (actually, a fully X64 build of OVMF)
>> provides a working S3 feature, including Windows 7 and later guests,
>> and Linux guests. Even a minimal boot script is executed correctly (it
>> has just an INFO opcode).
>>
>> If I remember correctly, quite a few code paths are possible through
>> S3Resume2Pei. I don't exactly recall which one is taken in the above
>> case, but I thought I'd point out that it works very well in practice.
>> (The fact notwithstanding that the lockbox is not protected from the
>> runtime guest OS.)
>>
>> The pure Ia32 case works well both with and without OVMF's SMM feature.
>>
>> I don't recall ever testing S3 with the Ia32X64 build; I plan to do
>> that soonish.
>
> Ia32X64 crashes (with SMM enabled) with the following messages leading up to
> it:
>
> --------
> SmmLockBoxPeiLib RestoreAllLockBoxInPlace - Exit (Success)
> S3NvsPageTableAddress - 7DFDE000 (1)
> SMM S3 Signature = 534D4D53
> SMM S3 Stack Base = 7FF8A000
> SMM S3 Stack Size = 8000
> SMM S3 Resume Entry Point = 7FFB5617
> SMM S3 CR0 = 80000033
> SMM S3 CR3 = 7FF84000
> SMM S3 CR4 = 668
> SMM S3 Return CS = 10
> SMM S3 Return Entry Point = 846C69
> SMM S3 Return Context1 = 7F6FA000
> SMM S3 Return Context2 = 7E039000
> SMM S3 Return Stack Pointer = 81730C
> SMM S3 Smst = 7FFFDE00
> SmmRestoreCpu()
> <CRASH>
> --------
>
> If I build without SMM, then Ia32X64 works fine as well.
>
> Summary:
> - without SMM: S3 works in all three of the Ia32, Ia32X64, and X64
> OVMF builds
> - with SMM: Ia32 works, the other two crash.
>
> I guess this just confirms what you've already determined from the code.
> But, at least, it confirms it. :)
>
> Thank you all for looking into it!
> Laszlo
>
>
>>
>> Thanks,
>> Laszlo
>>
>>> Thank you
>>> Yao Jiewen
>>>
>>> -----Original Message-----
>>> From: Laszlo Ersek [mailto:[email protected]]
>>> Sent: Thursday, October 29, 2015 1:34 AM
>>> To: Kinney, Michael D; Fan, Jeff; Yao, Jiewen
>>> Cc: edk2-devel-01
>>> Subject: Re: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei
>>>
>>> On 10/28/15 17:54, Kinney, Michael D wrote:
>>>> Laszlo,
>>>>
>>>> I do not believe any X64 PEI testing has not been performed with this
>>>> module. We will investigate a fix.
>>>
>>> Thank you.
>>>
>>> In any case, in OVMF we might be able to use this module nonetheless, with
>>> the OvmfPkgIa32X64.dsc build (== 32-bit PEI, 64-bit DXE).
>>>
>>> Thanks!
>>> Laszlo
>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>>> -----Original Message-----
>>>>> From: Laszlo Ersek [mailto:[email protected]]
>>>>> Sent: Wednesday, October 28, 2015 8:57 AM
>>>>> To: Fan, Jeff; Yao, Jiewen
>>>>> Cc: edk2-devel-01; Kinney, Michael D
>>>>> Subject: about the SMM_S3_RESUME_SMM_64 branch in S3Resume2Pei
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have a question about the following code in
>>>>> "UefiCpuPkg/Universal/Acpi/S3Resume2Pei/S3Resume.c", function
>>>>> S3RestoreConfig2():
>>>>>
>>>>>> if (SmmS3ResumeState->Signature == SMM_S3_RESUME_SMM_64) {
>>>>>> //
>>>>>> // Switch to long mode to complete resume.
>>>>>> //
>>>>>>
>>>>>> InterruptStatus = SaveAndDisableInterrupts ();
>>>>>> //
>>>>>> // Need to make sure the GDT is loaded with values that
>>>>>> support long
>>>>> mode and real mode.
>>>>>> //
>>>>>> AsmWriteGdtr (&mGdt);
>>>>>> //
>>>>>> // update segment selectors per the new GDT.
>>>>>> //
>>>>>> AsmSetDataSelectors (DATA_SEGEMENT_SELECTOR);
>>>>>> //
>>>>>> // Restore interrupt state.
>>>>>> //
>>>>>> SetInterruptState (InterruptStatus);
>>>>>>
>>>>>> AsmWriteCr3 ((UINTN)SmmS3ResumeState->SmmS3Cr3);
>>>>>>
>>>>>> //
>>>>>> // Disable interrupt of Debug timer, since IDT table cannot
>>>>>> work in long
>>>>> mode.
>>>>>> // NOTE: On x64 platforms, because DisablePaging64() will
>>>>>> disable
>>>>> interrupts,
>>>>>> // the code in S3ResumeExecuteBootScript() cannot be halted
>>>>>> by soft
>>>>> debugger.
>>>>>> //
>>>>>> SaveAndSetDebugTimerInterrupt (FALSE);
>>>>>>
>>>>>> AsmEnablePaging64 (
>>>>>> 0x38,
>>>>>> SmmS3ResumeState->SmmS3ResumeEntryPoint,
>>>>>> (UINT64)(UINTN)AcpiS3Context,
>>>>>> 0,
>>>>>> SmmS3ResumeState->SmmS3StackBase + SmmS3ResumeState-
>>>>>> SmmS3StackSize
>>>>>> );
>>>>>> }
>>>>>
>>>>> At the end of this block, the AsmEnablePaging64() function is called.
>>>>> That call results in the following call tree, *if* the module was built
>>>>> for X64:
>>>>>
>>>>> AsmEnablePaging64() [MdePkg/Library/BaseLib/X86EnablePaging64.c]
>>>>> InternalX86EnablePaging64() [MdePkg/Library/BaseLib/X64/Non-existing.c]
>>>>> ASSERT (FALSE)
>>>>>
>>>>> This is because the InternalX86EnablePaging64() functionality is
>>>>> unavailable in BaseLib on X64.
>>>>>
>>>>> My question: how is this branch in S3RestoreConfig2() supposed to
>>>>> work *at
>>>>> all* in an X64 PEI build?
>>>>>
>>>>> Thank you,
>>>>> Laszlo
>>>
>>
>
_______________________________________________
edk2-devel mailing list
[email protected]
https://lists.01.org/mailman/listinfo/edk2-devel
