On 10/20/15 10:52, Laszlo Ersek wrote: > On 10/15/15 00:25, Laszlo Ersek wrote: > >> Test environment and results: >> >> Host kernel: >> - latest RHEL-7 development kernel (3.10.0-323.el7), with Paolo's >> following patches backported by yours truly: >> - KVM: x86: clean up kvm_arch_vcpu_runnable >> - KVM: x86: fix SMI to halted VCPU >> >> QEMU: >> - current upstream (c49d3411faae), with Paolo's patch applied: >> - target-i386: allow any alignment for SMBASE >> >> Below, the meaning of "bitness=32" is: >> * qemu-system-i386 >> * -cpu coreduo,-nx >> >> Whereas "bitness=64" means: >> * qemu-system-x86_64 >> * no special -cpu flag >> >> For variable access verification, "efibootmgr" is invoked (without >> options) at the guest OS (Fedlet 20141209) root prompt. >> >> bitness accel VCPUs result >> ------- ----- ----- ----------------------------------------------- >> 32 KVM 1 Fedlet 20141209 boots, S3 works, variables work >> >> 32 KVM 2 stuck in SMBASE relocation, APIC IDs look valid >> >> 32 TCG 1 Fedlet 20141209 boots, S3 works, variables work >> >> 32 TCG 2 Fedlet 20141209 boots, variables (efibootmgr) >> are broken -- nothing is printed > > I'm trying to track down this failure. > > I wrote a separate reproducer, a small UEFI application, that does the > following: > - call GetMemoryMap() and ExitBootServices() > - enumerate variables with GetNextVariableName() > - shut down the VM with ResetSystem(). > > This app works perfectly with #vcpus = 1 and #vcpus = 2 as well, using > TCG. This proves that runtime variable access works with multiple CPUs too. > > However, when the GetNextVariableName() calls are issued from within > Fedlet, they work only with #vcpus = 1; the same EFI service call fails > with #vcpus = 2. > > In other words, I cannot reproduce the issue from a UEFI shell app, in > spite of calling GetNextVariableName() after ExitBootServices(). > > This is extremely annoying. Something gets corrupted when the SMM-backed > variable service is called from within the Fedlet OS (with #vcpus=2 > only), but I have no clue what. > > Maybe the problem is caused by the combination of #vcpus=2 and > SetVirtualAddressMap(). Because, one aspect of the (failing) Fedlet > environment that my UEFI app does *not* reflect is the non-1:1 mapping.
I managed to collect more details. The variable access fails *iff* the access is made by a VCPU that is *not* VCPU#0. I added a bunch of debug messages to the following functions: - InitCommunicateBuffer(), SendCommunicateBuffer() [MdeModulePkg/Universal/Variable/RuntimeDxe/VariableSmmRuntimeDxe.c] - BSPHandler(), APHandler() [UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c] Now, when the "efibootmgr" utility is run on CPU#0, everything works: > [root@ovmf-fedlet ~]# taskset -c 0 efibootmgr > BootCurrent: 0004 > Timeout: 0 seconds > BootOrder: 0004,0003 > Boot0003* EFI Internal Shell > Boot0004* fedlet grub and the OVMF debug log contains a sequence of the following snippets -- the snippet is repeated once per variable: > InitCommunicateBuffer: Function=1 > InitCommunicateBuffer: Status=Success > BSPHandler: 318: CpuIndex=0 > BSPHandler: 544: CpuIndex=0 > SendCommunicateBuffer: ReturnStatus=Buffer Too Small > > InitCommunicateBuffer: Function=1 > InitCommunicateBuffer: Status=Success > BSPHandler: 318: CpuIndex=0 > BSPHandler: 544: CpuIndex=0 > SendCommunicateBuffer: ReturnStatus=Success Here Function=1 is SMM_VARIABLE_FUNCTION_GET_VARIABLE [MdeModulePkg/Include/Guid/SmmVariableCommon.h] It is one of the message types that the "unprivileged" runtime variable driver composes for the "privileged" SMM variable driver. In the above log, you can see (1) the message buffer being initialized for this message type -- in the unprivileged driver --, (2) then the BSPHandler() function starting and completing (in SMM), (3) finally the unprivileged driver looking at the status code coming back, in the message buffer, from the privileged driver. The first triplet ends with "Buffer Too Small", because efibootmgr -- apparently -- only queries the buffer size it needs to allocate for fetching the variable's contents. The second triplet succeeds. Side note: when testing variable access from a UEFI application or the UEFI shell directly, the code *always* runs on VCPU#0! That's why such testing always succeeds. Now, here's what happens when the same is attempted from VCPU#1: > [root@ovmf-fedlet ~]# taskset -c 1 efibootmgr > efibootmgr: efibootmgr: Invalid argument and the matching OVMF debug log is: > InitCommunicateBuffer: Function=1 > InitCommunicateBuffer: Status=Success > SendCommunicateBuffer: ReturnStatus=Buffer Too Small > > InitCommunicateBuffer: Function=1 > InitCommunicateBuffer: Status=Success > SendCommunicateBBSPHandler: 318: CpuIndex=0 > BSPHandler: 544: CpuIndex=0 > uffer: ReturnStatus=Buffer Too Small Observations: - the first triplet seems to complete similarly, however we don't see either BSPHandler or APHandler logs for it - the second triplet fails too (which is why efibootmgr reports EINVAL -- the second call uses the right variable buffer size, so it should succeed) - there are no APHandler() log entries at all, despite the efibootmgr process being bound to VCPU#1. Also note that BSPHandler logs CpuIndex=0. This *cannot* be right. CpuIndex should be 1 in SmiRendezvous(), APHandler() should be entered, and APHandler() should bring in the BSP. The BSP should finally dispatch the SMI to the SMM core and the variable driver. Thanks Laszlo > Paolo, can you please advise me on how to set up a KVM env / host kernel > that is supposed to work? What host kernel should I build? I have a > spare laptop (with Fedora 22 I believe) with unrestricted guest support > in the hardware. I think it should be possible to boot a Fedora 22 > userland with a fresh upstream kernel. (I won't try that with my main, > new RHEL-7 laptop.) > > I'd like to test this on KVM, because the problem doesn't seem to > reproduce unless I boot a real guest OS, and doing that on TCG is about > as exciting as watching paint dry. > > Thanks! > Laszlo > >> >> 64 KVM >=1 "KVM: entry failed, hardware error 0x80000021" >> while guest in SMBASE relocation >> >> 64 TCG 1 F21 XFCE LiveCD boots, variable access OK, S3 >> resume triggers InternalX86EnablePaging64() >> ASSERT() in >> "MdePkg/Library/BaseLib/X64/Non-existing.c". >> Looks like a bug in S3Resume2Pei? >> >> 64 TCG 2 F21 XFCE LiveCD boots, variable access >> (efibootmgr) is broken -- reports EINVAL >> >> These results are not consistent with the ones seen by Paolo, which is >> why I didn't update the last (OvmfPkg/README) patch in the series, about >> the current level of functionality. (Added a note to it about this >> fact.) >> >> What makes me glad is that the scenario that I've been testing and >> re-testing successfully since May, i.e., 32-bit/KVM/UP, *does* work. >> >> Thanks >> Laszlo >> >> Cc: Paolo Bonzini <[email protected]> >> Cc: Jordan Justen <[email protected]> >> Cc: Michael Kinney <[email protected]> >> >> Laszlo Ersek (42): >> UefiCpuPkg: CpuDxe: broadcast MTRR changes to APs >> OvmfPkg: introduce -D SMM_REQUIRE and PcdSmmSmramRequire >> OvmfPkg: Sec: force reinit of BaseExtractGuidedSectionLib handler >> table >> OvmfPkg: Sec: assert the build-time calculated end of the scratch >> buffer >> OvmfPkg: decompress FVs on S3 resume if SMM_REQUIRE is set >> OvmfPkg: PlatformPei: allow caching in AddReservedMemoryBaseSizeHob() >> OvmfPkg: PlatformPei: account for TSEG size with PcdSmmSmramRequire >> set >> OvmfPkg: add PEIM for providing TSEG-as-SMRAM during PEI >> OvmfPkg: add DXE_DRIVER for providing TSEG-as-SMRAM during boot-time >> DXE >> OvmfPkg: implement EFI_SMM_CONTROL2_PROTOCOL with a DXE_RUNTIME_DRIVER >> OvmfPkg: pull in the SMM IPL and SMM core >> OvmfPkg: pull in CpuIo2Smm driver >> OvmfPkg: AcpiS3SaveDxe: don't fake LockBox protocol if SMM_REQUIRE >> OvmfPkg: LockBox: -D SMM_REQUIRE excludes our fake lockbox >> OvmfPkg: LockBox: use SMM stack with -D SMM_REQUIRE >> OvmfPkg: resolve ReportStatusCodeLib for DXE_SMM_DRIVER modules >> OvmfPkg: resolve CpuExceptionHandlerLib for DXE_SMM_DRIVER modules >> OvmfPkg: set gUefiCpuPkgTokenSpaceGuid.PcdCpuSmmEnableBspElection to >> FALSE >> OvmfPkg: build PiSmmCpuDxeSmm for -D SMM_REQUIRE >> OvmfPkg: add skeleton QuarkPort/CpuS3DataDxe >> OvmfPkg: QuarkPort/CpuS3DataDxe: fill in ACPI_CPU_DATA.StartupVector >> OvmfPkg: QuarkPort/CpuS3DataDxe: handle IDT, GDT and MCE in >> ACPI_CPU_DATA >> OvmfPkg: QuarkPort/CpuS3DataDxe: handle StackAddress and StackSize >> OvmfPkg: import CpuConfigLib header from >> Quark_EDKII_v1.1.0/IA32FamilyCpuBasePkg >> OvmfPkg: QuarkPort/CpuS3DataDxe: fill in ACPI_CPU_DATA.NumberOfCpus >> OvmfPkg: QuarkPort/CpuS3DataDxe: fill in ACPI_CPU_DATA.MtrrTable >> OvmfPkg: QuarkPort/CpuS3DataDxe: handle register tables in >> ACPI_CPU_DATA >> OvmfPkg: build QuarkPort/CpuS3DataDxe for -D SMM_REQUIRE >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: strip trailing whitespace >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: rewrap source code to 79 >> chars >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: fix VALID_ARCHITECTURES in >> INF >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: remove FvbDevLock field >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: remove FvbScratchSpace field >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: no dual addressing needed >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: split out runtime DXE >> specifics >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: clean up includes and >> libraries >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: add DXE_SMM_DRIVER build >> OvmfPkg: QemuFlashFvbServicesRuntimeDxe: adhere to -D SMM_REQUIRE >> OvmfPkg: consolidate variable driver stack in DSC and FDF files >> OvmfPkg: pull in SMM-based variable driver stack >> OvmfPkg: double the SMRAM (TSEG) size for the 64-bit DXE phase builds >> OvmfPkg: README: document SMM status >> >> Michael Kinney (2): >> UefiCpuPkg: CpuDxe: Fix ASSERT() when only 1 CPU detected >> OvmfPkg: resolve DebugAgentLib for DXE_SMM_DRIVER modules >> >> Paolo Bonzini (8): >> UefiCpuPkg: PiSmmCpuDxeSmm: prepare PT in InitPaging before filling in >> PDE >> UefiCpuPkg: PiSmmCpuDxeSmm: do not execute RSM from 64-bit mode >> OvmfPkg: import SmmCpuFeaturesLib from UefiCpuPkg >> OvmfPkg: SmmCpuFeaturesLib: remove unnecessary bits >> OvmfPkg: SmmCpuFeaturesLib: implement SMRAM state save map access >> OvmfPkg: SmmCpuFeaturesLib: customize state save map format >> OvmfPkg: use relaxed AP SMM synchronization mode >> OvmfPkg: port CpuS3DataDxe to X64 >> >> OvmfPkg/OvmfPkg.dec >> | 29 + >> OvmfPkg/OvmfPkgIa32.dsc >> | 119 ++- >> OvmfPkg/OvmfPkgIa32X64.dsc >> | 119 ++- >> OvmfPkg/OvmfPkgX64.dsc >> | 119 ++- >> OvmfPkg/OvmfPkgIa32.fdf >> | 57 +- >> OvmfPkg/OvmfPkgIa32X64.fdf >> | 57 +- >> OvmfPkg/OvmfPkgX64.fdf >> | 57 +- >> OvmfPkg/AcpiS3SaveDxe/AcpiS3SaveDxe.inf >> | 3 +- >> OvmfPkg/Library/LockBoxLib/LockBoxBaseLib.inf >> | 3 + >> OvmfPkg/Library/LockBoxLib/LockBoxDxeLib.inf >> | 3 + >> OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf >> | 38 + >> OvmfPkg/PlatformPei/PlatformPei.inf >> | 5 + >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FvbServicesRuntimeDxe.inf >> | 44 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/{FvbServicesRuntimeDxe.inf => >> FvbServicesSmm.inf} | 53 +- >> OvmfPkg/QuarkPort/CpuS3DataDxe/CpuS3DataDxe.inf >> | 107 +++ >> OvmfPkg/Sec/SecMain.inf >> | 6 + >> OvmfPkg/SmmAccess/SmmAccess2Dxe.inf >> | 57 ++ >> OvmfPkg/SmmAccess/SmmAccessPei.inf >> | 70 ++ >> OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.inf >> | 65 ++ >> OvmfPkg/Include/Register/QemuSmramSaveStateMap.h >> | 184 +++++ >> OvmfPkg/PlatformPei/Platform.h >> | 3 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockService.h >> | 74 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.h >> | 9 +- >> OvmfPkg/QuarkPort/CpuS3DataDxe/Cpu.h >> | 97 +++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/ArchSpecificDef.h >> | 57 ++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/MpApic.h >> | 69 ++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/MpCommon.h >> | 187 +++++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/MpService.h >> | 49 ++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/X64/ArchSpecificDef.h >> | 59 ++ >> OvmfPkg/QuarkPort/Include/Library/CpuConfigLib.h >> | 671 +++++++++++++++++ >> OvmfPkg/SmmAccess/SmramInternal.h >> | 89 +++ >> UefiCpuPkg/CpuDxe/CpuMp.h >> | 13 + >> OvmfPkg/AcpiS3SaveDxe/AcpiS3Save.c >> | 14 +- >> OvmfPkg/Library/LockBoxLib/LockBoxLib.c >> | 2 + >> OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >> | 790 ++++++++++++++++++++ >> OvmfPkg/PlatformPei/Fv.c >> | 27 +- >> OvmfPkg/PlatformPei/MemDetect.c >> | 85 ++- >> OvmfPkg/PlatformPei/Platform.c >> | 9 +- >> OvmfPkg/PlatformPei/Xen.c >> | 2 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FvbInfo.c >> | 56 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockService.c >> | 653 +++++++--------- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockServiceDxe.c >> | 157 ++++ >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockServiceSmm.c >> | 69 ++ >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlash.c >> | 27 +- >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c >> | 28 + >> OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/ArchSpecific.c >> | 105 +++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/MpApic.c >> | 93 +++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/MpCommon.c >> | 298 ++++++++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/ProcessorConfig.c >> | 335 +++++++++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/X64/ArchSpecific.c >> | 108 +++ >> OvmfPkg/Sec/SecMain.c >> | 52 +- >> OvmfPkg/SmmAccess/SmmAccess2Dxe.c >> | 156 ++++ >> OvmfPkg/SmmAccess/SmmAccessPei.c >> | 446 +++++++++++ >> OvmfPkg/SmmAccess/SmramInternal.c >> | 188 +++++ >> OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.c >> | 365 +++++++++ >> UefiCpuPkg/CpuDxe/CpuDxe.c >> | 26 + >> UefiCpuPkg/CpuDxe/CpuMp.c >> | 36 +- >> UefiCpuPkg/PiSmmCpuDxeSmm/SmmProfile.c >> | 8 +- >> OvmfPkg/DecomprScratchEnd.fdf.inc >> | 72 ++ >> OvmfPkg/OvmfPkg.fdf.inc >> | 2 + >> OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/CpuAsm.S >> | 56 ++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/CpuAsm.asm >> | 69 ++ >> OvmfPkg/QuarkPort/CpuS3DataDxe/X64/CpuAsm.nasm >> | 58 ++ >> OvmfPkg/README >> | 43 ++ >> UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmiEntry.S >> | 13 + >> UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmiEntry.asm >> | 13 + >> UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmInit.S >> | 13 + >> UefiCpuPkg/PiSmmCpuDxeSmm/X64/SmmInit.asm >> | 13 + >> 68 files changed, 6468 insertions(+), 591 deletions(-) >> create mode 100644 OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf >> copy OvmfPkg/QemuFlashFvbServicesRuntimeDxe/{FvbServicesRuntimeDxe.inf => >> FvbServicesSmm.inf} (59%) >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/CpuS3DataDxe.inf >> create mode 100644 OvmfPkg/SmmAccess/SmmAccess2Dxe.inf >> create mode 100644 OvmfPkg/SmmAccess/SmmAccessPei.inf >> create mode 100644 OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.inf >> create mode 100644 OvmfPkg/Include/Register/QemuSmramSaveStateMap.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/Cpu.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/ArchSpecificDef.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/MpApic.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/MpCommon.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/MpService.h >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/X64/ArchSpecificDef.h >> create mode 100644 OvmfPkg/QuarkPort/Include/Library/CpuConfigLib.h >> create mode 100644 OvmfPkg/SmmAccess/SmramInternal.h >> create mode 100644 OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c >> create mode 100644 >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockServiceDxe.c >> create mode 100644 >> OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FwBlockServiceSmm.c >> create mode 100644 OvmfPkg/QemuFlashFvbServicesRuntimeDxe/QemuFlashDxe.c >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/ArchSpecific.c >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/MpApic.c >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/MpCommon.c >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/ProcessorConfig.c >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/X64/ArchSpecific.c >> create mode 100644 OvmfPkg/SmmAccess/SmmAccess2Dxe.c >> create mode 100644 OvmfPkg/SmmAccess/SmmAccessPei.c >> create mode 100644 OvmfPkg/SmmAccess/SmramInternal.c >> create mode 100644 OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.c >> create mode 100644 OvmfPkg/DecomprScratchEnd.fdf.inc >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/CpuAsm.S >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/IA32/CpuAsm.asm >> create mode 100644 OvmfPkg/QuarkPort/CpuS3DataDxe/X64/CpuAsm.nasm >> > > _______________________________________________ > edk2-devel mailing list > [email protected] > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ edk2-devel mailing list [email protected] https://lists.01.org/mailman/listinfo/edk2-devel
