Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Andrew Cooper]

On 18/08/17 16:55, Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 16, 2017 at 06:47:23PM +, Andri Möll wrote:
>
>> (d1) Invoking OVMF ...
>> (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff ff ff 
>> 10 00 b8 40 06 00 00 0f 22
> That code is:
> cripts/decodecode 
> Code: 66 ea 5c ff ff ff 10 00 b8 40 06 00 00 0f 22
> Code: 66 ea 5c ff ff ff 10 00 b8 40 06 00 00 0f 22
> sed: -e expression #1, char 1: unknown command: `-'
>
> Code starting with the faulting instruction
> ===
>0:   66 ea   data16 (bad) 
>2:   5c  pop%rsp
>3:   ff  (bad)  
>4:   ff  (bad)  
>5:   ff 10   callq  *(%rax)
>7:   00 b8 40 06 00 00   add%bh,0x640(%rax)
>d:   0f  .byte 0xf
>e:   22  .byte 0x22
>
> Which looks to be garbage.

That is because you're disassembling it as 64bit code, not 16. :)

The offending instruction is actually ljmpl $0x10,$0xff5c, and is
almost certainly following a write to CR0 which enables protected mode.

0xea is not valid in 64bit mode.  Decoding it is already complicated
because it takes two adjacent immediate operands, with the offset
encoded before the segment. There is no "immediate operand override"
prefix in x86, so making the instruction usable in a 64bit code segment
is tricky.  Given how rarely it is used, I expect AMD decided it wasn't
worth the effort or silicon trying to make it work.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Konrad Rzeszutek Wilk]

On Wed, Aug 16, 2017 at 06:47:23PM +, Andri Möll wrote:
> Hey,
> 
> As per Andrew [Cooper]'s suggestion, writing here instead of #xen on
> Freenode.
> 
> I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having it

OK, so this is
ommit 3858b4a1ff09d3243fea8d07bd135478237cb8f7
Author: Ard Biesheuvel 
Date:   Wed Mar 1 18:34:33 2017 +

ArmPlatformPkg/PlatformIntelBdsLib: don't clobber ConSplitter handle

Which looks to be done right after the 4MB increase. What is 
the side of the binary blob?

> crash right on boot both with the 32b and 64b OVMF binaries. This is on Arch

Did you build them as RELEASE or DEBUG?

> Linux, AMD Ryzen on a X370 motherboard.
> 
> Given the following minimal VM declaration:
> > builder = "hvm"
> > maxmem = 512
> > memory = 512
> > vcpus = 1
> > on_poweroff = "destroy"
> > on_reboot = "destroy"
> > on_crash = "destroy"
> > bios = "ovmf"
> > device_model_version = "qemu-xen"
> > bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"
> and running it with `xl create vm.cfg`, I see it crash while booting with
> the following displayed by `xl dmesg`:
> 
> > (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff
> > ff ff 10 00 b8 40 06 00 00 0f 22
> > (XEN) d1v0 Triple fault - invoking HVM shutdown action 1
> I've run the hypervisor with `guest_loglvl=all` for more output and attached
> it here and uploaded it at
> https://gist.github.com/moll/a46dffc7466ced93a0365a6916a4db96 in case the
> file doesn't go through.
> 
> Any ideas anyone? Thanks in advance!
> 
> Andri

> (XEN) HVM1 save: CPU
> (XEN) HVM1 save: PIC
> (XEN) HVM1 save: IOAPIC
> (XEN) HVM1 save: LAPIC
> (XEN) HVM1 save: LAPIC_REGS
> (XEN) HVM1 save: PCI_IRQ
> (XEN) HVM1 save: ISA_IRQ
> (XEN) HVM1 save: PCI_LINK
> (XEN) HVM1 save: PIT
> (XEN) HVM1 save: RTC
> (XEN) HVM1 save: HPET
> (XEN) HVM1 save: PMTIMER
> (XEN) HVM1 save: MTRR
> (XEN) HVM1 save: VIRIDIAN_DOMAIN
> (XEN) HVM1 save: CPU_XSAVE
> (XEN) HVM1 save: VIRIDIAN_VCPU
> (XEN) HVM1 save: VMCE_VCPU
> (XEN) HVM1 save: TSC_ADJUST
> (XEN) HVM1 save: CPU_MSR
> (XEN) HVM1 restore: CPU 0
> (d1) HVM Loader
> (d1) Detected Xen v4.9.0
> (d1) Xenbus rings @0xfeffc000, event channel 1
> (d1) System requested OVMF
> (d1) CPU speed is 3001 MHz
> (d1) Relocating guest memory for lowmem MMIO space disabled
> (d1) PCI-ISA link 0 routed to IRQ5
> (d1) PCI-ISA link 1 routed to IRQ10
> (d1) PCI-ISA link 2 routed to IRQ11
> (d1) PCI-ISA link 3 routed to IRQ5
> (d1) pci dev 01:3 INTA->IRQ10
> (d1) pci dev 02:0 INTA->IRQ11
> (d1) No RAM in high memory; setting high_mem resource base to 1
> (d1) pci dev 03:0 bar 10 size 00200: 0f008
> (d1) pci dev 02:0 bar 14 size 00100: 0f208
> (d1) pci dev 03:0 bar 30 size 1: 0f300
> (d1) pci dev 03:0 bar 14 size 01000: 0f301
> (d1) pci dev 02:0 bar 10 size 00100: 0c001
> (d1) pci dev 01:1 bar 20 size 00010: 0c101
> (d1) Multiprocessor initialisation:
> (d1)  - CPU0 ... 48-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
> (d1) Writing SMBIOS tables ...
> (d1) Loading OVMF ...
> (XEN) d1v0 Over-allocation for domain 1: 131329 > 131328
> (d1) Loading ACPI ...
> (d1) CONV disabled
> (d1) vm86 TSS at fc00a400
> (d1) BIOS map:
> (d1)  ffe0-fffd: Main BIOS
> (d1) E820 table:
> (d1)  [00]: : - :000a: RAM
> (d1)  HOLE: :000a - :000f
> (d1)  [01]: :000f - :0010: RESERVED
> (d1)  [02]: :0010 - :1f715000: RAM
> (d1)  HOLE: :1f715000 - :fc00
> (d1)  [03]: :fc00 - 0001:: RESERVED
> (d1) Invoking OVMF ...
> (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff ff ff 
> 10 00 b8 40 06 00 00 0f 22

That code is:
cripts/decodecode 
Code: 66 ea 5c ff ff ff 10 00 b8 40 06 00 00 0f 22
Code: 66 ea 5c ff ff ff 10 00 b8 40 06 00 00 0f 22
sed: -e expression #1, char 1: unknown command: `-'

Code starting with the faulting instruction
===
   0:   66 ea   data16 (bad) 
   2:   5c  pop%rsp
   3:   ff  (bad)  
   4:   ff  (bad)  
   5:   ff 10   callq  *(%rax)
   7:   00 b8 40 06 00 00   add%bh,0x640(%rax)
   d:   0f  .byte 0xf
   e:   22  .byte 0x22

Which looks to be garbage.

Also can you share what version of compiler you are using GCC?

And did you build the OVMF out of tree or use the Makefile and such that
came with Xen?

There is a way to get an good idea of where things are going bad by
cranked the debug up and making an special port be pipped to
a file (which you should be able to do with the crafty usage of extra
guest config parameters).

> (XEN) d1v0 Triple fault - invoking HVM shutdown action 1
> (XEN) *** Dumping Dom1 vcpu#0 state: ***
> (XEN) [ Xen-4.9.0  x86_64  debug=n   Not tainted 

Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Alexey G]

On Thu, 17 Aug 2017 11:56:06 +0100
Andrew Cooper  wrote:

> On 17/08/17 09:49, Jan Beulich wrote:
>  On 16.08.17 at 20:47,  wrote:  
> >> Hey,
> >>
> >> As per Andrew [Cooper]'s suggestion, writing here instead of #xen on 
> >> Freenode.
> >>
> >> I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having
> >> it crash right on boot both with the 32b and 64b OVMF binaries. This
> >> is on Arch Linux, AMD Ryzen on a X370 motherboard.
> >>
> >> Given the following minimal VM declaration:  
> >>> builder = "hvm"
> >>> maxmem = 512
> >>> memory = 512
> >>> vcpus = 1
> >>> on_poweroff = "destroy"
> >>> on_reboot = "destroy"
> >>> on_crash = "destroy"
> >>> bios = "ovmf"
> >>> device_model_version = "qemu-xen"
> >>> bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"  
> >> and running it with `xl create vm.cfg`, I see it crash while booting 
> >> with the following displayed by `xl dmesg`:
> >>  
> >>> (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c
> >>> ff ff ff 10 00 b8 40 06 00 00 0f 22
> >>> (XEN) d1v0 Triple fault - invoking HVM shutdown action 1  
> >> I've run the hypervisor with `guest_loglvl=all` for more output and 
> >> attached it here and uploaded it at 
> >> https://gist.github.com/moll/a46dffc7466ced93a0365a6916a4db96 in case 
> >> the file doesn't go through.  
> > Looks to be an ordinary 32-bit far branch after having switched to
> > protected mode. I'm afraid without seeing the involved GDT entry
> > there's little chance of guessing what may go wrong in this case.
> > One question is why the emulator is being invoked in the first place:
> > Since you've truncated the log at the beginning, it's impossible to
> > tell whether you're using old Intel hardware lacking the Unrestricted
> > Guest feature.  
> 
> (As included above), This is on Arch Linux, AMD Ryzen on a X370
> motherboard.
> 
> I can't work out why we hitting the MMIO path in this case,
> independently of why the emulation of this instruction failed.

Seems like the root cause of the issue is supplying ovmf image of
non-aligned size.

(OVMF_MAXOFFSET is 0Fh, bios_length is 1920Kb in our case)

uint64_t addr = OVMF_END - ((bios_length + OVMF_MAXOFFSET) & ~OVMF_MAXOFFSET);
uint64_t ovmf_end = addr + bios_length;

-- this code expects bios_length to be aligned to 1MB boundary, otherwise
it won't be written next to 4Gb boundary. In this case bios image
actually written to the address (4GB - 2MB), while its length is less than
2MB.
Due to this, high memory copy of the BIOS appears shifted down. And
mem_hole_populate_ram() leaves gap between (4GB - 2MB + 1920Kb) and 4GB. So
when it jumps from 16bit F-seg to PM32 linear address space via jmp far
10h:0FF5Ch, there is no RAM mapped here, hence MMIO. 

A proper OVMF build (with ovmf_code_ia32.bin and ovmf_vars_ia32.bin
merging) will likely fix the issue, but above code looks a bit strange
anyway -- it does 1MB alignment, but if there was actual alignment, the
BIOS will be loaded to the wrong address, not near 4GB boundary.

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Andrew Cooper]

On 17/08/17 09:49, Jan Beulich wrote:
 On 16.08.17 at 20:47,  wrote:
>> Hey,
>>
>> As per Andrew [Cooper]'s suggestion, writing here instead of #xen on 
>> Freenode.
>>
>> I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having it 
>> crash right on boot both with the 32b and 64b OVMF binaries. This is on 
>> Arch Linux, AMD Ryzen on a X370 motherboard.
>>
>> Given the following minimal VM declaration:
>>> builder = "hvm"
>>> maxmem = 512
>>> memory = 512
>>> vcpus = 1
>>> on_poweroff = "destroy"
>>> on_reboot = "destroy"
>>> on_crash = "destroy"
>>> bios = "ovmf"
>>> device_model_version = "qemu-xen"
>>> bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"
>> and running it with `xl create vm.cfg`, I see it crash while booting 
>> with the following displayed by `xl dmesg`:
>>
>>> (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff 
>>> ff ff 10 00 b8 40 06 00 00 0f 22
>>> (XEN) d1v0 Triple fault - invoking HVM shutdown action 1
>> I've run the hypervisor with `guest_loglvl=all` for more output and 
>> attached it here and uploaded it at 
>> https://gist.github.com/moll/a46dffc7466ced93a0365a6916a4db96 in case 
>> the file doesn't go through.
> Looks to be an ordinary 32-bit far branch after having switched to
> protected mode. I'm afraid without seeing the involved GDT entry
> there's little chance of guessing what may go wrong in this case.
> One question is why the emulator is being invoked in the first place:
> Since you've truncated the log at the beginning, it's impossible to
> tell whether you're using old Intel hardware lacking the Unrestricted
> Guest feature.

(As included above), This is on Arch Linux, AMD Ryzen on a X370 motherboard.

I can't work out why we hitting the MMIO path in this case,
independently of why the emulation of this instruction failed.

~Andrew

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Anthony PERARD]

On Wed, Aug 16, 2017 at 06:47:23PM +, Andri Möll wrote:
> Hey,
> 
> As per Andrew [Cooper]'s suggestion, writing here instead of #xen on
> Freenode.
> 
> I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having it
> crash right on boot both with the 32b and 64b OVMF binaries. This is on Arch
> Linux, AMD Ryzen on a X370 motherboard.
> 
> Given the following minimal VM declaration:
> > builder = "hvm"
> > maxmem = 512
> > memory = 512
> > vcpus = 1
> > on_poweroff = "destroy"
> > on_reboot = "destroy"
> > on_crash = "destroy"
> > bios = "ovmf"
> > device_model_version = "qemu-xen"
> > bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"
> and running it with `xl create vm.cfg`, I see it crash while booting with
> the following displayed by `xl dmesg`:

I don't know if it is possible to use ovmf_code_ia32.bin (or _x64) with
Xen. You would need to supply ovmf_vars_ia32.bin somehow, but I never
looked at doing that.

Your best bet would be to rebuild the Xen pkg and edit the PKGBUILD to
remove "--with-system-ovmf" configure option (if your compile Xen from
AUR)
OR
Recompile OVMF from https://xenbits.xen.org/git-http/ovmf.git (or the
upstream URL) with `OvmfPkg/build.sh -n $nb_of_cpu -b RELEASE` and use
the OVMF file from Build/OvmfX64/RELEASE_GCC*/FV/OVMF.fd

-- 
Anthony PERARD

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Jan Beulich]

>>> On 16.08.17 at 20:47,  wrote:
> Hey,
> 
> As per Andrew [Cooper]'s suggestion, writing here instead of #xen on 
> Freenode.
> 
> I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having it 
> crash right on boot both with the 32b and 64b OVMF binaries. This is on 
> Arch Linux, AMD Ryzen on a X370 motherboard.
> 
> Given the following minimal VM declaration:
>> builder = "hvm"
>> maxmem = 512
>> memory = 512
>> vcpus = 1
>> on_poweroff = "destroy"
>> on_reboot = "destroy"
>> on_crash = "destroy"
>> bios = "ovmf"
>> device_model_version = "qemu-xen"
>> bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"
> and running it with `xl create vm.cfg`, I see it crash while booting 
> with the following displayed by `xl dmesg`:
> 
>> (XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff 
>> ff ff 10 00 b8 40 06 00 00 0f 22
>> (XEN) d1v0 Triple fault - invoking HVM shutdown action 1
> I've run the hypervisor with `guest_loglvl=all` for more output and 
> attached it here and uploaded it at 
> https://gist.github.com/moll/a46dffc7466ced93a0365a6916a4db96 in case 
> the file doesn't go through.

Looks to be an ordinary 32-bit far branch after having switched to
protected mode. I'm afraid without seeing the involved GDT entry
there's little chance of guessing what may go wrong in this case.
One question is why the emulator is being invoked in the first place:
Since you've truncated the log at the beginning, it's impossible to
tell whether you're using old Intel hardware lacking the Unrestricted
Guest feature.

Jan


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] "MMIO emulation failed" from booting OVMF on Xen v4.9.0

[Andri Möll]

Hey,

As per Andrew [Cooper]'s suggestion, writing here instead of #xen on 
Freenode.


I'm trying out Xen (4.9.0) with OVMF (r21243.3858b4a1ff-1) and having it 
crash right on boot both with the 32b and 64b OVMF binaries. This is on 
Arch Linux, AMD Ryzen on a X370 motherboard.


Given the following minimal VM declaration:

builder = "hvm"
maxmem = 512
memory = 512
vcpus = 1
on_poweroff = "destroy"
on_reboot = "destroy"
on_crash = "destroy"
bios = "ovmf"
device_model_version = "qemu-xen"
bios_path_override = "/usr/share/ovmf/ovmf_code_ia32.bin"
and running it with `xl create vm.cfg`, I see it crash while booting 
with the following displayed by `xl dmesg`:


(XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff 
ff ff 10 00 b8 40 06 00 00 0f 22

(XEN) d1v0 Triple fault - invoking HVM shutdown action 1
I've run the hypervisor with `guest_loglvl=all` for more output and 
attached it here and uploaded it at 
https://gist.github.com/moll/a46dffc7466ced93a0365a6916a4db96 in case 
the file doesn't go through.


Any ideas anyone? Thanks in advance!

Andri
(XEN) HVM1 save: CPU
(XEN) HVM1 save: PIC
(XEN) HVM1 save: IOAPIC
(XEN) HVM1 save: LAPIC
(XEN) HVM1 save: LAPIC_REGS
(XEN) HVM1 save: PCI_IRQ
(XEN) HVM1 save: ISA_IRQ
(XEN) HVM1 save: PCI_LINK
(XEN) HVM1 save: PIT
(XEN) HVM1 save: RTC
(XEN) HVM1 save: HPET
(XEN) HVM1 save: PMTIMER
(XEN) HVM1 save: MTRR
(XEN) HVM1 save: VIRIDIAN_DOMAIN
(XEN) HVM1 save: CPU_XSAVE
(XEN) HVM1 save: VIRIDIAN_VCPU
(XEN) HVM1 save: VMCE_VCPU
(XEN) HVM1 save: TSC_ADJUST
(XEN) HVM1 save: CPU_MSR
(XEN) HVM1 restore: CPU 0
(d1) HVM Loader
(d1) Detected Xen v4.9.0
(d1) Xenbus rings @0xfeffc000, event channel 1
(d1) System requested OVMF
(d1) CPU speed is 3001 MHz
(d1) Relocating guest memory for lowmem MMIO space disabled
(d1) PCI-ISA link 0 routed to IRQ5
(d1) PCI-ISA link 1 routed to IRQ10
(d1) PCI-ISA link 2 routed to IRQ11
(d1) PCI-ISA link 3 routed to IRQ5
(d1) pci dev 01:3 INTA->IRQ10
(d1) pci dev 02:0 INTA->IRQ11
(d1) No RAM in high memory; setting high_mem resource base to 1
(d1) pci dev 03:0 bar 10 size 00200: 0f008
(d1) pci dev 02:0 bar 14 size 00100: 0f208
(d1) pci dev 03:0 bar 30 size 1: 0f300
(d1) pci dev 03:0 bar 14 size 01000: 0f301
(d1) pci dev 02:0 bar 10 size 00100: 0c001
(d1) pci dev 01:1 bar 20 size 00010: 0c101
(d1) Multiprocessor initialisation:
(d1)  - CPU0 ... 48-bit phys ... fixed MTRRs ... var MTRRs [1/8] ... done.
(d1) Writing SMBIOS tables ...
(d1) Loading OVMF ...
(XEN) d1v0 Over-allocation for domain 1: 131329 > 131328
(d1) Loading ACPI ...
(d1) CONV disabled
(d1) vm86 TSS at fc00a400
(d1) BIOS map:
(d1)  ffe0-fffd: Main BIOS
(d1) E820 table:
(d1)  [00]: : - :000a: RAM
(d1)  HOLE: :000a - :000f
(d1)  [01]: :000f - :0010: RESERVED
(d1)  [02]: :0010 - :1f715000: RAM
(d1)  HOLE: :1f715000 - :fc00
(d1)  [03]: :fc00 - 0001:: RESERVED
(d1) Invoking OVMF ...
(XEN) MMIO emulation failed: d1v0 16bit @ f000:ff54 -> 66 ea 5c ff ff ff 10 
00 b8 40 06 00 00 0f 22
(XEN) d1v0 Triple fault - invoking HVM shutdown action 1
(XEN) *** Dumping Dom1 vcpu#0 state: ***
(XEN) [ Xen-4.9.0  x86_64  debug=n   Not tainted ]
(XEN) CPU:6
(XEN) RIP:f000:[]
(XEN) RFLAGS: 0046   CONTEXT: hvm guest (d1v0)
(XEN) rax: 4023   rbx: ff74   rcx: 
(XEN) rdx:    rsi:    rdi: 5042
(XEN) rbp:    rsp:    r8:  
(XEN) r9:     r10:    r11: 
(XEN) r12:    r13:    r14: 
(XEN) r15:    cr0: 4033   cr4: 
(XEN) cr3:    cr2: 
(XEN) ds: f000   es:    fs:    gs:    ss:    cs: f000___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel