On 02/09/2025 1:24 pm, Manuel Bouyer wrote: > On Tue, Sep 02, 2025 at 01:14:29PM +0100, Andrew Cooper wrote: >> On 02/09/2025 12:23 pm, Manuel Bouyer wrote: >>> On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote: >>>> On 02/09/2025 11:56 am, Manuel Bouyer wrote: >>>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote: >>>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote: >>>>>>> Hello, >>>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20. >>>>>>> The same NetBSD kernel works fine with Xen 4.18 >>>>>>> >>>>>>> The boot options are: >>>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH >>>>>>> console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M >>>>>>> console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all >>>>>>> gnttab_max_nr_frames=64 sync_console=1 dom0=pvh >>>>>>> >>>>>>> and the full log from serial console is attached. >>>>>>> >>>>>>> With 4.20 the boot fails with: >>>>>>> >>>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch >>>>>>> input) >>>>>>> (XEN) Freed 664kB init memory >>>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1 >>>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: *** >>>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C >>>>>>> ]---- >>>>>>> (XEN) CPU: 7 >>>>>>> (XEN) RIP: 0008:[<000000000020e268>] >>>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0) >>>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: >>>>>>> 00000000000dfeb7 >>>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: >>>>>>> 000000000013e000 >>>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: >>>>>>> 0000000000000000 >>>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: >>>>>>> 0000000000000000 >>>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: >>>>>>> 0000000000000000 >>>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: >>>>>>> 0000000000000000 >>>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000 >>>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: >>>>>>> 0000000000000000 >>>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008 >>>>>>> >>>>>>> because of the triple fault the RIP above doens't point to the code. >>>>>>> >>>>>>> I tracked it down to this code: >>>>>>> cmpl $0,%ecx ; /* zero-sized? */ >>>>>>> \ >>>>>>> je 2f ; \ >>>>>>> pushl %ebp ; \ >>>>>>> movl RELOC(nox_flag),%ebp ; \ >>>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ >>>>>>> \ >>>>>>> movl %eax,(%ebx) ; /* store phys addr */ >>>>>>> \ >>>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ >>>>>>> \ >>>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ >>>>>>> \ >>>>>>> loop 1b ; \ >>>>>>> popl %ebp ; \ >>>>>>> 2: ; >>>>>>> >>>>>>> there are others pushl/popl before so I don't think that's the problem >>>>>>> (in fact the exact same fragment is called just before with different >>>>>>> inputs and it doesn't fault). So the culprit it probably the write to >>>>>>> (%ebx), >>>>>>> which would be 0x20e260 >>>>>>> This is in the range: >>>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable) >>>>>>> so I can't see why this would be a problem. >>>>>>> >>>>>>> Any idea, including how to debug this further, welcome >>>>>> Even though triple fault's are aborts, they're generally accurate under >>>>>> virt, so 0x20e268 is most likely where things die. >>>>> but that's the RIP of the last fault, not the first one, right ? >>>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the >>>>> first fault triggers an exception, but the exeption handler isn't set up >>>>> yet >>>>> so we end up jumping to some random value. >>>> Double and Triple faults occur when trying to deliver an exception >>>> generates an exception. So while multiple faults are involved, only one >>>> instruction typically is. >>>> >>>> Is this an Intel or an AMD system? One thing virt can do is break apart >>>> a triple fault, but the logic to do so is vendor specific. >>> it's an old intel system: >>> cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz" >>> cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz >>> cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2) >>> >> Hmm. Westmere EP. Are you running with EPT active, or with Shadow Paging? > How do I know ? Something like:
HVM: Hardware Assisted Paging (HAP) detected on boot. > Note that the same problem shows up on much newer systems: an i9, and a > Xeon W-2223. Both boots fine with the same NetBSD kernel and Xen 4.18 or 4.15. > > I'm using this old Xeon for debug because this one has a serial console. Sure. It's just that HAP vs Shadow is also relevant to breaking apart a triple fault. ~Andrew
