Hello lists, I’m currently playing around with getting rump kernels built with rumpkernel(7) running on FreeBSD’s bhyve(4). I’m using a custom boot loader [1] which builds on some patches to bhyveload / user boot [2].
To test, I’m using a simple “helloer” unikernel from the tutorial [3]: #include <stdio.h> #include <unistd.h> int main() { printf("Hello, Rumprun ... I'm feeling tired\n"); sleep(2); printf("much better!\n"); return 0; } I’m compiling this for the hw_virtio platform: x86_64-rumprun-netbsd-gcc -o helloer-rumprun helloer.c rumprun-bake hw_virtio helloer-rumprun.elf helloer-rumprun Starting this kernel in bhyve gives me the following VM Exit: From the Intel SDM 3D, Appendix C [4], VM Exit 2 is a triple fault. To further debug this, I’m using the ktr(4) support introduced in FreeBSD r276098 [5]. I’ve appended a full KTR trace further down in my mail, but I’ll go along the trace here with some commentary in execution order, and attempt to annotate the source code lines from rumpkernel: 0x104000 is the entry address given in rump kernel’s multi boot header. 0000000000104000 <_start>: 632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000 636 0 350886961656 vm testing[0]: Resume execution at 0x104000 This is the cpuid instruction in x86_cpuid which is called from hypervisor_detect: rumprun/platform/hw/arch/x86/x86_subr.c: 91 0000000000102f90 <x86_cpuid>: 102f90: 49 89 d2 mov r10,rdx 102f93: 49 89 c9 mov r9,rcx 102f96: 53 push rbx 102f97: 89 f8 mov eax,edi 102f99: 0f a2 cpuid rumprun/platform/hw/arch/x86/hypervisor.c: 38 637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8 638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f99 639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b rumprun/platform/hw/arch/x86/hypervisor.c: 40 640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8 641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f99 642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b rumprun/platform/hw/arch/x86/hypervisor.c: 51 643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8 644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f99 645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b The next vexit is in cons_init, directly after the call to hypervisor_detect. 0000000000102a50 <cons_init>: 102a50: 53 push rbx 102a51: e8 ea 0a 00 00 call 103540 <hypervisor_detect> 102a56: 66 83 3d a4 d5 ef ff cmp WORD PTR [rip-0x102a5c],0x0 # 2 <current_lwp+0x2> Due to compiler optimisations, the check here isn’t the (hypervisor == HYPERVISOR_XEN) check directly after the call to hypervisor_detect, but the check (bios_crtc_base == 0) in rumprun/platform/hw/arch/x86/cons.c:59: 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56 Therefore, I’m assuming this is the origin of the fault. Tracking down bios_crtc_base, I find that it’s loaded in rumprun/platform/hw/arch/amd64/locore.S:70: /* save BIOS data area values */ movw BIOS_COM1_BASE, %bx movw %bx, bios_com1_base movw BIOS_CRTC_BASE, %bx movw %bx, bios_crtc_base Where BIOS_CRTC_BASE is 0x463 and BIOS_COM1_BASE is 0x400. Checking the bhyve device node in /dev/vmm with xxd(1), I find the words at these addresses to be Uninitialised: 00000400: 0000 .. 00000483: 0000 .. I’m not sure where to go from here. Is this a bug in bhyve(4), should these values be initialised somehow, or should I patch rumpkernel(7) to skip this check when running on bhyve(4)? Fabian Full KTR trace: index cpu timestamp trace ------ --- ---------------- ----- 673 0 350887364560 vm testing[0]: vcpu state changed from frozen to idle 672 0 350887363766 vm testing[0]: retu 0/14 671 0 350887362932 vm testing[0]: All vcpus suspended 670 0 350887357212 vm testing[0]: vcpu state changed from running to frozen 669 0 350887350444 vm testing[0]: returning from vmx_run: exitcode 14 668 0 350887348806 vm testing: virtual machine successfully suspended 4 667 0 350887347712 vm testing[0]: triple fault: info1(0x80000b08), info2(0x80000b0e) 666 0 350887347590 vm testing[0]: Exception 14 delivered: 0x80000b0e 665 0 350887346312 vm testing[0]: handled exception vmexit at 0x102a56 664 0 350887346178 vm testing[0]: Exception 14 pending 663 0 350887346060 vm testing[0]: Setting intr_shadow to 0 succeeded 662 0 350887341876 vm testing[0]: Reflecting exception 14/0 into the guest 661 0 350887339570 vm testing[0]: vm_exit_intinfo: info1(0x80000b08) 660 0 350887315326 vm testing[0]: Resume execution at 0x102a56 659 0 350887311168 vm testing[0]: vm_entry_intinfo: info1(0x80000b0e), info2(0x80000b0e), retinfo(0x80000b08) 658 0 350887310988 vm testing[0]: Exception 14 delivered: 0x80000b0e 657 0 350887309700 vm testing[0]: handled exception vmexit at 0x102a56 656 0 350887309570 vm testing[0]: Exception 14 pending 655 0 350887309442 vm testing[0]: Setting intr_shadow to 0 succeeded 654 0 350887305126 vm testing[0]: Reflecting exception 14/0 into the guest 653 0 350887302436 vm testing[0]: vm_exit_intinfo: info1(0x80000b0e) 652 0 350887248280 vm testing[0]: Resume execution at 0x102a56 651 0 350887184160 vm testing[0]: vm_entry_intinfo: info1(0), info2(0x80000b0e), retinfo(0x80000b0e) 650 0 350887184040 vm testing[0]: Exception 14 delivered: 0x80000b0e 649 0 350887182668 vm testing[0]: handled exception vmexit at 0x102a56 648 0 350887182374 vm testing[0]: Exception 14 pending 647 0 350887182240 vm testing[0]: Setting intr_shadow to 0 succeeded 646 0 350887176682 vm testing[0]: Reflecting exception 14/0 into the guest 645 0 350887149146 vm testing[0]: Resume execution at 0x102f9b 644 0 350887145218 vm testing[0]: handled cpuid vmexit at 0x102f99 643 0 350887143374 vm testing[0]: cpuid 0x40000000,0x100fd8 642 0 350887121556 vm testing[0]: Resume execution at 0x102f9b 641 0 350887117424 vm testing[0]: handled cpuid vmexit at 0x102f99 640 0 350887102692 vm testing[0]: cpuid 0x1,0x100fd8 639 0 350887079146 vm testing[0]: Resume execution at 0x102f9b 638 0 350887074874 vm testing[0]: handled cpuid vmexit at 0x102f99 637 0 350887061536 vm testing[0]: cpuid 0,0x100fd8 636 0 350886961656 vm testing[0]: Resume execution at 0x104000 635 0 350886900428 vm testing[0]: vcpu state changed from frozen to running 634 0 350886895942 vm testing[0]: vcpu state changed from idle to frozen 633 0 350886886630 vm testing[0]: vcpu state changed from frozen to idle 632 0 350886885990 vm testing[0]: Setting nextrip to 0x104000 631 0 350886802596 vm testing[0]: vcpu state changed from idle to frozen 630 3 350886414966 vm testing[0]: vcpu state changed from frozen to idle 629 3 350886414184 vm testing[0]: activated 628 3 350886412730 vm testing[0]: vcpu state changed from idle to frozen 627 3 350886137831 vm testing[0]: vcpu state changed from frozen to idle 626 3 350885989045 vm testing[0]: vcpu state changed from idle to frozen 625 3 350885773367 vm testing: RTC time set to 0x5a9e154e 624 3 350885773017 vm testing: Updating RTC base uptime from 0x31c2762314 to 0x8bd6b051a2 623 3 350885772543 vm testing: Updating RTC secs from 0x5a9e14f4 to 0x5a9e154e 622 3 350885017699 vm testing: RTC nvram write 0 to offset 0x5d 621 3 350885015953 vm testing: RTC nvram write 0 to offset 0x5c 620 3 350885013951 vm testing: RTC nvram write 0 to offset 0x5b 619 3 350885010615 vm testing: RTC nvram write 0x1 to offset 0x35 618 3 350885008163 vm testing: RTC nvram write 0 to offset 0x34 [1] https://github.com/fubarnetes/bhyve-multiboot [2] https://reviews.FreeBSD.org/D14473 [3] https://github.com/rumpkernel/wiki/wiki/Tutorial%3A-Building-Rumprun-Unikernels#2-building-applications [4] https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3d-part-4-manual.pdf [5] https://svnweb.freebsd.org/base?view=revision&revision=276098
signature.asc
Description: OpenPGP digital signature