Hi,
I'm currently trying to pull together the threads for booting Xen on Debian. I'm currently doing this within QEMU's TCG emulation and the "virt" machine model: -machine type=virt,virtualization=on,gic-version=3 \ -cpu max,pauth-impdef=on with the firmware on my Ubuntu machine: -drive if=pflash,file=/usr/share/AAVMF/AAVMF_CODE.fd,format=raw,readonly=on -drive if=pflash,file=$HOME/images/AAVMF_VARS.fd,format=raw (qemu-efi-aarch64 Version: 0~20180205.c0d9813c-2ubuntu0.3) When booting this way I get the Grub menu and Xen is loaded by Grub but falls over later: (XEN) MODULE[0]: 00000000f5869000 - 00000000f59b60c8 Xen (XEN) MODULE[1]: 000000013857d000 - 0000000138580000 Device Tree (XEN) MODULE[2]: 00000000f73a1000 - 00000000f8da0780 Kernel (XEN) MODULE[3]: 00000000f59b7000 - 00000000f739f99b Ramdisk (XEN) (XEN) CMDLINE[00000000f73a1000]:chosen placeholder root=UUID=435201aa-c5cf-4e7a-8107-5eef28844188 ro console=hvc0 (XEN) (XEN) Command line: placeholder dom0_mem=2G loglvl=all guest_loglvl=all no-real-mode edd=off (XEN) parameter "placeholder" unknown! (XEN) parameter "no-real-mode" unknown! (XEN) parameter "edd" unknown! (XEN) Domain heap initialised (XEN) Booting using Device Tree (XEN) Platform: Generic System (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) Unable to find a compatible timer in the device tree (XEN) **************************************** It seems like there are bits of the DT missing. I can however successfully boot Xen with the Linux guest using the guest-loader device and bypassing the firmware/boot code step. This gives: (XEN) MODULE[0]: 0000000040200000 - 000000004034d0c8 Xen (XEN) MODULE[1]: 0000000048000000 - 0000000048100000 Device Tree (XEN) MODULE[2]: 0000000046000000 - 0000000046eb2200 Kernel (XEN) (XEN) CMDLINE[0000000046000000]:chosen root=/dev/sda2 console=hvc0 earlyprintk=xen (XEN) (XEN) Command line: dom0_mem=4G dom0_max_vcpus=4 (XEN) Domain heap initialised (XEN) Booting using Device Tree (XEN) Platform: Generic System (XEN) Taking dtuart configuration from /chosen/stdout-path (XEN) Looking for dtuart at "/pl011@9000000", options "" Xen 4.15.1-pre (XEN) Xen version 4.15.1-pre (alex.bennee@) (aarch64-linux-gnu-gcc (Debian 8.3.0-2) 8.3.0) debug=y Tue May 18 09:34:55 UTC 2021 (XEN) Latest ChangeSet: (XEN) build-id: a50d8f03a1a15662ac7c4e5f73f2f544a6739df2 (XEN) Processor: 411fd070: "ARM Limited", variant: 0x1, part 0xd07, rev 0x0 (XEN) 64-bit Execution: (XEN) Processor Features: 0000000001000222 0000000000000000 (XEN) Exception Levels: EL3:No EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg (XEN) Debug Features: 0000000010305106 0000000000000000 (XEN) Auxiliary Features: 0000000000000000 0000000000000000 (XEN) Memory Model Features: 0000000000001124 0000000000000000 (XEN) ISA Features: 0000000000011120 0000000000000000 (XEN) 32-bit Execution: (XEN) Processor Features: 00000131:10011001 (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle (XEN) Extensions: GenericTimer (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: 00000000 (XEN) Memory Model Features: 10101105 40000000 01260000 02102211 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 (XEN) Using SMC Calling Convention v1.0 (XEN) Using PSCI v0.2 (XEN) SMP: Allowing 8 CPUs (XEN) enabled workaround for: ARM erratum 832075 (XEN) enabled workaround for: ARM erratum 834220 (XEN) enabled workaround for: ARM erratum 1319367 (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 62500 KHz (XEN) GICv3 initialization: (XEN) gic_dist_addr=0x00000008000000 (XEN) gic_maintenance_irq=25 (XEN) gic_rdist_stride=0 (XEN) gic_rdist_regions=1 (XEN) redistributor regions: (XEN) - region 0: 0x000000080a0000 - 0x00000009000000 (XEN) GICv3: 256 lines, (IID 0000043b). (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000 Attempting to boot with acpi=on still sees Grub attempt to use DT to boot the hypervisor. However selecting the kernel directly boots with ACPI (which is a shame as I'd like to see what FDT it gets presented with). The full command line for booting via the guest-loader is: ./qemu-system-aarch64 \ -machine virt,virtualization=on,gic-version=3 \ -cpu max,pauth-impdef=on \ -serial mon:stdio \ -netdev user,id=net1,hostfwd=tcp::2222-:22 \ -device virtio-net-pci,netdev=net1 \ -device virtio-scsi-pci \ -drive file=/dev/zvol/hackpool-0/debian-buster-arm64,id=hd0,index=0,if=none,format=raw \ -device scsi-hd,drive=hd0 \ -display none \ -m 16384 \ -kernel ~/lsrc/xen/xen.build.arm64-xen/xen/xen \ -append "dom0_mem=4G dom0_max_vcpus=4" \ -device guest-loader,addr=0x46000000,kernel=$HOME/lsrc/linux.git/builds/arm64/arch/arm64/boot/Image,bootargs="root=/dev/sda2 console=hvc0 earlyprintk=xen" \ -smp 8 So some questions: - is Xen on arm64 tested on Debian Bullseye? If so what platform? - how do I tell Grub to do a straight FDT boot with the DT from the firmware? - are there any missing pieces I should be aware of? I appreciate that ACPI is the preferred enterprise way of booting but at the moment I think FDT is probably preferred because: - lack of real HW with decent ACPI (my MachiatoBin only boots with DT) - I want to try additional hypervisors who don't have ACPI aware implementations That said if I can get an ACPI version of Xen booting via Grub that would be an improvement. -- Alex Bennée