Hi Bertrand, I did this yesterday and it does indeed silence the warnings saying to disable those cores, however the IRQ errors still remain.
I think I made some progress by changing "interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>;" to "interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH 0>;". I noticed when trying to use XEN-4.10 that it said "IRQ only through SPI" and failed to boot XEN where 4.17 unstable boots without this error. Now that I changed GIC_PPI to GIC_SPI, I get the following output: zaku => run boot_pci_enum; run nvme_init; fatload nvme 0:1 0x02000000 xen4.10-uImage; fatload nvme 0:1 0x01f00000 rk3399-zaku.dtb; fatload nvme 0:1 0x03F80000 Image;fatload nvme 0:1 0 885136 bytes read in 5 ms (168.8 MiB/s) 56185 bytes read in 3 ms (17.9 MiB/s) 27885576 bytes read in 65 ms (409.1 MiB/s) 8674182 bytes read in 22 ms (376 MiB/s) ## Booting kernel from Legacy Image at 02000000 ... Image Name: Image Type: AArch64 Linux Kernel Image (uncompressed) Data Size: 885072 Bytes = 864.3 KiB Load Address: 02000000 Entry Point: 02000000 Verifying Checksum ... OK ## Flattened Device Tree blob at 01f00000 Booting using the fdt blob at 0x1f00000 Loading Kernel Image Loading Device Tree to 00000000f1efd000, end 00000000f1f0db78 ... OK Starting kernel ... - UART enabled - - CPU 00000000 booting - - Current EL 00000008 - - Xen starting at EL2 - - Zero BSS - - Setting up control registers - - Turning on paging - - Ready - (XEN) Checking for initrd in /chosen (XEN) RAM: 0000000000200000 - 00000000f7ffffff (XEN) (XEN) MODULE[0]: 00000000f1efd000 - 00000000f1f0b000 Device Tree (XEN) MODULE[1]: 0000000003f80000 - 0000000005a28008 Kernel (XEN) MODULE[2]: 0000000006000000 - 0000000008000000 Ramdisk (XEN) (XEN) Command line: <NULL> (XEN) Placing Xen at 0x00000000f7e00000-0x00000000f8000000 (XEN) Update BOOTMOD_XEN from 0000000002000000-0000000002120d81 => 00000000f7e00000-00000000f7f20d81 (XEN) Domain heap initialised (XEN) Booting using Device Tree (XEN) Platform: Generic System (XEN) Taking dtuart configuration from /chosen/stdout-path (XEN) Looking for dtuart at "serial2", options "1500000n8" Xen 4.10.1 (XEN) Xen version 4.10.1 (root@) (aarch64-linux-gnu-gcc (Linaro GCC 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701]) debug=y Tue J2 (XEN) Latest ChangeSet: (XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4 (XEN) 64-bit Execution: (XEN) Processor Features: 0000000001002222 0000000000000000 (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 (XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg (XEN) Debug Features: 0000000010305106 0000000000000000 (XEN) Auxiliary Features: 0000000000000000 0000000000000000 (XEN) Memory Model Features: 0000000000001122 0000000000000000 (XEN) ISA Features: 0000000000011120 0000000000000000 (XEN) 32-bit Execution: (XEN) Processor Features: 00000131:10011011 (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle (XEN) Extensions: GenericTimer Security (XEN) Debug Features: 03010066 (XEN) Auxiliary Features: 00000000 (XEN) Memory Model Features: 10201105 40000000 01260000 02102211 (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 (XEN) Using SMC Calling Convention v1.0 (XEN) Using PSCI v1.0 (XEN) SMP: Allowing 4 CPUs (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 24000 KHz (XEN) GICv3 initialization: (XEN) gic_dist_addr=0x000000fee00000 (XEN) gic_maintenance_irq=41 (XEN) gic_rdist_stride=0 (XEN) gic_rdist_regions=1 (XEN) redistributor regions: (XEN) - region 0: 0x000000fef00000 - 0x000000fefc0000 (XEN) GICv3 compatible with GICv2 cbase 0x000000fff00000 vbase 0x000000fff20000 (XEN) GICv3: 288 lines, (IID 0001143b). (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000 (XEN) Using scheduler: SMP Credit Scheduler (credit) (XEN) Allocated console ring of 32 KiB. (XEN) Bringing up CPU1 - CPU 00000001 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) GICv3: CPU1: Found redistributor in region 0 @000000004003c000 (XEN) CPU 1 booted. (XEN) Bringing up CPU2 - CPU 00000002 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) GICv3: CPU2: Found redistributor in region 0 @000000004005c000 (XEN) CPU 2 booted. (XEN) Bringing up CPU3 - CPU 00000003 booting - - Current EL 00000008 - - Xen starting at EL2 - - Setting up control registers - - Turning on paging - - Ready - (XEN) GICv3: CPU3: Found redistributor in region 0 @000000004007c000 (XEN) CPU 3 booted. (XEN) Brought up 4 CPUs (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 (XEN) I/O virtualisation disabled (XEN) build-id: b7279757e82e3e9ea4c4115dabdbfe25d56d762b (XEN) alternatives: Patching with alt table 000000004018fe70 -> 0000000040190350 (XEN) grant_table.c:1731:IDLEv0 Expanding d0 grant table from 0 to 1 frames (XEN) *** LOADING DOMAIN 0 *** (XEN) Loading kernel from boot module @ 0000000003f80000 (XEN) Loading ramdisk from boot module @ 0000000006000000 (XEN) Allocating 1:1 mappings totalling 512MB for dom0: (XEN) BANK[0] 0x00000020000000-0x00000040000000 (512MB) (XEN) Grant table range: 0x000000f7e00000-0x000000f7e40000 (XEN) IRQ 41 is already used by Xen (XEN) Unable to map IRQ41 to dom0 (XEN) Device tree generation failed (-16). (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) Could not set up DOM0 guest OS (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... It looks like XEN is now using the interrupt controller but there are conflicts. I was able to get this error to change to "Unable to map IRQ43 to dom0" by changing all instances of "IRQ_TYPE_LEVEL_HIGH " to "IRQ_TYPE_LEVEL_LOW", but I can't get any further. I made sure that there was nothing in the device tree that is obviously using IRQ 43 as well. Any thoughts on this? Cheers, Brad On Wed, Jul 13, 2022, 8:00 AM Bertrand Marquis <[email protected]> wrote: > Hi Brad > > > On 12 Jul 2022, at 17:24, Brad Churchwell <[email protected]> wrote: > > > > Hi Bertrand, > > > > I believe I understand, but just to clarify, should I leave the > ppi-partitions block in rk3399.dtsi as is and disable the little cores, or > should I also modify that block? > > Disable the little cores and remove the partitions. > > Cheers > Bertrand > > > > > Brad > > > > On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis < > [email protected]> wrote: > > Hi Brad, > > > > > On 12 Jul 2022, at 16:59, Brad Churchwell <[email protected]> wrote: > > > > > > Hi Bertrand, > > > > > > Thanks so much for the quick response! > > > > > > I should have mentioned previously that this device tree and kernel > Image (5.15.16) does boot properly with the rootfs without XEN. The > interrupt errors are only present when booting with XEN. > > > > > > These are custom boards and they do have usb c, however we are unable > to boot from usb as it doesn't receive power. We currently only use the > usb to flash u-boot to spi which requires us to use a separate power > adapter. These boards get power and networking through a custom backplane > and interface with the backplane via a sata style justice (I'm not sure > what the proper terminology would be here). > > > > > > Since I cannot boot to rootfs with XEN, I'm unable to show the device > tree there. The only bit that I added to accommodate XEN is this bit here: > > > > > > chosen { > > > stdout-path = "serial2:1500000n8"; > > > bootargs = "hmp-unsafe=true"; > > > xen,dom0-bootargs = "console=hvc0 earlyprintk=xen > clk_ignore_unused root=/dev/nvme0n1p2 rw init=/sbin/init rootwait > rootfstype=ext4 ignore_loglevel"; > > > modules { > > > #address-cells = <2>; > > > #size-cells = <2>; > > > > > > module@1 { > > > compatible = "xen,linux-zimage", "xen,multiboot-module"; > > > reg = <0x0 0x03F80000 0x0 0x01aa8008>; > > > }; > > > module@2 { > > > compatible = "xen,linux-initrd", "xen,multiboot-module"; > > > reg = <0x0 0x06000000 0x0 0x02000000>; > > > }; > > > }; > > > > > > }; > > > > > > The gic declaration is in the vanilla rk3399.dtsi file and hasn't been > modified: > > > > > > gic: interrupt-controller@fee00000 { > > > compatible = "arm,gic-v3"; > > > #interrupt-cells = <4>; > > > #address-cells = <2>; > > > #size-cells = <2>; > > > ranges; > > > interrupt-controller; > > > > > > reg = <0x0 0xfee00000 0 0x10000>, /* GICD */ > > > <0x0 0xfef00000 0 0xc0000>, /* GICR */ > > > <0x0 0xfff00000 0 0x10000>, /* GICC */ > > > <0x0 0xfff10000 0 0x10000>, /* GICH */ > > > <0x0 0xfff20000 0 0x10000>; /* GICV */ > > > interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>; > > > its: interrupt-controller@fee20000 { > > > compatible = "arm,gic-v3-its"; > > > msi-controller; > > > #msi-cells = <1>; > > > reg = <0x0 0xfee20000 0x0 0x20000>; > > > }; > > > > > > ppi-partitions { > > > ppi_cluster0: interrupt-partition-0 { > > > affinity = <&cpu_l0 &cpu_l1 &cpu_l2 > &cpu_l3>; > > > }; > > > > > > ppi_cluster1: interrupt-partition-1 { > > > affinity = <&cpu_b0 &cpu_b1>; > > > }; > > > }; > > > > Xen is not supporting PPI partitions. > > I also saw that your booting using SMP unsafe so all core are enabled by > Xen. > > > > Right now big/little is not supported by Xen and here you might have > some interrupts in your system which can only occur on some specific cores. > > I would suggest to try the following: > > - only boot enable in your dtb the cores with the same type than the > boot core > > - check in your device tree which devices are assign to the ppi_cluster > not available and disable them > > > > This kind of configuration will be tricky to handle so it will depend on > which devices have interrupts that can only be fired on one cluster type. > > > > Sorry but I cannot right now check in the device tree and do those > checks myself but if you do get the idea send back a mail :-) > > > > Cheers > > Bertrand > > > > > }; > > > > > > I've attached the device tree and include files just in case you'd > like to take a look at those. > > > > > > Thanks so much for your help! I've been at this for weeks with very > little progress. > > > > > > Cheers, > > > Brad > > > > > > > > > On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis < > [email protected]> wrote: > > > Hi Brad, > > > > > > > On 11 Jul 2022, at 19:38, Brad Churchwell <[email protected]> wrote: > > > > > > > > Hello, > > > > > > > > I've been trying to get Xen to boot dom0 with my kernel for weeks on > an rk3399 based board and thought I'd reach out for help. It looks like > either Xen is not properly recreating the device tree or the interrupt > controller is just failing. The hypervisor boots but falls to initramfs > because it cannot find the root device (nvme on pcie). Any help would be > greatly appreciated. Here is the complete boot log > > > > > > From the logs you have an issue with the interrupt controller and the > fact that your NVME is behind PCIE and as such depends on ITS is probably > not helping. > > > > > > I would suggest to try to boot on usb as root fs for a try. > > > > > > Also it could be useful to compare the device tree on xen and without > xen to understand what is going on (using /proc/device-tree). > > > > > > Xen seems to be ok but Linux is not happy with interrupts and is > showing several issues around this area on your logs. > > > Could you show us an extract of your device tree around the gic > declaration ? > > > > > > Cheers > > > Bertrand > > > > > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > > <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi> > > > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. >
