I just wanted to add that I realize that it's just remapping the same IRQ. I've tried changing the values to accommodate SPI (+16) vs PPI (+32) which has only resulted in failure.
On Wed, Jul 13, 2022, 8:32 AM Brad Churchwell <[email protected]> wrote: > Hi Bertrand, > > I did this yesterday and it does indeed silence the warnings saying to > disable those cores, however the IRQ errors still remain. > > I think I made some progress by changing "interrupts = <GIC_PPI 9 > IRQ_TYPE_LEVEL_HIGH 0>;" to "interrupts = <GIC_SPI 9 IRQ_TYPE_LEVEL_HIGH > 0>;". I noticed when trying to use XEN-4.10 that it said "IRQ only through > SPI" and failed to boot XEN where 4.17 unstable boots without this error. > Now that I changed GIC_PPI to GIC_SPI, I get the following output: > zaku => run boot_pci_enum; run nvme_init; fatload nvme 0:1 0x02000000 > xen4.10-uImage; fatload nvme 0:1 0x01f00000 rk3399-zaku.dtb; fatload nvme > 0:1 0x03F80000 Image;fatload nvme 0:1 0 > 885136 bytes read in 5 ms (168.8 MiB/s) > 56185 bytes read in 3 ms (17.9 MiB/s) > 27885576 bytes read in 65 ms (409.1 MiB/s) > 8674182 bytes read in 22 ms (376 MiB/s) > ## Booting kernel from Legacy Image at 02000000 ... > Image Name: > Image Type: AArch64 Linux Kernel Image (uncompressed) > Data Size: 885072 Bytes = 864.3 KiB > Load Address: 02000000 > Entry Point: 02000000 > Verifying Checksum ... OK > ## Flattened Device Tree blob at 01f00000 > Booting using the fdt blob at 0x1f00000 > Loading Kernel Image > Loading Device Tree to 00000000f1efd000, end 00000000f1f0db78 ... OK > > Starting kernel ... > > - UART enabled - > - CPU 00000000 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Zero BSS - > - Setting up control registers - > - Turning on paging - > - Ready - > (XEN) Checking for initrd in /chosen > (XEN) RAM: 0000000000200000 - 00000000f7ffffff > (XEN) > (XEN) MODULE[0]: 00000000f1efd000 - 00000000f1f0b000 Device Tree > (XEN) MODULE[1]: 0000000003f80000 - 0000000005a28008 Kernel > (XEN) MODULE[2]: 0000000006000000 - 0000000008000000 Ramdisk > (XEN) > (XEN) Command line: <NULL> > (XEN) Placing Xen at 0x00000000f7e00000-0x00000000f8000000 > (XEN) Update BOOTMOD_XEN from 0000000002000000-0000000002120d81 => > 00000000f7e00000-00000000f7f20d81 > (XEN) Domain heap initialised > (XEN) Booting using Device Tree > (XEN) Platform: Generic System > (XEN) Taking dtuart configuration from /chosen/stdout-path > (XEN) Looking for dtuart at "serial2", options "1500000n8" > Xen 4.10.1 > (XEN) Xen version 4.10.1 (root@) (aarch64-linux-gnu-gcc (Linaro GCC > 7.3-2018.05) 7.3.1 20180425 [linaro-7.3-2018.05 revision > d29120a424ecfbc167ef90065c0eeb7f91977701]) debug=y Tue J2 > (XEN) Latest ChangeSet: > (XEN) Processor: 410fd034: "ARM Limited", variant: 0x0, part 0xd03, rev 0x4 > (XEN) 64-bit Execution: > (XEN) Processor Features: 0000000001002222 0000000000000000 > (XEN) Exception Levels: EL3:64+32 EL2:64+32 EL1:64+32 EL0:64+32 > (XEN) Extensions: FloatingPoint AdvancedSIMD GICv3-SysReg > (XEN) Debug Features: 0000000010305106 0000000000000000 > (XEN) Auxiliary Features: 0000000000000000 0000000000000000 > (XEN) Memory Model Features: 0000000000001122 0000000000000000 > (XEN) ISA Features: 0000000000011120 0000000000000000 > (XEN) 32-bit Execution: > (XEN) Processor Features: 00000131:10011011 > (XEN) Instruction Sets: AArch32 A32 Thumb Thumb-2 Jazelle > (XEN) Extensions: GenericTimer Security > (XEN) Debug Features: 03010066 > (XEN) Auxiliary Features: 00000000 > (XEN) Memory Model Features: 10201105 40000000 01260000 02102211 > (XEN) ISA Features: 02101110 13112111 21232042 01112131 00011142 00011121 > (XEN) Using SMC Calling Convention v1.0 > (XEN) Using PSCI v1.0 > (XEN) SMP: Allowing 4 CPUs > (XEN) Generic Timer IRQ: phys=30 hyp=26 virt=27 Freq: 24000 KHz > (XEN) GICv3 initialization: > (XEN) gic_dist_addr=0x000000fee00000 > (XEN) gic_maintenance_irq=41 > (XEN) gic_rdist_stride=0 > (XEN) gic_rdist_regions=1 > (XEN) redistributor regions: > (XEN) - region 0: 0x000000fef00000 - 0x000000fefc0000 > (XEN) GICv3 compatible with GICv2 cbase 0x000000fff00000 vbase > 0x000000fff20000 > (XEN) GICv3: 288 lines, (IID 0001143b). > (XEN) GICv3: CPU0: Found redistributor in region 0 @000000004001c000 > (XEN) Using scheduler: SMP Credit Scheduler (credit) > (XEN) Allocated console ring of 32 KiB. > (XEN) Bringing up CPU1 > - CPU 00000001 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Setting up control registers - > - Turning on paging - > - Ready - > (XEN) GICv3: CPU1: Found redistributor in region 0 @000000004003c000 > (XEN) CPU 1 booted. > (XEN) Bringing up CPU2 > - CPU 00000002 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Setting up control registers - > - Turning on paging - > - Ready - > (XEN) GICv3: CPU2: Found redistributor in region 0 @000000004005c000 > (XEN) CPU 2 booted. > (XEN) Bringing up CPU3 > - CPU 00000003 booting - > - Current EL 00000008 - > - Xen starting at EL2 - > - Setting up control registers - > - Turning on paging - > - Ready - > (XEN) GICv3: CPU3: Found redistributor in region 0 @000000004007c000 > (XEN) CPU 3 booted. > (XEN) Brought up 4 CPUs > (XEN) P2M: 40-bit IPA with 40-bit PA and 8-bit VMID > (XEN) P2M: 3 levels with order-1 root, VTCR 0x80023558 > (XEN) I/O virtualisation disabled > (XEN) build-id: b7279757e82e3e9ea4c4115dabdbfe25d56d762b > (XEN) alternatives: Patching with alt table 000000004018fe70 -> > 0000000040190350 > (XEN) grant_table.c:1731:IDLEv0 Expanding d0 grant table from 0 to 1 frames > (XEN) *** LOADING DOMAIN 0 *** > (XEN) Loading kernel from boot module @ 0000000003f80000 > (XEN) Loading ramdisk from boot module @ 0000000006000000 > (XEN) Allocating 1:1 mappings totalling 512MB for dom0: > (XEN) BANK[0] 0x00000020000000-0x00000040000000 (512MB) > (XEN) Grant table range: 0x000000f7e00000-0x000000f7e40000 > (XEN) IRQ 41 is already used by Xen > (XEN) Unable to map IRQ41 to dom0 > (XEN) Device tree generation failed (-16). > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Could not set up DOM0 guest OS > (XEN) **************************************** > (XEN) > (XEN) Reboot in five seconds... > > It looks like XEN is now using the interrupt controller but there are > conflicts. I was able to get this error to change to "Unable to map IRQ43 > to dom0" by changing all instances of "IRQ_TYPE_LEVEL_HIGH " to > "IRQ_TYPE_LEVEL_LOW", but I can't get any further. I made sure that there > was nothing in the device tree that is obviously using IRQ 43 as well. > > Any thoughts on this? > > Cheers, > Brad > > > On Wed, Jul 13, 2022, 8:00 AM Bertrand Marquis <[email protected]> > wrote: > >> Hi Brad >> >> > On 12 Jul 2022, at 17:24, Brad Churchwell <[email protected]> wrote: >> > >> > Hi Bertrand, >> > >> > I believe I understand, but just to clarify, should I leave the >> ppi-partitions block in rk3399.dtsi as is and disable the little cores, or >> should I also modify that block? >> >> Disable the little cores and remove the partitions. >> >> Cheers >> Bertrand >> >> > >> > Brad >> > >> > On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis < >> [email protected]> wrote: >> > Hi Brad, >> > >> > > On 12 Jul 2022, at 16:59, Brad Churchwell <[email protected]> wrote: >> > > >> > > Hi Bertrand, >> > > >> > > Thanks so much for the quick response! >> > > >> > > I should have mentioned previously that this device tree and kernel >> Image (5.15.16) does boot properly with the rootfs without XEN. The >> interrupt errors are only present when booting with XEN. >> > > >> > > These are custom boards and they do have usb c, however we are unable >> to boot from usb as it doesn't receive power. We currently only use the >> usb to flash u-boot to spi which requires us to use a separate power >> adapter. These boards get power and networking through a custom backplane >> and interface with the backplane via a sata style justice (I'm not sure >> what the proper terminology would be here). >> > > >> > > Since I cannot boot to rootfs with XEN, I'm unable to show the device >> tree there. The only bit that I added to accommodate XEN is this bit here: >> > > >> > > chosen { >> > > stdout-path = "serial2:1500000n8"; >> > > bootargs = "hmp-unsafe=true"; >> > > xen,dom0-bootargs = "console=hvc0 earlyprintk=xen >> clk_ignore_unused root=/dev/nvme0n1p2 rw init=/sbin/init rootwait >> rootfstype=ext4 ignore_loglevel"; >> > > modules { >> > > #address-cells = <2>; >> > > #size-cells = <2>; >> > > >> > > module@1 { >> > > compatible = "xen,linux-zimage", "xen,multiboot-module"; >> > > reg = <0x0 0x03F80000 0x0 0x01aa8008>; >> > > }; >> > > module@2 { >> > > compatible = "xen,linux-initrd", "xen,multiboot-module"; >> > > reg = <0x0 0x06000000 0x0 0x02000000>; >> > > }; >> > > }; >> > > >> > > }; >> > > >> > > The gic declaration is in the vanilla rk3399.dtsi file and hasn't >> been modified: >> > > >> > > gic: interrupt-controller@fee00000 { >> > > compatible = "arm,gic-v3"; >> > > #interrupt-cells = <4>; >> > > #address-cells = <2>; >> > > #size-cells = <2>; >> > > ranges; >> > > interrupt-controller; >> > > >> > > reg = <0x0 0xfee00000 0 0x10000>, /* GICD */ >> > > <0x0 0xfef00000 0 0xc0000>, /* GICR */ >> > > <0x0 0xfff00000 0 0x10000>, /* GICC */ >> > > <0x0 0xfff10000 0 0x10000>, /* GICH */ >> > > <0x0 0xfff20000 0 0x10000>; /* GICV */ >> > > interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>; >> > > its: interrupt-controller@fee20000 { >> > > compatible = "arm,gic-v3-its"; >> > > msi-controller; >> > > #msi-cells = <1>; >> > > reg = <0x0 0xfee20000 0x0 0x20000>; >> > > }; >> > > >> > > ppi-partitions { >> > > ppi_cluster0: interrupt-partition-0 { >> > > affinity = <&cpu_l0 &cpu_l1 &cpu_l2 >> &cpu_l3>; >> > > }; >> > > >> > > ppi_cluster1: interrupt-partition-1 { >> > > affinity = <&cpu_b0 &cpu_b1>; >> > > }; >> > > }; >> > >> > Xen is not supporting PPI partitions. >> > I also saw that your booting using SMP unsafe so all core are enabled >> by Xen. >> > >> > Right now big/little is not supported by Xen and here you might have >> some interrupts in your system which can only occur on some specific cores. >> > I would suggest to try the following: >> > - only boot enable in your dtb the cores with the same type than the >> boot core >> > - check in your device tree which devices are assign to the ppi_cluster >> not available and disable them >> > >> > This kind of configuration will be tricky to handle so it will depend >> on which devices have interrupts that can only be fired on one cluster type. >> > >> > Sorry but I cannot right now check in the device tree and do those >> checks myself but if you do get the idea send back a mail :-) >> > >> > Cheers >> > Bertrand >> > >> > > }; >> > > >> > > I've attached the device tree and include files just in case you'd >> like to take a look at those. >> > > >> > > Thanks so much for your help! I've been at this for weeks with very >> little progress. >> > > >> > > Cheers, >> > > Brad >> > > >> > > >> > > On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis < >> [email protected]> wrote: >> > > Hi Brad, >> > > >> > > > On 11 Jul 2022, at 19:38, Brad Churchwell <[email protected]> >> wrote: >> > > > >> > > > Hello, >> > > > >> > > > I've been trying to get Xen to boot dom0 with my kernel for weeks >> on an rk3399 based board and thought I'd reach out for help. It looks like >> either Xen is not properly recreating the device tree or the interrupt >> controller is just failing. The hypervisor boots but falls to initramfs >> because it cannot find the root device (nvme on pcie). Any help would be >> greatly appreciated. Here is the complete boot log >> > > >> > > From the logs you have an issue with the interrupt controller and the >> fact that your NVME is behind PCIE and as such depends on ITS is probably >> not helping. >> > > >> > > I would suggest to try to boot on usb as root fs for a try. >> > > >> > > Also it could be useful to compare the device tree on xen and without >> xen to understand what is going on (using /proc/device-tree). >> > > >> > > Xen seems to be ok but Linux is not happy with interrupts and is >> showing several issues around this area on your logs. >> > > Could you show us an extract of your device tree around the gic >> declaration ? >> > > >> > > Cheers >> > > Bertrand >> > > >> > > >> > > IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> > > <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi> >> > >> > IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> >
