Sorry for all of the emails, but here is /proc/device-tree with XEN and without XEN:
with Xen: With XEN qos@ffa60080 interrupt-parent qos@ffa90000 qos@ffaa8080 pmu-clock-controller@ff750000 watchdog@ff848000 pwm@ff420030 serial@ff1b0000 vcc5v0-usb-regulator qos@ffa76000 mmc@fe310000 iommu@ff903f00 i2c@ff110000 #size-cells spdif@ff870000 qos@ffab0080 interrupt-controller qos@ffae0000 serial@ff370000 spi@ff200000 qos@ffac8080 xin24m name hypervisor vop@ff8f0000 psci vcc12v-dcin vcc-0v9 clock-controller@ff760000 leds qos@ffab8000 qos@ffaa0000 qos@ffa60100 iommu@ff914000 qos@ffac0000 model i2c@ff3d0000 vcc5v0-sys cpus video-codec@ff650000 mipi@ff968000 thermal-zones isp0@ff910000 i2s@ff890000 external-gmac-clock rga@ff680000 dma-controller@ff6d0000 spi@ff1e0000 spi@ff1c0000 iommu@ff670800 qos@ffa5c000 dp@fec00000 iommu@ff650800 opp-table0 serial@ff190000 display-subsystem vop@ff900000 qos@ffa98000 chosen i2c@ff160000 i2c@ff140000 qos@ffa70080 edp@ff970000 efuse@ff690000 opp-table1 syscon@ff770000 i2s@ff8a0000 hdmi-sound usb@fe3c0000 mmc@fe320000 qos@ffa60180 qos@ffa58000 usb@fe800000 i2c@ff120000 qos@ffac0080 usb@fe3e0000 aliases qos@ffad8080 vcc3v3-pcie-regulator saradc@ff100000 qos@ffaa8000 usb@fe3a0000 phy@ff800000 opp-table2 hdmi@ff940000 qos@ffac8000 qos@ffab0000 qos@ffaa0080 i2c@ff3c0000 rktimer@ff850000 iommu@ff924000 qos@ffad0000 compatible pcie@f8000000 sdio-pwrseq i2c@ff3e0000 video-codec@ff660000 memory@20000000 timer gpu@ff9a0000 iommu@ff8f3f00 qos@ffa74000 mipi@ff960000 #address-cells syscon@ff320000 power-management@ff310000 spi@ff1f0000 pinctrl vcc3v3-sys pwm@ff420000 vcc-phy-regulator vcc1v8-s0 spi@ff350000 i2s@ff880000 ethernet@fe300000 dma-controller@ff6e0000 usb@fe380000 spi@ff1d0000 serial@ff180000 iommu@ff660480 usb@fe900000 pwm@ff420010 tsadc@ff260000 phy@ff7c0000 mmc@fe330000 pwm@ff420020 i2c@ff130000 i2c@ff150000 qos@ffa70000 Without Xen root@zaku:/home/.cachengo# ls /proc/device-tree '#address-cells' qos@ffa5c000 '#size-cells' qos@ffa60080 aliases qos@ffa60100 chosen qos@ffa60180 clock-controller@ff760000 qos@ffa70000 compatible qos@ffa70080 cpus qos@ffa74000 display-subsystem qos@ffa76000 dma-controller@ff6d0000 qos@ffa90000 dma-controller@ff6e0000 qos@ffa98000 dp@fec00000 qos@ffaa0000 edp@ff970000 qos@ffaa0080 efuse@ff690000 qos@ffaa8000 ethernet@fe300000 qos@ffaa8080 external-gmac-clock qos@ffab0000 gpu@ff9a0000 qos@ffab0080 hdmi-sound qos@ffab8000 hdmi@ff940000 qos@ffac0000 i2c@ff110000 qos@ffac0080 i2c@ff120000 qos@ffac8000 i2c@ff130000 qos@ffac8080 i2c@ff140000 qos@ffad0000 i2c@ff150000 qos@ffad8080 i2c@ff160000 qos@ffae0000 i2c@ff3c0000 rga@ff680000 i2c@ff3d0000 rktimer@ff850000 i2c@ff3e0000 saradc@ff100000 i2s@ff880000 sdio-pwrseq i2s@ff890000 serial@ff180000 i2s@ff8a0000 serial@ff190000 interrupt-controller@fee00000 serial@ff1a0000 interrupt-parent serial@ff1b0000 iommu@ff650800 serial@ff370000 iommu@ff660480 spdif@ff870000 iommu@ff670800 spi@ff1c0000 iommu@ff8f3f00 spi@ff1d0000 iommu@ff903f00 spi@ff1e0000 iommu@ff914000 spi@ff1f0000 iommu@ff924000 spi@ff200000 isp0@ff910000 spi@ff350000 leds syscon@ff320000 memory syscon@ff770000 mipi@ff960000 thermal-zones mipi@ff968000 timer mmc@fe310000 tsadc@ff260000 mmc@fe320000 usb@fe380000 mmc@fe330000 usb@fe3a0000 model usb@fe3c0000 name usb@fe3e0000 opp-table0 usb@fe800000 opp-table1 usb@fe900000 opp-table2 vcc-0v9 pcie@f8000000 vcc-phy-regulator phy@ff7c0000 vcc12v-dcin phy@ff800000 vcc1v8-s0 pinctrl vcc3v3-pcie-regulator pmu-clock-controller@ff750000 vcc3v3-sys pmu_a53 vcc5v0-sys pmu_a72 vcc5v0-usb-regulator power-management@ff310000 video-codec@ff650000 psci video-codec@ff660000 pwm@ff420000 vop@ff8f0000 pwm@ff420010 vop@ff900000 pwm@ff420020 watchdog@ff848000 pwm@ff420030 xin24m qos@ffa58000 With XEN qos@ffa60080 interrupt-parent qos@ffa90000 qos@ffaa8080 pmu-clock-controller@ff750000 watchdog@ff848000 pwm@ff420030 serial@ff1b0000 vcc5v0-usb-regulator qos@ffa76000 mmc@fe310000 iommu@ff903f00 i2c@ff110000 #size-cells spdif@ff870000 qos@ffab0080 interrupt-controller qos@ffae0000 serial@ff370000 spi@ff200000 qos@ffac8080 xin24m name hypervisor vop@ff8f0000 psci vcc12v-dcin vcc-0v9 clock-controller@ff760000 leds qos@ffab8000 qos@ffaa0000 qos@ffa60100 iommu@ff914000 qos@ffac0000 model i2c@ff3d0000 vcc5v0-sys cpus video-codec@ff650000 mipi@ff968000 thermal-zones isp0@ff910000 i2s@ff890000 external-gmac-clock rga@ff680000 dma-controller@ff6d0000 spi@ff1e0000 spi@ff1c0000 iommu@ff670800 qos@ffa5c000 dp@fec00000 iommu@ff650800 opp-table0 serial@ff190000 display-subsystem vop@ff900000 qos@ffa98000 chosen i2c@ff160000 i2c@ff140000 qos@ffa70080 edp@ff970000 efuse@ff690000 opp-table1 syscon@ff770000 i2s@ff8a0000 hdmi-sound usb@fe3c0000 mmc@fe320000 qos@ffa60180 qos@ffa58000 usb@fe800000 i2c@ff120000 qos@ffac0080 usb@fe3e0000 aliases qos@ffad8080 vcc3v3-pcie-regulator saradc@ff100000 qos@ffaa8000 usb@fe3a0000 phy@ff800000 opp-table2 hdmi@ff940000 qos@ffac8000 qos@ffab0000 qos@ffaa0080 i2c@ff3c0000 rktimer@ff850000 iommu@ff924000 qos@ffad0000 compatible pcie@f8000000 sdio-pwrseq i2c@ff3e0000 video-codec@ff660000 memory@20000000 timer gpu@ff9a0000 iommu@ff8f3f00 qos@ffa74000 mipi@ff960000 #address-cells syscon@ff320000 power-management@ff310000 spi@ff1f0000 pinctrl vcc3v3-sys pwm@ff420000 vcc-phy-regulator vcc1v8-s0 spi@ff350000 i2s@ff880000 ethernet@fe300000 dma-controller@ff6e0000 usb@fe380000 spi@ff1d0000 serial@ff180000 iommu@ff660480 usb@fe900000 pwm@ff420010 tsadc@ff260000 phy@ff7c0000 mmc@fe330000 pwm@ff420020 i2c@ff130000 i2c@ff150000 qos@ffa70000 Without Xen root@zaku:/home/.cachengo# ls /proc/device-tree '#address-cells' qos@ffa5c000 '#size-cells' qos@ffa60080 aliases qos@ffa60100 chosen qos@ffa60180 clock-controller@ff760000 qos@ffa70000 compatible qos@ffa70080 cpus qos@ffa74000 display-subsystem qos@ffa76000 dma-controller@ff6d0000 qos@ffa90000 dma-controller@ff6e0000 qos@ffa98000 dp@fec00000 qos@ffaa0000 edp@ff970000 qos@ffaa0080 efuse@ff690000 qos@ffaa8000 ethernet@fe300000 qos@ffaa8080 external-gmac-clock qos@ffab0000 gpu@ff9a0000 qos@ffab0080 hdmi-sound qos@ffab8000 hdmi@ff940000 qos@ffac0000 i2c@ff110000 qos@ffac0080 i2c@ff120000 qos@ffac8000 i2c@ff130000 qos@ffac8080 i2c@ff140000 qos@ffad0000 i2c@ff150000 qos@ffad8080 i2c@ff160000 qos@ffae0000 i2c@ff3c0000 rga@ff680000 i2c@ff3d0000 rktimer@ff850000 i2c@ff3e0000 saradc@ff100000 i2s@ff880000 sdio-pwrseq i2s@ff890000 serial@ff180000 i2s@ff8a0000 serial@ff190000 interrupt-controller@fee00000 serial@ff1a0000 interrupt-parent serial@ff1b0000 iommu@ff650800 serial@ff370000 iommu@ff660480 spdif@ff870000 iommu@ff670800 spi@ff1c0000 iommu@ff8f3f00 spi@ff1d0000 iommu@ff903f00 spi@ff1e0000 iommu@ff914000 spi@ff1f0000 iommu@ff924000 spi@ff200000 isp0@ff910000 spi@ff350000 leds syscon@ff320000 memory syscon@ff770000 mipi@ff960000 thermal-zones mipi@ff968000 timer mmc@fe310000 tsadc@ff260000 mmc@fe320000 usb@fe380000 mmc@fe330000 usb@fe3a0000 model usb@fe3c0000 name usb@fe3e0000 opp-table0 usb@fe800000 opp-table1 usb@fe900000 opp-table2 vcc-0v9 pcie@f8000000 vcc-phy-regulator phy@ff7c0000 vcc12v-dcin phy@ff800000 vcc1v8-s0 pinctrl vcc3v3-pcie-regulator pmu-clock-controller@ff750000 vcc3v3-sys pmu_a53 vcc5v0-sys pmu_a72 vcc5v0-usb-regulator power-management@ff310000 video-codec@ff650000 psci video-codec@ff660000 pwm@ff420000 vop@ff8f0000 pwm@ff420010 vop@ff900000 pwm@ff420020 watchdog@ff848000 pwm@ff420030 xin24m qos@ffa58000 On Tue, Jul 12, 2022 at 11:24 AM Brad Churchwell <[email protected]> wrote: > Hi Bertrand, > > I believe I understand, but just to clarify, should I leave the > ppi-partitions block in rk3399.dtsi as is and disable the little cores, or > should I also modify that block? > > Brad > > On Tue, Jul 12, 2022 at 11:11 AM Bertrand Marquis < > [email protected]> wrote: > >> Hi Brad, >> >> > On 12 Jul 2022, at 16:59, Brad Churchwell <[email protected]> wrote: >> > >> > Hi Bertrand, >> > >> > Thanks so much for the quick response! >> > >> > I should have mentioned previously that this device tree and kernel >> Image (5.15.16) does boot properly with the rootfs without XEN. The >> interrupt errors are only present when booting with XEN. >> > >> > These are custom boards and they do have usb c, however we are unable >> to boot from usb as it doesn't receive power. We currently only use the >> usb to flash u-boot to spi which requires us to use a separate power >> adapter. These boards get power and networking through a custom backplane >> and interface with the backplane via a sata style justice (I'm not sure >> what the proper terminology would be here). >> > >> > Since I cannot boot to rootfs with XEN, I'm unable to show the device >> tree there. The only bit that I added to accommodate XEN is this bit here: >> > >> > chosen { >> > stdout-path = "serial2:1500000n8"; >> > bootargs = "hmp-unsafe=true"; >> > xen,dom0-bootargs = "console=hvc0 earlyprintk=xen clk_ignore_unused >> root=/dev/nvme0n1p2 rw init=/sbin/init rootwait rootfstype=ext4 >> ignore_loglevel"; >> > modules { >> > #address-cells = <2>; >> > #size-cells = <2>; >> > >> > module@1 { >> > compatible = "xen,linux-zimage", "xen,multiboot-module"; >> > reg = <0x0 0x03F80000 0x0 0x01aa8008>; >> > }; >> > module@2 { >> > compatible = "xen,linux-initrd", "xen,multiboot-module"; >> > reg = <0x0 0x06000000 0x0 0x02000000>; >> > }; >> > }; >> > >> > }; >> > >> > The gic declaration is in the vanilla rk3399.dtsi file and hasn't been >> modified: >> > >> > gic: interrupt-controller@fee00000 { >> > compatible = "arm,gic-v3"; >> > #interrupt-cells = <4>; >> > #address-cells = <2>; >> > #size-cells = <2>; >> > ranges; >> > interrupt-controller; >> > >> > reg = <0x0 0xfee00000 0 0x10000>, /* GICD */ >> > <0x0 0xfef00000 0 0xc0000>, /* GICR */ >> > <0x0 0xfff00000 0 0x10000>, /* GICC */ >> > <0x0 0xfff10000 0 0x10000>, /* GICH */ >> > <0x0 0xfff20000 0 0x10000>; /* GICV */ >> > interrupts = <GIC_PPI 9 IRQ_TYPE_LEVEL_HIGH 0>; >> > its: interrupt-controller@fee20000 { >> > compatible = "arm,gic-v3-its"; >> > msi-controller; >> > #msi-cells = <1>; >> > reg = <0x0 0xfee20000 0x0 0x20000>; >> > }; >> > >> > ppi-partitions { >> > ppi_cluster0: interrupt-partition-0 { >> > affinity = <&cpu_l0 &cpu_l1 &cpu_l2 >> &cpu_l3>; >> > }; >> > >> > ppi_cluster1: interrupt-partition-1 { >> > affinity = <&cpu_b0 &cpu_b1>; >> > }; >> > }; >> >> Xen is not supporting PPI partitions. >> I also saw that your booting using SMP unsafe so all core are enabled by >> Xen. >> >> Right now big/little is not supported by Xen and here you might have some >> interrupts in your system which can only occur on some specific cores. >> I would suggest to try the following: >> - only boot enable in your dtb the cores with the same type than the boot >> core >> - check in your device tree which devices are assign to the ppi_cluster >> not available and disable them >> >> This kind of configuration will be tricky to handle so it will depend on >> which devices have interrupts that can only be fired on one cluster type. >> >> Sorry but I cannot right now check in the device tree and do those checks >> myself but if you do get the idea send back a mail :-) >> >> Cheers >> Bertrand >> >> > }; >> > >> > I've attached the device tree and include files just in case you'd like >> to take a look at those. >> > >> > Thanks so much for your help! I've been at this for weeks with very >> little progress. >> > >> > Cheers, >> > Brad >> > >> > >> > On Tue, Jul 12, 2022 at 10:05 AM Bertrand Marquis < >> [email protected]> wrote: >> > Hi Brad, >> > >> > > On 11 Jul 2022, at 19:38, Brad Churchwell <[email protected]> wrote: >> > > >> > > Hello, >> > > >> > > I've been trying to get Xen to boot dom0 with my kernel for weeks on >> an rk3399 based board and thought I'd reach out for help. It looks like >> either Xen is not properly recreating the device tree or the interrupt >> controller is just failing. The hypervisor boots but falls to initramfs >> because it cannot find the root device (nvme on pcie). Any help would be >> greatly appreciated. Here is the complete boot log >> > >> > From the logs you have an issue with the interrupt controller and the >> fact that your NVME is behind PCIE and as such depends on ITS is probably >> not helping. >> > >> > I would suggest to try to boot on usb as root fs for a try. >> > >> > Also it could be useful to compare the device tree on xen and without >> xen to understand what is going on (using /proc/device-tree). >> > >> > Xen seems to be ok but Linux is not happy with interrupts and is >> showing several issues around this area on your logs. >> > Could you show us an extract of your device tree around the gic >> declaration ? >> > >> > Cheers >> > Bertrand >> > >> > >> > IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> > <rk3399-zaku.dts><rk3399-zaku.dtsi><rk3399.dtsi> >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> >
