> On 15 Sep 2020, at 11:00, Jan Kiszka <[email protected]> wrote: > > On 15.09.20 09:07, Oliver Schwartz wrote: >> I’m currently trying out the arm64-zero-exits branch and got stuck. >> System is a Xilinx ZU9EG on a custom board, similar to zcu102. I’ve brought >> ATF up to date and patched it with Jans patch to enable SDEI. If I don’t >> enable SDEI in ATF everything works as expected (with VM exits for >> interrupts, of course). Jailhouse source is the tip of branch >> arm64-zero-exits. >> If I enable SDEI in ATF, jailhouse works most of the time, except for when >> it doesn’t. Sometimes, ‘jailhouse enable’ results in: >>> Initializing processors: >>> CPU 1... OK >>> CPU 0... >>> /home/oliver/0.12-gitAUTOINC+98061469d0-r0/git/hypervisor/arch/arm64/setup.c:73: >>> returning error -EIO > > Weird - that the SDEI event enable call. > >>> FAILED >>> JAILHOUSE_ENABLE: Input/output error >> I’ve seen this error only when I enable jailhouse through some init script >> during the boot process, when the system is also busy otherwise. When >> starting jailhouse on an idle system I haven’t seen this. > > Possibly a regression of my recent refactoring which I didn't manage to test > yet. Could you try if > > https://github.com/siemens/jailhouse/commits/e0ef829c85895dc6387d5ea11b08aa65a456255f > > <https://github.com/siemens/jailhouse/commits/e0ef829c85895dc6387d5ea11b08aa65a456255f> > > was any better? > >> Sometimes it may hang later during ‘jailhouse enable’: >>> Initializing processors: >>> CPU 1... OK >>> CPU 0... OK >>> CPU 2... OK >>> CPU 3... OK >>> Initializing unit: irqchip >>> Using SDEI-based management interrupt >>> Initializing unit: ARM SMMU v3 >>> Initializing unit: PVU IOMMU >>> Initializing unit: PCI >>> Adding virtual PCI device 00:00.0 to cell "root" >>> Page pool usage after late setup: mem 67/992, remap 5/131072 >>> Activating hypervisor >>> [ 5.847540] The Jailhouse is opening. >> Using a JTAG debugger I see that one or more cores are stuck in >> hypervisor/arch/arm-common/psci.c, line 105. >> It may also succeed in stopping one or more CPUs and then hang (again with >> one or more cores stuck in psci.c, line 105): >>> [ 5.810220] The Jailhouse is opening. >>> [ 5.860054] CPU1: shutdown >>> [ 5.862677] psci: CPU1 killed.
Now, with the first problem solved I’ve digged into the second one. It’s actually a bit worse than in my initial description: If I just do ‘jailhouse enable’ the system will always hang a few milliseconds after the command completes - the only exception is when ‘jailhouse create’ is executed immediately afterwards (which creates an inmate that uses 3 of 4 CPU cores, leaving just one for Linux), which succeeds roughly on every second try. I didn’t notice this initially because I usually start jailhouse with a script that does ‘enable’ and ‘create’. The reason for the hangs seems to be the psci emulation in Jailhouse, in particular the CPU_SUSPEND calls. These are issued from my (Xilinx-) kernel frequently if Linux has more than one core available. With SDEI disabled the core can be woken up again by some interrupt. With SDEI enabled, the core waits forever on the wfi intstruction. Because a suspended core never wakes up again the whole system hangs at some point. Any ideas why no interrupts are seen anymore in psci? My guess is that it’s because the inmate (Linux) now has full control over the GIC, so it may disable any interrupts before suspending a core, without Jailhouse noticing. If this is the case, it may be necessary to re-enable the IRQs before executing wfi. But I’m missing the big picture here - what interrupt is the core waiting for in the first place? Any other thoughts? Oliver -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/0F31A760-66D4-4430-BE71-6126FCF5AD60%40gmx.de.
