On 19.05.22 11:44, Lad, Prabhakar wrote: > On Thu, May 19, 2022 at 6:30 AM Jan Kiszka <[email protected]> wrote: >> >> On 17.05.22 18:19, Lad, Prabhakar wrote: >>> Hi Jan, >>> >>> On Tue, May 17, 2022 at 12:44 PM Jan Kiszka <[email protected]> wrote: >>>> >>>> On 16.05.22 20:01, Lad, Prabhakar wrote: >>>>> Hi Jan, >>>>> >>>>> On Mon, May 16, 2022 at 6:57 AM Jan Kiszka <[email protected]> wrote: >>>>>> >>>>>> On 13.05.22 18:28, Lad, Prabhakar wrote: >>>>>>> Hi Jan, >>>>>>> >>>>>>> On Fri, May 13, 2022 at 4:33 PM Jan Kiszka <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> On 13.05.22 17:20, Lad, Prabhakar wrote: >>>>>>>>> On Fri, May 13, 2022 at 1:18 PM Jan Kiszka <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> On 13.05.22 13:32, Lad, Prabhakar wrote: >>>>>>>>>>>> Could it be that arm_dcaches_flush was a red herring? Maybe we are >>>>>>>>>>>> actually crashing on hvc in arch_entry, and that because of Linux >>>>>>>>>>>> not >>>>>>>>>>>> starting in HVC mode, thus no hypervisor take-over stub being >>>>>>>>>>>> installed. >>>>>>>>>>>> >>>>>>>>>>>> Could you share the boot log of the kernel? >>>>>>>>>>>> >>>>>>>>>>> Attached is the complete log. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> "CPU: All CPU(s) started at EL1" >>>>>>>>>> >>>>>>>>>> As suspected. Must be EL2 or HYP. Reconfigure your firmware to >>>>>>>>>> enable that. >>>>>>>>>> >>>>>>>>> Argh my bad, I've now enabled it to start in EL2 mode (attached are >>>>>>>>> the complete logs). Now I see the below panic, >>>>>>>>> >>>>>>>>> [ 33.920507] jailhouse: loading out-of-tree module taints kernel. >>>>>>>>> Reading configuration set: >>>>>>>>> Root cell: Renesas RZ/V2L SMARC (renesas-r9a07g054l2.cell) >>>>>>>>> Overlapping memory regions inside cell: None >>>>>>>>> Overlapping memory regions with hypervisor: None >>>>>>>>> Missing resource interceptions for architecture arm64: None >>>>>>>>> [ 34.741847] Bad mode in Synchronous Abort handler detected on CPU0, >>>>>>>>> code 0x5a000000 -- HVC (AArch64) >>>>>> >>>>>> "HVC instruction execution in AArch64 state, when HVC is not disabled", >>>>>> origin: EL2 (hypervisor mode). >>>>>> >>>>>> Is KVM off again when you start Jailhouse here? Does the CPU support VHE >>>>>> (Virtualization Host Extension)? Is CONFIG_ARM64_VHE enabled? Try to >>>>>> turn that off. >>>>>> >>>>> KVM was OFF, I have enabled both CONFIG_VIRTUALIZATION and CONFIG_KVM >>>>> (I have attached the complete kernel config). Yes the CPU does have >>>>> VHE support. I did it by disabling CONFIG_ARM64_VHE which did not make >>>>> any difference. >>>>> >>>>> With the VHE disable now I get the below output: >>>>> root@smarc-rzg2l:~# dmesg | grep -i kvm >>>>> [ 0.095927] kvm [1]: IPA Size Limit: 40 bits >>>>> [ 0.096895] kvm [1]: GICv3: no GICV resource entry >>>>> [ 0.096916] kvm [1]: disabling GICv2 emulation >>>>> [ 0.096949] kvm [1]: GIC system register CPU interface enabled >>>>> [ 0.097045] kvm [1]: vgic interrupt IRQ9 >>>>> [ 0.097156] kvm [1]: Hyp mode initialized successfully >>>>> root@smarc-rzg2l:~# >>>>> >>>>> Now with the above configs I don't get any output on the console when >>>>> the root cell is enabled: >>>>> >>>>> root@smarc-rzg2l:~# sh -x ./hyp.sh >>>>> + insmod jailhouse.ko >>>>> [ 35.379219] jailhouse: loading out-of-tree module taints kernel. >>>>> + jailhouse config check -a arm64 renesas-r9a07g054l2.cell >>>>> Reading configuration set: >>>>> Root cell: Renesas RZ/V2L SMARC (renesas-r9a07g054l2.cell) >>>>> Overlapping memory regions inside cell: None >>>>> Overlapping memory regions with hypervisor: None >>>>> Missing resource interceptions for architecture arm64: None >>>>> + jailhouse enable renesas-r9a07g054l2.cell >>>>> [ 36.217465] obcode @arm_dcaches_flush: d53b0024 >>>>> [ 36.217469] obcode @arm_dcaches_flush: d53b0024 >>>>> >>>>> Any pointers on debugging this further, has anyone tried jailhouse on >>>>> Cortex-A55 before? >>>> >>>> Hmm, at least not I so far. Might be the point that we are missing some >>>> subtle difference here around the way the setup works then. >>>> >>> Right. >>> >>>> In any case, the fact that at least the previous crash is gone indicates >>>> that something changed. You could check if you reach now the second hvc >>>> by adding a ret in front of it. If you, you may end up in hyp mode, >>>> maybe just not have a working UART yet (is the model you have on the >>>> board in principle already supported by Jailhouse?), run into a >>>> different issue, but just don't see Jailhouse report of it. >>>> >>> It's not going pass the arm_dcaches_flush statement. >>> >>> With the below diff, I don't see any logs on console >>> option #1 >>> diff --git a/hypervisor/arch/arm64/caches.S b/hypervisor/arch/arm64/caches.S >>> index 39dad4af..ce29b8e8 100644 >>> --- a/hypervisor/arch/arm64/caches.S >>> +++ b/hypervisor/arch/arm64/caches.S >>> @@ -40,6 +40,7 @@ >>> */ >>> .global arm_dcaches_flush >>> arm_dcaches_flush: >>> + ret >>> dcache_line_size x3, x4 >>> add x1, x0, x1 >>> sub x4, x3, #1 >>> diff --git a/hypervisor/arch/arm64/entry.S b/hypervisor/arch/arm64/entry.S >>> index a9cabf7f..c8ebf695 100644 >>> --- a/hypervisor/arch/arm64/entry.S >>> +++ b/hypervisor/arch/arm64/entry.S >>> @@ -97,6 +97,7 @@ arch_entry: >>> ldr x1, [x0, #HEADER_CORE_SIZE] >>> mov x2, DCACHE_CLEAN_AND_INVALIDATE_ASM >>> bl arm_dcaches_flush >>> + ret >>> >>> /* install bootstrap_vectors */ >>> ldr x1, =bootstrap_vectors >>> >>> >> >> Forgot: that cannot work. The call of arm_dcaches_flush will overwrite >> lr, thus the second ret will only return to where the first ret jumped >> to - endless loop. You would have to restore lr (x30) from x17 in >> arch_entry first: >> >> mov x30, x17 >> ret >> > That did the trick thanks! > > diff --git a/hypervisor/arch/arm64/entry.S b/hypervisor/arch/arm64/entry.S > index a9cabf7f..7b340bd1 100644 > --- a/hypervisor/arch/arm64/entry.S > +++ b/hypervisor/arch/arm64/entry.S > @@ -109,6 +109,8 @@ arch_entry: > mov x0, #LINUX_HVC_SET_VECTORS_LEGACY > 1: > hvc #0 > + mov x30, x17 > + ret > > hvc #0 /* bootstrap vectors enter EL2 at el2_entry */ > b . /* we don't expect to return here */ > > > With the above diff I do get the below: > > [ 42.980805] jailhouse: loading out-of-tree module taints kernel. > Reading configuration set: > Root cell: Renesas RZ/V2L SMARC (renesas-r9a07g054l2.cell) > Overlapping memory regions inside cell: None > Overlapping memory regions with hypervisor: None > Missing resource interceptions for architecture arm64: None > [ 46.582588] obcode @arm_dcaches_flush: d53b0024 > [ 46.582616] obcode @arm_dcaches_flush: d53b0024 > [ 46.611311] The Jailhouse is opening. > > So it looks like something to do with the debug console. This has to > be poked in the dark or any easy way to debug?
Well, we do not yet know what goes wrong. We do know that we can call into the hyp take-over stub and register Jailhouse with it. We do not know if we will then end up in Jailhouse in hyp mode and just lack console output or if we crash on entry already. To move the uart console out of the picture: Did you already check if the driver you select in the Jailhouse config is actually one that should support the UART on your board? Next is to double check if poking registers in the way the Jailhouse driver will do at the addresses you configured will work: Pull the code into the kernel module or even into a userspace application with /dev/mem raw register access and try out if that works in a "safe" environment (without hypervisor mode). Jan -- Siemens AG, Technology Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/a87eb794-7dc1-f447-c13a-de600a48d59e%40siemens.com.
