On 28.05.22 15:22, Lad, Prabhakar wrote:
> On Fri, May 27, 2022 at 6:07 AM Jan Kiszka <[email protected]> wrote:
>>
>> On 26.05.22 17:48, Lad, Prabhakar wrote:
>>> Hi Jan,
>>>
>>> On Tue, May 24, 2022 at 12:55 PM Lad, Prabhakar
>>> <[email protected]> wrote:
>>>>
>>>> On Mon, May 23, 2022 at 4:13 PM Jan Kiszka <[email protected]> wrote:
>>>>>
>>>>> On 23.05.22 15:55, Lad, Prabhakar wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> On Fri, May 20, 2022 at 7:08 AM Jan Kiszka <[email protected]>
>>>>>> wrote:
>>>>>>>
>>>>>>> On 19.05.22 15:23, Lad, Prabhakar wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On Thu, May 19, 2022 at 12:39 PM Jan Kiszka <[email protected]>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On 19.05.22 11:44, Lad, Prabhakar wrote:
>>>>>>>>>> On Thu, May 19, 2022 at 6:30 AM Jan Kiszka <[email protected]>
>>>>>>>>>> wrote:
>>>>>>>>>>> Forgot: that cannot work. The call of arm_dcaches_flush will
>>>>>>>>>>> overwrite
>>>>>>>>>>> lr, thus the second ret will only return to where the first ret
>>>>>>>>>>> jumped
>>>>>>>>>>> to - endless loop. You would have to restore lr (x30) from x17 in
>>>>>>>>>>> arch_entry first:
>>>>>>>>>>>
>>>>>>>>>>> mov x30, x17
>>>>>>>>>>> ret
>>>>>>>>>>>
>>>>>>>>>> That did the trick thanks!
>>>>>>>>>>
>>>>>>>>>> diff --git a/hypervisor/arch/arm64/entry.S
>>>>>>>>>> b/hypervisor/arch/arm64/entry.S
>>>>>>>>>> index a9cabf7f..7b340bd1 100644
>>>>>>>>>> --- a/hypervisor/arch/arm64/entry.S
>>>>>>>>>> +++ b/hypervisor/arch/arm64/entry.S
>>>>>>>>>> @@ -109,6 +109,8 @@ arch_entry:
>>>>>>>>>> mov x0, #LINUX_HVC_SET_VECTORS_LEGACY
>>>>>>>>>> 1:
>>>>>>>>>> hvc #0
>>>>>>>>>> + mov x30, x17
>>>>>>>>>> + ret
>>>>>>>>>>
>>>>>>>>>> hvc #0 /* bootstrap vectors enter EL2 at el2_entry
>>>>>>>>>> */
>>>>>>>>>> b . /* we don't expect to return here */
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> With the above diff I do get the below:
>>>>>>>>>>
>>>>>>>>>> [ 42.980805] jailhouse: loading out-of-tree module taints kernel.
>>>>>>>>>> Reading configuration set:
>>>>>>>>>> Root cell: Renesas RZ/V2L SMARC (renesas-r9a07g054l2.cell)
>>>>>>>>>> Overlapping memory regions inside cell: None
>>>>>>>>>> Overlapping memory regions with hypervisor: None
>>>>>>>>>> Missing resource interceptions for architecture arm64: None
>>>>>>>>>> [ 46.582588] obcode @arm_dcaches_flush: d53b0024
>>>>>>>>>> [ 46.582616] obcode @arm_dcaches_flush: d53b0024
>>>>>>>>>> [ 46.611311] The Jailhouse is opening.
>>>>>>>>>>
>>>>>>>>>> So it looks like something to do with the debug console. This has to
>>>>>>>>>> be poked in the dark or any easy way to debug?
>>>>>>>>>
>>>>>>>>> Well, we do not yet know what goes wrong. We do know that we can call
>>>>>>>>> into the hyp take-over stub and register Jailhouse with it. We do not
>>>>>>>>> know if we will then end up in Jailhouse in hyp mode and just lack
>>>>>>>>> console output or if we crash on entry already.
>>>>>>>>>
>>>>>>>> Right agreed.
>>>>>>>>
>>>>>>>>> To move the uart console out of the picture: Did you already check if
>>>>>>>>> the driver you select in the Jailhouse config is actually one that
>>>>>>>>> should support the UART on your board? Next is to double check if
>>>>>>>>> poking
>>>>>>>> The UART on this platform is almost identical to
>>>>>>>> JAILHOUSE_CON_TYPE_SCIFA type, but with some differences which I have
>>>>>>>> patched to work on this platform.
>>>>>>>>
>>>>>>>>> registers in the way the Jailhouse driver will do at the addresses you
>>>>>>>>> configured will work: Pull the code into the kernel module or even
>>>>>>>>> into
>>>>>>>>> a userspace application with /dev/mem raw register access and try out
>>>>>>>>> if
>>>>>>>>> that works in a "safe" environment (without hypervisor mode).
>>>>>>>>>
>>>>>>>> Sure will give that a shot, any pointers on doing this from userspace?
>>>>>>>>
>>>>>>>
>>>>>>> Something like this may help if you do that the first time:
>>>>>>> https://bakhi.github.io/devmem/
>>>>>>>
>>>>>> Thanks for the pointer.
>>>>>> I have tried reading/writing from the hypervisor location and that
>>>>>> goes all OK.
>>>>>
>>>>> Means, you were able to generate output on the UART. Hmm, would have
>>>>> been too easy.
>>>>>
>>>> No I meant I was able to read/write into the hypervisor memory which
>>>> is reserved in DTS.
>>>>
>>>>>> To avoid any issue related to debug UART is there any way
>>>>>> I could test this prior to enabling?
>>>>>
>>>>> Not without extra measures: Without JAILHOUSE_BORROW_ROOT_PT, which
>>>>> applies to arm64, the driver will not map the physical UART page. That
>>>>> only happens in init_bootstrap_pt which is run after jumping to EL2. So,
>>>>> we the jump goes wrong, you cannot find out where you are.
>>>>>
>>>> I see. Just to confirm it's not the debug UART the watchdog is enabled
>>>> in Linux and I see the platform reboots after 60 seconds, which is
>>>> hinting we are seeing a kernel freeze.
>>>>
>>>> Just a fyi I tried the queues/jailhouse branch from [0] and still
>>>> seeing the same issue.
>>>>
>>>>> Do you have the chance to get hold of some jtag to find out where the
>>>>> CPUs are?
>>>>>
>>>
>>> X0 FFFF00063F7ECD88 X16 0 ^S+ ^Stack_________+
>>> X1 0 X17 0
>>> X2 0 X18 FFFFFFFFFFFFFFFF
>>> X3 FFFF8000112870E8 X19 80
>>> X4 FFFF00063F7ECD80 X20 FFFF800011179000
>>> X5 FFFF800011179A48 X21 FFFF80001130BE70
>>> X6 FFFF80001101E000 X22 FFFF800010DFDED8
>>> X7 FFFF800011179000 X23 86000004
>>> X8 FFFF00063F7ECD80 X24 FFFF80001101CB38
>>> X9 0 X25 FFFF800011308000
>>> X10 00040000 X26 FFFF80001130C000
>>> X11 0 X27 FFFF800011182A40
>>> X12 0 X28 FFFF800011182A40
>>> X13 FFFFFFFFFFFE0000 X29 FFFF80001130BC40
>>> X14 FFFF800011192008 X30 FFFF800010B3B464
>>> X15 FFFF800011192048 PC FFFF8000100D9F78
>>> --------------------------------------------
>>> CPSR 80000085 N N I I SS _
>>> EL1h Z _ F _ IL _
>>> nsec C _ A _
>>> V _ D _
>>> --------------------------------------------
>>> Current ELx: SP FFFF80001130BC40
>>> ELR 0
>>> SPSR 20000085
>>> --------------------------------------------
>>> EL0: EL1:
>>> SP FFFF800011182A40 SP FFFF80001130BC40
>>> ELR 0
>>> SPSR 20000085
>>>
>>> EL2: EL3:
>>> SP 0000FF0000011000 SP 4400A500
>>> ELR FFFF8000104CC8A8 ELR 0000FFFFC02064AC
>>> SPSR 20000085 SPSR 200003C9
>>> --------------------------------------------
>>> SPSR_ABT 0 SPSR_SVC 20000085
>>> SPSR_IRQ 0 SPSR_HYP 20000085
>>> SPSR_FIQ 0
>>> SPSR_UND 0 ELR_HYP 104CC8A8
>>>
>>>
>>> Above is the CPU state, when the kernel freezes. Any hints on what
>>> might have happened?
>>
>> Can you correlate the PC value with the hypervisor disassembly? Or are
>> we actually back in Linux? Current SP == EL1 SP...
>>
>> If that should be the case, you could use the JTAG to "trace" how far
>> you get: Add an infinite loop at some point the setup should get along,
>> and then check with the debugger if PC points to that address and if EL2
>> is active. With that, you could also check if the UART print-out is
>> executed.
>>
> After tracing back I see it's looping infinitely somewhere in the
> hypervisor.o file, below is the code where it repeatedly loops.
>
> ffffc0209550: b9400680 ldr w0, [x20, #4]
> ffffc0209554: b9403481 ldr w1, [x4, #52]
> ffffc0209558: 6b00003f cmp w1, w0
> ffffc020955c: 540013a8 b.hi ffffc02097d0 <entry+0x2d8>
> .....
> ffffc02097d0: 17ffff60 b ffffc0209550 <entry+0x58>
>
> I haven't managed to find where exactly in the C file this is
> happening yet. Any thoughts on what could be happening?
>
That's very likely
while (entered_cpus < hypervisor_header.online_cpus)
cpu_relax();
Did you configure more CPUs than there are in the system?
Jan
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/dd337a1c-d678-7c7b-d371-f8bd2b019c28%40web.de.
