On Fri, May 27, 2022 at 6:07 AM Jan Kiszka <[email protected]> wrote:
>
> On 26.05.22 17:48, Lad, Prabhakar wrote:
> > Hi Jan,
> >
> > On Tue, May 24, 2022 at 12:55 PM Lad, Prabhakar
> > <[email protected]> wrote:
> >>
> >> On Mon, May 23, 2022 at 4:13 PM Jan Kiszka <[email protected]> wrote:
> >>>
> >>> On 23.05.22 15:55, Lad, Prabhakar wrote:
> >>>> Hi Jan,
> >>>>
> >>>> On Fri, May 20, 2022 at 7:08 AM Jan Kiszka <[email protected]>
> >>>> wrote:
> >>>>>
> >>>>> On 19.05.22 15:23, Lad, Prabhakar wrote:
> >>>>>> Hi Jan,
> >>>>>>
> >>>>>> On Thu, May 19, 2022 at 12:39 PM Jan Kiszka <[email protected]>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On 19.05.22 11:44, Lad, Prabhakar wrote:
> >>>>>>>> On Thu, May 19, 2022 at 6:30 AM Jan Kiszka <[email protected]>
> >>>>>>>> wrote:
> >>>>>>>>> Forgot: that cannot work. The call of arm_dcaches_flush will
> >>>>>>>>> overwrite
> >>>>>>>>> lr, thus the second ret will only return to where the first ret
> >>>>>>>>> jumped
> >>>>>>>>> to - endless loop. You would have to restore lr (x30) from x17 in
> >>>>>>>>> arch_entry first:
> >>>>>>>>>
> >>>>>>>>> mov x30, x17
> >>>>>>>>> ret
> >>>>>>>>>
> >>>>>>>> That did the trick thanks!
> >>>>>>>>
> >>>>>>>> diff --git a/hypervisor/arch/arm64/entry.S
> >>>>>>>> b/hypervisor/arch/arm64/entry.S
> >>>>>>>> index a9cabf7f..7b340bd1 100644
> >>>>>>>> --- a/hypervisor/arch/arm64/entry.S
> >>>>>>>> +++ b/hypervisor/arch/arm64/entry.S
> >>>>>>>> @@ -109,6 +109,8 @@ arch_entry:
> >>>>>>>> mov x0, #LINUX_HVC_SET_VECTORS_LEGACY
> >>>>>>>> 1:
> >>>>>>>> hvc #0
> >>>>>>>> + mov x30, x17
> >>>>>>>> + ret
> >>>>>>>>
> >>>>>>>> hvc #0 /* bootstrap vectors enter EL2 at el2_entry
> >>>>>>>> */
> >>>>>>>> b . /* we don't expect to return here */
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> With the above diff I do get the below:
> >>>>>>>>
> >>>>>>>> [ 42.980805] jailhouse: loading out-of-tree module taints kernel.
> >>>>>>>> Reading configuration set:
> >>>>>>>> Root cell: Renesas RZ/V2L SMARC (renesas-r9a07g054l2.cell)
> >>>>>>>> Overlapping memory regions inside cell: None
> >>>>>>>> Overlapping memory regions with hypervisor: None
> >>>>>>>> Missing resource interceptions for architecture arm64: None
> >>>>>>>> [ 46.582588] obcode @arm_dcaches_flush: d53b0024
> >>>>>>>> [ 46.582616] obcode @arm_dcaches_flush: d53b0024
> >>>>>>>> [ 46.611311] The Jailhouse is opening.
> >>>>>>>>
> >>>>>>>> So it looks like something to do with the debug console. This has to
> >>>>>>>> be poked in the dark or any easy way to debug?
> >>>>>>>
> >>>>>>> Well, we do not yet know what goes wrong. We do know that we can call
> >>>>>>> into the hyp take-over stub and register Jailhouse with it. We do not
> >>>>>>> know if we will then end up in Jailhouse in hyp mode and just lack
> >>>>>>> console output or if we crash on entry already.
> >>>>>>>
> >>>>>> Right agreed.
> >>>>>>
> >>>>>>> To move the uart console out of the picture: Did you already check if
> >>>>>>> the driver you select in the Jailhouse config is actually one that
> >>>>>>> should support the UART on your board? Next is to double check if
> >>>>>>> poking
> >>>>>> The UART on this platform is almost identical to
> >>>>>> JAILHOUSE_CON_TYPE_SCIFA type, but with some differences which I have
> >>>>>> patched to work on this platform.
> >>>>>>
> >>>>>>> registers in the way the Jailhouse driver will do at the addresses you
> >>>>>>> configured will work: Pull the code into the kernel module or even
> >>>>>>> into
> >>>>>>> a userspace application with /dev/mem raw register access and try out
> >>>>>>> if
> >>>>>>> that works in a "safe" environment (without hypervisor mode).
> >>>>>>>
> >>>>>> Sure will give that a shot, any pointers on doing this from userspace?
> >>>>>>
> >>>>>
> >>>>> Something like this may help if you do that the first time:
> >>>>> https://bakhi.github.io/devmem/
> >>>>>
> >>>> Thanks for the pointer.
> >>>> I have tried reading/writing from the hypervisor location and that
> >>>> goes all OK.
> >>>
> >>> Means, you were able to generate output on the UART. Hmm, would have
> >>> been too easy.
> >>>
> >> No I meant I was able to read/write into the hypervisor memory which
> >> is reserved in DTS.
> >>
> >>>> To avoid any issue related to debug UART is there any way
> >>>> I could test this prior to enabling?
> >>>
> >>> Not without extra measures: Without JAILHOUSE_BORROW_ROOT_PT, which
> >>> applies to arm64, the driver will not map the physical UART page. That
> >>> only happens in init_bootstrap_pt which is run after jumping to EL2. So,
> >>> we the jump goes wrong, you cannot find out where you are.
> >>>
> >> I see. Just to confirm it's not the debug UART the watchdog is enabled
> >> in Linux and I see the platform reboots after 60 seconds, which is
> >> hinting we are seeing a kernel freeze.
> >>
> >> Just a fyi I tried the queues/jailhouse branch from [0] and still
> >> seeing the same issue.
> >>
> >>> Do you have the chance to get hold of some jtag to find out where the
> >>> CPUs are?
> >>>
> >
> > X0 FFFF00063F7ECD88 X16 0 ^S+ ^Stack_________+
> > X1 0 X17 0
> > X2 0 X18 FFFFFFFFFFFFFFFF
> > X3 FFFF8000112870E8 X19 80
> > X4 FFFF00063F7ECD80 X20 FFFF800011179000
> > X5 FFFF800011179A48 X21 FFFF80001130BE70
> > X6 FFFF80001101E000 X22 FFFF800010DFDED8
> > X7 FFFF800011179000 X23 86000004
> > X8 FFFF00063F7ECD80 X24 FFFF80001101CB38
> > X9 0 X25 FFFF800011308000
> > X10 00040000 X26 FFFF80001130C000
> > X11 0 X27 FFFF800011182A40
> > X12 0 X28 FFFF800011182A40
> > X13 FFFFFFFFFFFE0000 X29 FFFF80001130BC40
> > X14 FFFF800011192008 X30 FFFF800010B3B464
> > X15 FFFF800011192048 PC FFFF8000100D9F78
> > --------------------------------------------
> > CPSR 80000085 N N I I SS _
> > EL1h Z _ F _ IL _
> > nsec C _ A _
> > V _ D _
> > --------------------------------------------
> > Current ELx: SP FFFF80001130BC40
> > ELR 0
> > SPSR 20000085
> > --------------------------------------------
> > EL0: EL1:
> > SP FFFF800011182A40 SP FFFF80001130BC40
> > ELR 0
> > SPSR 20000085
> >
> > EL2: EL3:
> > SP 0000FF0000011000 SP 4400A500
> > ELR FFFF8000104CC8A8 ELR 0000FFFFC02064AC
> > SPSR 20000085 SPSR 200003C9
> > --------------------------------------------
> > SPSR_ABT 0 SPSR_SVC 20000085
> > SPSR_IRQ 0 SPSR_HYP 20000085
> > SPSR_FIQ 0
> > SPSR_UND 0 ELR_HYP 104CC8A8
> >
> >
> > Above is the CPU state, when the kernel freezes. Any hints on what
> > might have happened?
>
> Can you correlate the PC value with the hypervisor disassembly? Or are
> we actually back in Linux? Current SP == EL1 SP...
>
> If that should be the case, you could use the JTAG to "trace" how far
> you get: Add an infinite loop at some point the setup should get along,
> and then check with the debugger if PC points to that address and if EL2
> is active. With that, you could also check if the UART print-out is
> executed.
>
After tracing back I see it's looping infinitely somewhere in the
hypervisor.o file, below is the code where it repeatedly loops.
ffffc0209550: b9400680 ldr w0, [x20, #4]
ffffc0209554: b9403481 ldr w1, [x4, #52]
ffffc0209558: 6b00003f cmp w1, w0
ffffc020955c: 540013a8 b.hi ffffc02097d0 <entry+0x2d8>
.....
ffffc02097d0: 17ffff60 b ffffc0209550 <entry+0x58>
I haven't managed to find where exactly in the C file this is
happening yet. Any thoughts on what could be happening?
Below is the snippet of dissembled code:
=================================>>
0000ffffc02094f8 <entry>:
ffffc02094f8: a9b87bfd stp x29, x30, [sp, #-128]!
ffffc02094fc: 910003fd mov x29, sp
ffffc0209500: a90153f3 stp x19, x20, [sp, #16]
ffffc0209504: a9025bf5 stp x21, x22, [sp, #32]
ffffc0209508: 2a0003f6 mov w22, w0
ffffc020950c: a90363f7 stp x23, x24, [sp, #48]
ffffc0209510: aa0103f5 mov x21, x1
ffffc0209514: a9046bf9 stp x25, x26, [sp, #64]
ffffc0209518: f9002bfb str x27, [sp, #80]
ffffc020951c: b9300020 str w0, [x1, #12288]
ffffc0209520: 97ffffe7 bl ffffc02094bc <spin_lock.constprop.1>
ffffc0209524: d5033bbf dmb ish
ffffc0209528: b0000093 adrp x19, ffffc021a000 <init_lock>
ffffc020952c: 91000274 add x20, x19, #0x0
ffffc0209530: b9400680 ldr w0, [x20, #4]
ffffc0209534: 11000400 add w0, w0, #0x1
ffffc0209538: b9000680 str w0, [x20, #4]
ffffc020953c: 79400260 ldrh w0, [x19]
ffffc0209540: 11000400 add w0, w0, #0x1
ffffc0209544: 489ffe80 stlrh w0, [x20]
ffffc0209548: f0ffffb8 adrp x24, ffffc0200000 <hypervisor_header>
ffffc020954c: 91000304 add x4, x24, #0x0
ffffc0209550: b9400680 ldr w0, [x20, #4]
==== >>
ffffc0209554: b9403481 ldr w1, [x4, #52]
====>> It loops here.
ffffc0209558: 6b00003f cmp w1, w0
====>>
ffffc020955c: 540013a8 b.hi ffffc02097d0 <entry+0x2d8>
// b.pmore ====>>
ffffc0209560: 97ffffd7 bl ffffc02094bc <spin_lock.constprop.1>
ffffc0209564: b0000042 adrp x2, ffffc0212000 <uart_hscif_ops>
ffffc0209568: b9419840 ldr w0, [x2, #408]
ffffc020956c: 3100041f cmn w0, #0x1
......
.....
ffffc02097b4: 35001300 cbnz w0, ffffc0209a14 <entry+0x51c>
ffffc02097b8: 340012f7 cbz w23, ffffc0209a14 <entry+0x51c>
ffffc02097bc: b00000b4 adrp x20, ffffc021e000 <root_cell>
ffffc02097c0: 52800015 mov w21, #0x0 // #0
ffffc02097c4: 12800000 mov w0, #0xffffffff // #-1
ffffc02097c8: 91000299 add x25, x20, #0x0
ffffc02097cc: 1400002a b ffffc0209874 <entry+0x37c>
ffffc02097d0: 17ffff60 b ffffc0209550 <entry+0x58>
ffffc02097d4: b94002e2 ldr w2, [x23]
ffffc02097d8: f0000080 adrp x0, ffffc021c000 <virtual_console>
ffffc02097dc: f9400400 ldr x0, [x0, #8]
ffffc02097e0: 34000202 cbz w2, ffffc0209820 <entry+0x328>
ffffc02097e4: cb000362 sub x2, x27, x0
ffffc02097e8: eb01005f cmp x2, x1
ffffc02097ec: 540001a1 b.ne ffffc0209820 <entry+0x328> // b.any
ffffc02097f0: f90033a2 str x2, [x29, #96]
ffffc02097f4: 910183a1 add x1, x29, #0x60
ffffc02097f8: 91000320 add x0, x25, #0x0
ffffc02097fc: 97ffe681 bl ffffc0203200 <arch_map_memory_region>
ffffc0209800: 91000261 add x1, x19, #0x0
ffffc0209804: b9000820 str w0, [x1, #8]
ffffc0209808: b9400820 ldr w0, [x1, #8]
Cheers,
Prabhakar
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/CA%2BV-a8vxVLxV8iog0-JUH-Bd4nnCj5ELYkd_SkDDmPuKOAiJKg%40mail.gmail.com.
