Hi Jan,
On Thu, May 12, 2022 at 12:16 PM Jan Kiszka <[email protected]> wrote:
>
> On 12.05.22 13:06, Lad, Prabhakar wrote:
> > Hi Jan,
> >
> > On Thu, May 12, 2022 at 11:24 AM Jan Kiszka <[email protected]> wrote:
> >>
> >> On 12.05.22 09:01, Lad, Prabhakar wrote:
> >>> Hi Jan,
> >>>
> >>> On Thu, May 12, 2022 at 6:45 AM Jan Kiszka <[email protected]> wrote:
> >>>>
> >>>> On 11.05.22 19:09, Lad, Prabhakar wrote:
> >>>>> Hi Jan,
> >>>>>
> >>>>> On Wed, May 11, 2022 at 4:11 PM Jan Kiszka <[email protected]>
> >>>>> wrote:
> >>>>>>
> >>>>>> On 11.05.22 13:20, Prabhakar Lad wrote:
> >>>>>>> To add further more details:
> >>>>>>>
> >>>>>>> I am using jailhouse-enabling/5.10 Linux branch [0] with -next branch
> >>>>>>> for jailhouse [1].
> >>>>>>>
> >>>>>>> I added some debug prints and I see the panic is caused when entry()
> >>>>>>> function is called (in enter_hypervisor). The entry function lands
> >>>>>>> into
> >>>>>>> assembly code (entry.S). I dont have a JTAG to see which exact
> >>>>>>> instruction is causing this issue.
> >>>>>>
> >>>>>> So, already the first instruction in the loaded hypervisor binary is
> >>>>>> not
> >>>>>> executable? That would explain why we see no hypervisor output at all.
> >>>>>>
> >>>>> To clarify when the hypervisor is loaded the output will be via
> >>>>> debug_console specified in the root cell config?
> >>>>>
> >>>>
> >>>> Correct - if you reach entry() in setup.c.
> >>>>
> >>>>>> Was that memory region properly reserved from Linux (via DTB carve-out
> >>>>>> e.g.)?
> >>>>>>
> >>>>> Yes I have the below memory reserved in my dts:
> >>>>>
> >>>>> memory@48000000 {
> >>>>> device_type = "memory";
> >>>>> /* first 128MB is reserved for secure area. */
> >>>>> reg = <0x0 0x48000000 0x0 0x78000000>;
> >>>>> };
> >>>>>
> >>>>> reserved-memory {
> >>>>> #address-cells = <2>;
> >>>>> #size-cells = <2>;
> >>>>> ranges;
> >>>>>
> >>>>> jh_inmate@a7f00000 {
> >>>>> status = "okay";
> >>>>> no-map;
> >>>>> reg = <0x00 0xa7f00000 0x00 0xf000000>;
> >>>>> };
> >>>>>
> >>>>> jailhouse: jailhouse@b6f00000 {
> >>>>> status = "okay";
> >>>>> reg = <0x0 0xb6f00000 0x0 0x1000000>;
> >>>>> no-map;
> >>>>> };
> >>>>> };
> >>>>>
> >>>>> Linux does report the hole in RAM:
> >>>>>
> >>>>> root@smarc-rzg2l:~# cat /proc/iomem | grep RAM
> >>>>> 48000000-a7efffff : System RAM
> >>>>> b7f00000-bfffffff : System RAM
> >>>>>
> >>>>> And below is my root cell config related to hypervisor memory:
> >>>>>
> >>>>> .hypervisor_memory = {
> >>>>> .phys_start = 0xb6f00000,
> >>>>> .size = 0x1000000,
> >>>>> },
> >>>>>
> >>>>> Is there anything obvious I have missed above?
> >>>>>
> >>>>
> >>>> Nothing obvious to me so far.
> >>>>
> >>>> So, is this really the first instruction in
> >>>> hypervisor/arch/arm64/entry.S, arch_entry, that triggers the undefinstr?
> >>>> Check the reported address by Linux against the disassembly of the
> >>>> hypervisor. You could also play with adding 'ret' as first instruction,
> >>>> to check if that returns without a crash.
> >>>>
> >>> I did play around with ret, below is my observation:
> >>>
> >>> Up until line 98 [0] just before calling arm_dcaches_flush adding ret
> >>> returned without a crash. Adding a ret at line 99 [1] ie after
> >>> arm_dcaches_flush it never returned.
> >>>
> >>> So I continued with adding ret in arm_dcaches_flush, I added ret as a
> >>> first statement in arm_dcaches_flush and was able to see the panic!
> >>
> >> Which Jailhouse revision are you building? Already disassembled
> >> hypervisor.o around arch_entry and arm_dcaches_flush? This is what I
> >> have here for next:
> >>
> > I'm using the next branch too.
> >
> >> 0000ffffc0203efc <arm_dcaches_flush>:
> >> ffffc0203efc: d53b0024 mrs x4, ctr_el0
> >> ffffc0203f00: d3504c84 ubfx x4, x4, #16, #4
> >> ...
> >>
> >> 0000ffffc0204800 <arch_entry>:
> >> ffffc0204800: aa0003f0 mov x16, x0
> >> ffffc0204804: aa1e03f1 mov x17, x30
> >> ...
> >> ffffc0204844: d2800042 mov x2, #0x2
> >> // #2
> >> ffffc0204848: 97fffdad bl ffffc0203efc
> >> <arm_dcaches_flush>
> >>
> >> You could check if there is such a relative call in your case as well.
> > yes it does exist, below is the snippet:
> >
> > 0000ffffc0204000 <arch_entry>:
> > ffffc0204000: aa0003f0 mov x16, x0
> > ffffc0204004: aa1e03f1 mov x17, x30
> > ffffc0204008: 10fdffc0 adr x0, ffffc0200000
> > <hypervisor_header>
> > ffffc020400c: f9402412 ldr x18, [x0, #72]
> > ffffc0204010: 5800fd0f ldr x15, ffffc0205fb0
> > <sdei_handler+0x2c>
> > ffffc0204014: 900000e1 adrp x1, ffffc0220000 <__page_pool>
> > ffffc0204018: 79406002 ldrh w2, [x0, #48]
> > ffffc020401c: d2880003 mov x3, #0x4000
> > // #16384
> > ffffc0204020: 9b030441 madd x1, x2, x3, x1
> > ffffc0204024: f842c02e ldur x14, [x1, #44]
> > ffffc0204028: 5800fc8d ldr x13, ffffc0205fb8
> > <sdei_handler+0x34>
> > ffffc020402c: f840c02c ldur x12, [x1, #12]
> > ffffc0204030: cb0d018b sub x11, x12, x13
> > ffffc0204034: 924051c1 and x1, x14, #0x1fffff
> > ffffc0204038: 8b0101ef add x15, x15, x1
> > ffffc020403c: f9001c0f str x15, [x0, #56]
> > ffffc0204040: f9400401 ldr x1, [x0, #8]
> > ffffc0204044: d2800042 mov x2, #0x2 //
> > #2
> > ffffc0204048: 97ffff6c bl ffffc0203df8 <arm_dcaches_flush>
> > ffffc020404c: 5800fba1 ldr x1, ffffc0205fc0
> > <sdei_handler+0x3c>
> > ffffc0204050: 8b0b0021 add x1, x1, x11
> > ffffc0204054: d2800000 mov x0, #0x0 //
> > #0
> > ffffc0204058: f100025f cmp x18, #0x0
> > ffffc020405c: 54000041 b.ne ffffc0204064
> > <arch_entry+0x64> // b.any
> > ffffc0204060: d2800020 mov x0, #0x1 //
> > #1
> > ffffc0204064: d4000002 hvc #0x0
> > ffffc0204068: d4000002 hvc #0x0
> > ffffc020406c: 14000000 b ffffc020406c <arch_entry+0x6c>
> >
> >> Then you could check, before jumping into arch_entry from the
> >> hypervisor, that the opcode at the actual arm_dcaches_flush address is
> >> as expected. But maybe already the building injects an issue here.
> >>
> > Any pointers on how I could do that?
> >
>
> arm_dcaches_flush is loaded at arch_entry (header->entry +
> hypervisor_mem) - 0x208. Read the u32 at that address and check if it is
> what is in your hypervisor.o (likely also d53b0024).
>
> If that is the case, not the jump but that "mrs x4, ctr_el0" may
> actually be the problem. Check out hypervisor/arch/arm64/caches.S and
> see if that code, specifically dcache_line_size, causes trouble outside
> of Jailhouse as well, maybe as inline assembly in the driver module.
>
With the below ret added, I get "JAILHOUSE_ENABLE: Success"
diff --git a/hypervisor/arch/arm64/entry.S b/hypervisor/arch/arm64/entry.S
index a9cabf7f..4e98b292 100644
--- a/hypervisor/arch/arm64/entry.S
+++ b/hypervisor/arch/arm64/entry.S
@@ -96,6 +96,7 @@ arch_entry:
*/
ldr x1, [x0, #HEADER_CORE_SIZE]
mov x2, DCACHE_CLEAN_AND_INVALIDATE_ASM
+ ret
bl arm_dcaches_flush
/* install bootstrap_vectors */
Now when I undo the above and do the below, Im seeing a panic:
diff --git a/hypervisor/arch/arm64/caches.S b/hypervisor/arch/arm64/caches.S
index 39dad4af..ce29b8e8 100644
--- a/hypervisor/arch/arm64/caches.S
+++ b/hypervisor/arch/arm64/caches.S
@@ -40,6 +40,7 @@
*/
.global arm_dcaches_flush
arm_dcaches_flush:
+ ret
dcache_line_size x3, x4
add x1, x0, x1
sub x4, x3, #1
Issue is seen even without dcache_line_size being called. Does that
mean we are not landing in arm_dcaches_flush?
Cheers,
Prabhakar
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/CA%2BV-a8tM_umwZ-%2Bvt7VST1pBJF2MxmbOJSonzDDbJz_OV%3DGE3w%40mail.gmail.com.
