Hi Jan, On Thu, May 12, 2022 at 6:05 PM Jan Kiszka <[email protected]> wrote: > > On 12.05.22 13:37, Lad, Prabhakar wrote: > > Hi Jan, > > > > On Thu, May 12, 2022 at 12:16 PM Jan Kiszka <[email protected]> wrote: > >> > >> On 12.05.22 13:06, Lad, Prabhakar wrote: > >>> Hi Jan, > >>> > >>> On Thu, May 12, 2022 at 11:24 AM Jan Kiszka <[email protected]> > >>> wrote: > >>>> > >>>> On 12.05.22 09:01, Lad, Prabhakar wrote: > >>>>> Hi Jan, > >>>>> > >>>>> On Thu, May 12, 2022 at 6:45 AM Jan Kiszka <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>> On 11.05.22 19:09, Lad, Prabhakar wrote: > >>>>>>> Hi Jan, > >>>>>>> > >>>>>>> On Wed, May 11, 2022 at 4:11 PM Jan Kiszka <[email protected]> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> On 11.05.22 13:20, Prabhakar Lad wrote: > >>>>>>>>> To add further more details: > >>>>>>>>> > >>>>>>>>> I am using jailhouse-enabling/5.10 Linux branch [0] with -next > >>>>>>>>> branch > >>>>>>>>> for jailhouse [1]. > >>>>>>>>> > >>>>>>>>> I added some debug prints and I see the panic is caused when entry() > >>>>>>>>> function is called (in enter_hypervisor). The entry function lands > >>>>>>>>> into > >>>>>>>>> assembly code (entry.S). I dont have a JTAG to see which exact > >>>>>>>>> instruction is causing this issue. > >>>>>>>> > >>>>>>>> So, already the first instruction in the loaded hypervisor binary is > >>>>>>>> not > >>>>>>>> executable? That would explain why we see no hypervisor output at > >>>>>>>> all. > >>>>>>>> > >>>>>>> To clarify when the hypervisor is loaded the output will be via > >>>>>>> debug_console specified in the root cell config? > >>>>>>> > >>>>>> > >>>>>> Correct - if you reach entry() in setup.c. > >>>>>> > >>>>>>>> Was that memory region properly reserved from Linux (via DTB > >>>>>>>> carve-out > >>>>>>>> e.g.)? > >>>>>>>> > >>>>>>> Yes I have the below memory reserved in my dts: > >>>>>>> > >>>>>>> memory@48000000 { > >>>>>>> device_type = "memory"; > >>>>>>> /* first 128MB is reserved for secure area. */ > >>>>>>> reg = <0x0 0x48000000 0x0 0x78000000>; > >>>>>>> }; > >>>>>>> > >>>>>>> reserved-memory { > >>>>>>> #address-cells = <2>; > >>>>>>> #size-cells = <2>; > >>>>>>> ranges; > >>>>>>> > >>>>>>> jh_inmate@a7f00000 { > >>>>>>> status = "okay"; > >>>>>>> no-map; > >>>>>>> reg = <0x00 0xa7f00000 0x00 0xf000000>; > >>>>>>> }; > >>>>>>> > >>>>>>> jailhouse: jailhouse@b6f00000 { > >>>>>>> status = "okay"; > >>>>>>> reg = <0x0 0xb6f00000 0x0 0x1000000>; > >>>>>>> no-map; > >>>>>>> }; > >>>>>>> }; > >>>>>>> > >>>>>>> Linux does report the hole in RAM: > >>>>>>> > >>>>>>> root@smarc-rzg2l:~# cat /proc/iomem | grep RAM > >>>>>>> 48000000-a7efffff : System RAM > >>>>>>> b7f00000-bfffffff : System RAM > >>>>>>> > >>>>>>> And below is my root cell config related to hypervisor memory: > >>>>>>> > >>>>>>> .hypervisor_memory = { > >>>>>>> .phys_start = 0xb6f00000, > >>>>>>> .size = 0x1000000, > >>>>>>> }, > >>>>>>> > >>>>>>> Is there anything obvious I have missed above? > >>>>>>> > >>>>>> > >>>>>> Nothing obvious to me so far. > >>>>>> > >>>>>> So, is this really the first instruction in > >>>>>> hypervisor/arch/arm64/entry.S, arch_entry, that triggers the > >>>>>> undefinstr? > >>>>>> Check the reported address by Linux against the disassembly of the > >>>>>> hypervisor. You could also play with adding 'ret' as first instruction, > >>>>>> to check if that returns without a crash. > >>>>>> > >>>>> I did play around with ret, below is my observation: > >>>>> > >>>>> Up until line 98 [0] just before calling arm_dcaches_flush adding ret > >>>>> returned without a crash. Adding a ret at line 99 [1] ie after > >>>>> arm_dcaches_flush it never returned. > >>>>> > >>>>> So I continued with adding ret in arm_dcaches_flush, I added ret as a > >>>>> first statement in arm_dcaches_flush and was able to see the panic! > >>>> > >>>> Which Jailhouse revision are you building? Already disassembled > >>>> hypervisor.o around arch_entry and arm_dcaches_flush? This is what I > >>>> have here for next: > >>>> > >>> I'm using the next branch too. > >>> > >>>> 0000ffffc0203efc <arm_dcaches_flush>: > >>>> ffffc0203efc: d53b0024 mrs x4, ctr_el0 > >>>> ffffc0203f00: d3504c84 ubfx x4, x4, #16, #4 > >>>> ... > >>>> > >>>> 0000ffffc0204800 <arch_entry>: > >>>> ffffc0204800: aa0003f0 mov x16, x0 > >>>> ffffc0204804: aa1e03f1 mov x17, x30 > >>>> ... > >>>> ffffc0204844: d2800042 mov x2, #0x2 > >>>> // #2 > >>>> ffffc0204848: 97fffdad bl ffffc0203efc > >>>> <arm_dcaches_flush> > >>>> > >>>> You could check if there is such a relative call in your case as well. > >>> yes it does exist, below is the snippet: > >>> > >>> 0000ffffc0204000 <arch_entry>: > >>> ffffc0204000: aa0003f0 mov x16, x0 > >>> ffffc0204004: aa1e03f1 mov x17, x30 > >>> ffffc0204008: 10fdffc0 adr x0, ffffc0200000 > >>> <hypervisor_header> > >>> ffffc020400c: f9402412 ldr x18, [x0, #72] > >>> ffffc0204010: 5800fd0f ldr x15, ffffc0205fb0 > >>> <sdei_handler+0x2c> > >>> ffffc0204014: 900000e1 adrp x1, ffffc0220000 <__page_pool> > >>> ffffc0204018: 79406002 ldrh w2, [x0, #48] > >>> ffffc020401c: d2880003 mov x3, #0x4000 > >>> // #16384 > >>> ffffc0204020: 9b030441 madd x1, x2, x3, x1 > >>> ffffc0204024: f842c02e ldur x14, [x1, #44] > >>> ffffc0204028: 5800fc8d ldr x13, ffffc0205fb8 > >>> <sdei_handler+0x34> > >>> ffffc020402c: f840c02c ldur x12, [x1, #12] > >>> ffffc0204030: cb0d018b sub x11, x12, x13 > >>> ffffc0204034: 924051c1 and x1, x14, #0x1fffff > >>> ffffc0204038: 8b0101ef add x15, x15, x1 > >>> ffffc020403c: f9001c0f str x15, [x0, #56] > >>> ffffc0204040: f9400401 ldr x1, [x0, #8] > >>> ffffc0204044: d2800042 mov x2, #0x2 > >>> // #2 > >>> ffffc0204048: 97ffff6c bl ffffc0203df8 <arm_dcaches_flush> > >>> ffffc020404c: 5800fba1 ldr x1, ffffc0205fc0 > >>> <sdei_handler+0x3c> > >>> ffffc0204050: 8b0b0021 add x1, x1, x11 > >>> ffffc0204054: d2800000 mov x0, #0x0 > >>> // #0 > >>> ffffc0204058: f100025f cmp x18, #0x0 > >>> ffffc020405c: 54000041 b.ne ffffc0204064 > >>> <arch_entry+0x64> // b.any > >>> ffffc0204060: d2800020 mov x0, #0x1 > >>> // #1 > >>> ffffc0204064: d4000002 hvc #0x0 > >>> ffffc0204068: d4000002 hvc #0x0 > >>> ffffc020406c: 14000000 b ffffc020406c <arch_entry+0x6c> > >>> > >>>> Then you could check, before jumping into arch_entry from the > >>>> hypervisor, that the opcode at the actual arm_dcaches_flush address is > >>>> as expected. But maybe already the building injects an issue here. > >>>> > >>> Any pointers on how I could do that? > >>> > >> > >> arm_dcaches_flush is loaded at arch_entry (header->entry + > >> hypervisor_mem) - 0x208. Read the u32 at that address and check if it is > >> what is in your hypervisor.o (likely also d53b0024). > >> > >> If that is the case, not the jump but that "mrs x4, ctr_el0" may > >> actually be the problem. Check out hypervisor/arch/arm64/caches.S and > >> see if that code, specifically dcache_line_size, causes trouble outside > >> of Jailhouse as well, maybe as inline assembly in the driver module. > >> > > > > With the below ret added, I get "JAILHOUSE_ENABLE: Success" > > > > diff --git a/hypervisor/arch/arm64/entry.S b/hypervisor/arch/arm64/entry.S > > index a9cabf7f..4e98b292 100644 > > --- a/hypervisor/arch/arm64/entry.S > > +++ b/hypervisor/arch/arm64/entry.S > > @@ -96,6 +96,7 @@ arch_entry: > > */ > > ldr x1, [x0, #HEADER_CORE_SIZE] > > mov x2, DCACHE_CLEAN_AND_INVALIDATE_ASM > > + ret > > bl arm_dcaches_flush > > > > /* install bootstrap_vectors */ > > > > Now when I undo the above and do the below, Im seeing a panic: > > > > diff --git a/hypervisor/arch/arm64/caches.S b/hypervisor/arch/arm64/caches.S > > index 39dad4af..ce29b8e8 100644 > > --- a/hypervisor/arch/arm64/caches.S > > +++ b/hypervisor/arch/arm64/caches.S > > @@ -40,6 +40,7 @@ > > */ > > .global arm_dcaches_flush > > arm_dcaches_flush: > > + ret > > dcache_line_size x3, x4 > > add x1, x0, x1 > > sub x4, x3, #1 > > > > Issue is seen even without dcache_line_size being called. Does that > > mean we are not landing in arm_dcaches_flush? > > Likely. I've never seen such an effect. > > If you look the reported fault address, when making it relative > (subtract hypervisor_mem), is that arm_dcaches_flush (relative to > arch_entry)? > Could you please elaborate on it more. I moved the cache.S code in entry.S file but still seeing the same issue.
In some of the reference platforms LPAE is enabled in the u-boot. Is that a strict requirement? Also in the requirement section it's mentioned "Linux is started in HYP mode" does that mean Before loading the jailhouse the Linux should be running on EL2? Also to be sure, do we need any special configs enabled in TF-A at all? Fyi, I am using arm64_defconfig_5.10 [0] (+ additional configs to enable my platform) to build the Linux kernel, should these configs be sufficient for Jailhouse? [0] https://github.com/siemens/jailhouse-images/blob/next/recipes-kernel/linux/files/arm64_defconfig_5.10 Cheers, Prabhakar -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CA%2BV-a8uh_mJNrN8R8cHb%2BMwTG1%3DDPyjy3kEOwfozRdOL%2Bjz3zw%40mail.gmail.com.
