On 12.05.22 13:06, Lad, Prabhakar wrote: > Hi Jan, > > On Thu, May 12, 2022 at 11:24 AM Jan Kiszka <[email protected]> wrote: >> >> On 12.05.22 09:01, Lad, Prabhakar wrote: >>> Hi Jan, >>> >>> On Thu, May 12, 2022 at 6:45 AM Jan Kiszka <[email protected]> wrote: >>>> >>>> On 11.05.22 19:09, Lad, Prabhakar wrote: >>>>> Hi Jan, >>>>> >>>>> On Wed, May 11, 2022 at 4:11 PM Jan Kiszka <[email protected]> wrote: >>>>>> >>>>>> On 11.05.22 13:20, Prabhakar Lad wrote: >>>>>>> To add further more details: >>>>>>> >>>>>>> I am using jailhouse-enabling/5.10 Linux branch [0] with -next branch >>>>>>> for jailhouse [1]. >>>>>>> >>>>>>> I added some debug prints and I see the panic is caused when entry() >>>>>>> function is called (in enter_hypervisor). The entry function lands into >>>>>>> assembly code (entry.S). I dont have a JTAG to see which exact >>>>>>> instruction is causing this issue. >>>>>> >>>>>> So, already the first instruction in the loaded hypervisor binary is not >>>>>> executable? That would explain why we see no hypervisor output at all. >>>>>> >>>>> To clarify when the hypervisor is loaded the output will be via >>>>> debug_console specified in the root cell config? >>>>> >>>> >>>> Correct - if you reach entry() in setup.c. >>>> >>>>>> Was that memory region properly reserved from Linux (via DTB carve-out >>>>>> e.g.)? >>>>>> >>>>> Yes I have the below memory reserved in my dts: >>>>> >>>>> memory@48000000 { >>>>> device_type = "memory"; >>>>> /* first 128MB is reserved for secure area. */ >>>>> reg = <0x0 0x48000000 0x0 0x78000000>; >>>>> }; >>>>> >>>>> reserved-memory { >>>>> #address-cells = <2>; >>>>> #size-cells = <2>; >>>>> ranges; >>>>> >>>>> jh_inmate@a7f00000 { >>>>> status = "okay"; >>>>> no-map; >>>>> reg = <0x00 0xa7f00000 0x00 0xf000000>; >>>>> }; >>>>> >>>>> jailhouse: jailhouse@b6f00000 { >>>>> status = "okay"; >>>>> reg = <0x0 0xb6f00000 0x0 0x1000000>; >>>>> no-map; >>>>> }; >>>>> }; >>>>> >>>>> Linux does report the hole in RAM: >>>>> >>>>> root@smarc-rzg2l:~# cat /proc/iomem | grep RAM >>>>> 48000000-a7efffff : System RAM >>>>> b7f00000-bfffffff : System RAM >>>>> >>>>> And below is my root cell config related to hypervisor memory: >>>>> >>>>> .hypervisor_memory = { >>>>> .phys_start = 0xb6f00000, >>>>> .size = 0x1000000, >>>>> }, >>>>> >>>>> Is there anything obvious I have missed above? >>>>> >>>> >>>> Nothing obvious to me so far. >>>> >>>> So, is this really the first instruction in >>>> hypervisor/arch/arm64/entry.S, arch_entry, that triggers the undefinstr? >>>> Check the reported address by Linux against the disassembly of the >>>> hypervisor. You could also play with adding 'ret' as first instruction, >>>> to check if that returns without a crash. >>>> >>> I did play around with ret, below is my observation: >>> >>> Up until line 98 [0] just before calling arm_dcaches_flush adding ret >>> returned without a crash. Adding a ret at line 99 [1] ie after >>> arm_dcaches_flush it never returned. >>> >>> So I continued with adding ret in arm_dcaches_flush, I added ret as a >>> first statement in arm_dcaches_flush and was able to see the panic! >> >> Which Jailhouse revision are you building? Already disassembled >> hypervisor.o around arch_entry and arm_dcaches_flush? This is what I >> have here for next: >> > I'm using the next branch too. > >> 0000ffffc0203efc <arm_dcaches_flush>: >> ffffc0203efc: d53b0024 mrs x4, ctr_el0 >> ffffc0203f00: d3504c84 ubfx x4, x4, #16, #4 >> ... >> >> 0000ffffc0204800 <arch_entry>: >> ffffc0204800: aa0003f0 mov x16, x0 >> ffffc0204804: aa1e03f1 mov x17, x30 >> ... >> ffffc0204844: d2800042 mov x2, #0x2 >> // #2 >> ffffc0204848: 97fffdad bl ffffc0203efc >> <arm_dcaches_flush> >> >> You could check if there is such a relative call in your case as well. > yes it does exist, below is the snippet: > > 0000ffffc0204000 <arch_entry>: > ffffc0204000: aa0003f0 mov x16, x0 > ffffc0204004: aa1e03f1 mov x17, x30 > ffffc0204008: 10fdffc0 adr x0, ffffc0200000 <hypervisor_header> > ffffc020400c: f9402412 ldr x18, [x0, #72] > ffffc0204010: 5800fd0f ldr x15, ffffc0205fb0 <sdei_handler+0x2c> > ffffc0204014: 900000e1 adrp x1, ffffc0220000 <__page_pool> > ffffc0204018: 79406002 ldrh w2, [x0, #48] > ffffc020401c: d2880003 mov x3, #0x4000 > // #16384 > ffffc0204020: 9b030441 madd x1, x2, x3, x1 > ffffc0204024: f842c02e ldur x14, [x1, #44] > ffffc0204028: 5800fc8d ldr x13, ffffc0205fb8 <sdei_handler+0x34> > ffffc020402c: f840c02c ldur x12, [x1, #12] > ffffc0204030: cb0d018b sub x11, x12, x13 > ffffc0204034: 924051c1 and x1, x14, #0x1fffff > ffffc0204038: 8b0101ef add x15, x15, x1 > ffffc020403c: f9001c0f str x15, [x0, #56] > ffffc0204040: f9400401 ldr x1, [x0, #8] > ffffc0204044: d2800042 mov x2, #0x2 // #2 > ffffc0204048: 97ffff6c bl ffffc0203df8 <arm_dcaches_flush> > ffffc020404c: 5800fba1 ldr x1, ffffc0205fc0 <sdei_handler+0x3c> > ffffc0204050: 8b0b0021 add x1, x1, x11 > ffffc0204054: d2800000 mov x0, #0x0 // #0 > ffffc0204058: f100025f cmp x18, #0x0 > ffffc020405c: 54000041 b.ne ffffc0204064 > <arch_entry+0x64> // b.any > ffffc0204060: d2800020 mov x0, #0x1 // #1 > ffffc0204064: d4000002 hvc #0x0 > ffffc0204068: d4000002 hvc #0x0 > ffffc020406c: 14000000 b ffffc020406c <arch_entry+0x6c> > >> Then you could check, before jumping into arch_entry from the >> hypervisor, that the opcode at the actual arm_dcaches_flush address is >> as expected. But maybe already the building injects an issue here. >> > Any pointers on how I could do that? >
arm_dcaches_flush is loaded at arch_entry (header->entry + hypervisor_mem) - 0x208. Read the u32 at that address and check if it is what is in your hypervisor.o (likely also d53b0024). If that is the case, not the jump but that "mrs x4, ctr_el0" may actually be the problem. Check out hypervisor/arch/arm64/caches.S and see if that code, specifically dcache_line_size, causes trouble outside of Jailhouse as well, maybe as inline assembly in the driver module. Jan -- Siemens AG, Technology Competence Center Embedded Linux -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/5ea8c025-13f2-4754-479a-21395de9306d%40siemens.com.
