Hi Jan, On Thu, May 12, 2022 at 6:45 AM Jan Kiszka <[email protected]> wrote: > > On 11.05.22 19:09, Lad, Prabhakar wrote: > > Hi Jan, > > > > On Wed, May 11, 2022 at 4:11 PM Jan Kiszka <[email protected]> wrote: > >> > >> On 11.05.22 13:20, Prabhakar Lad wrote: > >>> To add further more details: > >>> > >>> I am using jailhouse-enabling/5.10 Linux branch [0] with -next branch > >>> for jailhouse [1]. > >>> > >>> I added some debug prints and I see the panic is caused when entry() > >>> function is called (in enter_hypervisor). The entry function lands into > >>> assembly code (entry.S). I dont have a JTAG to see which exact > >>> instruction is causing this issue. > >> > >> So, already the first instruction in the loaded hypervisor binary is not > >> executable? That would explain why we see no hypervisor output at all. > >> > > To clarify when the hypervisor is loaded the output will be via > > debug_console specified in the root cell config? > > > > Correct - if you reach entry() in setup.c. > > >> Was that memory region properly reserved from Linux (via DTB carve-out > >> e.g.)? > >> > > Yes I have the below memory reserved in my dts: > > > > memory@48000000 { > > device_type = "memory"; > > /* first 128MB is reserved for secure area. */ > > reg = <0x0 0x48000000 0x0 0x78000000>; > > }; > > > > reserved-memory { > > #address-cells = <2>; > > #size-cells = <2>; > > ranges; > > > > jh_inmate@a7f00000 { > > status = "okay"; > > no-map; > > reg = <0x00 0xa7f00000 0x00 0xf000000>; > > }; > > > > jailhouse: jailhouse@b6f00000 { > > status = "okay"; > > reg = <0x0 0xb6f00000 0x0 0x1000000>; > > no-map; > > }; > > }; > > > > Linux does report the hole in RAM: > > > > root@smarc-rzg2l:~# cat /proc/iomem | grep RAM > > 48000000-a7efffff : System RAM > > b7f00000-bfffffff : System RAM > > > > And below is my root cell config related to hypervisor memory: > > > > .hypervisor_memory = { > > .phys_start = 0xb6f00000, > > .size = 0x1000000, > > }, > > > > Is there anything obvious I have missed above? > > > > Nothing obvious to me so far. > > So, is this really the first instruction in > hypervisor/arch/arm64/entry.S, arch_entry, that triggers the undefinstr? > Check the reported address by Linux against the disassembly of the > hypervisor. You could also play with adding 'ret' as first instruction, > to check if that returns without a crash. > I did play around with ret, below is my observation:
Up until line 98 [0] just before calling arm_dcaches_flush adding ret returned without a crash. Adding a ret at line 99 [1] ie after arm_dcaches_flush it never returned. So I continued with adding ret in arm_dcaches_flush, I added ret as a first statement in arm_dcaches_flush and was able to see the panic! [ 225.694752] ------------[ cut here ]------------ [ 225.697709] ------------[ cut here ]------------ [ 225.700663] kernel BUG at arch/arm64/kernel/traps.c:407! [ 225.705253] kernel BUG at arch/arm64/kernel/traps.c:407! [ 225.709846] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP [ 225.725867] Modules linked in: jailhouse(O) [ 225.730050] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 5.10.112-cip6+ #10 [ 225.738105] Hardware name: Renesas SMARC EVK based on r9a07g054l2 (DT) [ 225.744606] pstate: 00400085 (nzcv daIf +PAN -UAO -TCO BTYPE=--) [ 225.750597] pc : do_undefinstr+0x26c/0x320 [ 225.754677] lr : do_undefinstr+0x1cc/0x320 [ 225.758753] sp : ffff800011293cf0 [ 225.762052] x29: ffff800011293cf0 x28: ffff00000989c740 [ 225.767344] x27: ffff00000989c740 x26: ffff800011294000 [ 225.772635] x25: ffff800011290000 x24: ffff8000110fa344 [ 225.777927] x23: 0000000020400085 x22: ffff800012004064 [ 225.783218] x21: ffff800011293ed0 x20: ffff800011293d80 [ 225.788509] x19: ffff800010e4f000 x18: 0000000000000001 [ 225.793800] x17: ffff800008b01870 x16: 0000000000000001 [ 225.799091] x15: ffff800012004064 x14: 000000001004b800 [ 225.804382] x13: 0000ffffc0200000 x12: 00000000b6f00000 [ 225.809673] x11: ffff0000f6d00000 x10: ffff800011293ed0 [ 225.814963] x9 : ffff800011293ed0 x8 : ffff800011293ed0 [ 225.820254] x7 : 0000000000000000 x6 : ffff800011293d48 [ 225.825544] x5 : 00000000d5300000 x4 : ffff800011104f50 [ 225.830835] x3 : 00000000d4000000 x2 : 0000000000000000 [ 225.836125] x1 : ffff00000989c740 x0 : 0000000020400085 [ 225.841417] Call trace: [ 225.843855] do_undefinstr+0x26c/0x320 [ 225.847593] el1_undef+0x30/0x50 [ 225.850808] el1_sync_handler+0xc4/0xe0 [ 225.854626] el1_sync+0x84/0x140 [ 225.857841] 0xffff800012004064 [ 225.860971] flush_smp_call_function_queue+0xf8/0x268 [ 225.866002] generic_smp_call_function_single_interrupt+0x14/0x20 [ 225.872070] ipi_handler+0x8c/0x158 [ 225.875545] handle_percpu_devid_fasteoi_ipi+0x74/0x88 [ 225.880664] generic_handle_irq+0x30/0x48 [ 225.884657] __handle_domain_irq+0x60/0xb8 [ 225.888738] gic_handle_irq+0x58/0x128 [ 225.892470] el1_irq+0xc8/0x180 [ 225.895598] arch_cpu_idle+0x18/0x28 [ 225.899159] default_idle_call+0x24/0x5c [ 225.903065] do_idle+0x1ec/0x288 [ 225.906278] cpu_startup_entry+0x28/0x68 [ 225.910184] secondary_start_kernel+0x1c0/0x208 [ 225.914700] Code: f94013b5 17fffff1 a9025bb5 f9001bb7 (d4210000) [ 225.920774] ---[ end trace 2c06dcfb273cddfd ]--- [ 225.925373] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt [ 225.932738] SMP: stopping secondary CPUs [ 227.020425] SMP: failed to stop secondary CPUs 0-1 [ 227.025194] Kernel Offset: disabled [ 227.028667] CPU features: 0x0040026,2a00a238 [ 227.032915] Memory Limit: none [ 227.035958] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]--- [0] https://github.com/siemens/jailhouse/blob/next/hypervisor/arch/arm64/entry.S#L98 [1] https://github.com/siemens/jailhouse/blob/next/hypervisor/arch/arm64/entry.S#L99 Cheers, Prabhakar -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/CA%2BV-a8t-wvud8MF0oyAsHBSUuVQrrkqSr9RWE2tQLCoOzuQikw%40mail.gmail.com.
