On 13.05.22 09:31, Lad, Prabhakar wrote:
> Hi Jan,
>
> On Thu, May 12, 2022 at 6:05 PM Jan Kiszka <[email protected]> wrote:
>>
>> On 12.05.22 13:37, Lad, Prabhakar wrote:
>>> Hi Jan,
>>>
>>> On Thu, May 12, 2022 at 12:16 PM Jan Kiszka <[email protected]> wrote:
>>>>
>>>> On 12.05.22 13:06, Lad, Prabhakar wrote:
>>>>> Hi Jan,
>>>>>
>>>>> On Thu, May 12, 2022 at 11:24 AM Jan Kiszka <[email protected]>
>>>>> wrote:
>>>>>>
>>>>>> On 12.05.22 09:01, Lad, Prabhakar wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> On Thu, May 12, 2022 at 6:45 AM Jan Kiszka <[email protected]>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On 11.05.22 19:09, Lad, Prabhakar wrote:
>>>>>>>>> Hi Jan,
>>>>>>>>>
>>>>>>>>> On Wed, May 11, 2022 at 4:11 PM Jan Kiszka <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On 11.05.22 13:20, Prabhakar Lad wrote:
>>>>>>>>>>> To add further more details:
>>>>>>>>>>>
>>>>>>>>>>> I am using jailhouse-enabling/5.10 Linux branch [0] with -next
>>>>>>>>>>> branch
>>>>>>>>>>> for jailhouse [1].
>>>>>>>>>>>
>>>>>>>>>>> I added some debug prints and I see the panic is caused when entry()
>>>>>>>>>>> function is called (in enter_hypervisor). The entry function lands
>>>>>>>>>>> into
>>>>>>>>>>> assembly code (entry.S). I dont have a JTAG to see which exact
>>>>>>>>>>> instruction is causing this issue.
>>>>>>>>>>
>>>>>>>>>> So, already the first instruction in the loaded hypervisor binary is
>>>>>>>>>> not
>>>>>>>>>> executable? That would explain why we see no hypervisor output at
>>>>>>>>>> all.
>>>>>>>>>>
>>>>>>>>> To clarify when the hypervisor is loaded the output will be via
>>>>>>>>> debug_console specified in the root cell config?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Correct - if you reach entry() in setup.c.
>>>>>>>>
>>>>>>>>>> Was that memory region properly reserved from Linux (via DTB
>>>>>>>>>> carve-out
>>>>>>>>>> e.g.)?
>>>>>>>>>>
>>>>>>>>> Yes I have the below memory reserved in my dts:
>>>>>>>>>
>>>>>>>>> memory@48000000 {
>>>>>>>>> device_type = "memory";
>>>>>>>>> /* first 128MB is reserved for secure area. */
>>>>>>>>> reg = <0x0 0x48000000 0x0 0x78000000>;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> reserved-memory {
>>>>>>>>> #address-cells = <2>;
>>>>>>>>> #size-cells = <2>;
>>>>>>>>> ranges;
>>>>>>>>>
>>>>>>>>> jh_inmate@a7f00000 {
>>>>>>>>> status = "okay";
>>>>>>>>> no-map;
>>>>>>>>> reg = <0x00 0xa7f00000 0x00 0xf000000>;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> jailhouse: jailhouse@b6f00000 {
>>>>>>>>> status = "okay";
>>>>>>>>> reg = <0x0 0xb6f00000 0x0 0x1000000>;
>>>>>>>>> no-map;
>>>>>>>>> };
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Linux does report the hole in RAM:
>>>>>>>>>
>>>>>>>>> root@smarc-rzg2l:~# cat /proc/iomem | grep RAM
>>>>>>>>> 48000000-a7efffff : System RAM
>>>>>>>>> b7f00000-bfffffff : System RAM
>>>>>>>>>
>>>>>>>>> And below is my root cell config related to hypervisor memory:
>>>>>>>>>
>>>>>>>>> .hypervisor_memory = {
>>>>>>>>> .phys_start = 0xb6f00000,
>>>>>>>>> .size = 0x1000000,
>>>>>>>>> },
>>>>>>>>>
>>>>>>>>> Is there anything obvious I have missed above?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Nothing obvious to me so far.
>>>>>>>>
>>>>>>>> So, is this really the first instruction in
>>>>>>>> hypervisor/arch/arm64/entry.S, arch_entry, that triggers the
>>>>>>>> undefinstr?
>>>>>>>> Check the reported address by Linux against the disassembly of the
>>>>>>>> hypervisor. You could also play with adding 'ret' as first instruction,
>>>>>>>> to check if that returns without a crash.
>>>>>>>>
>>>>>>> I did play around with ret, below is my observation:
>>>>>>>
>>>>>>> Up until line 98 [0] just before calling arm_dcaches_flush adding ret
>>>>>>> returned without a crash. Adding a ret at line 99 [1] ie after
>>>>>>> arm_dcaches_flush it never returned.
>>>>>>>
>>>>>>> So I continued with adding ret in arm_dcaches_flush, I added ret as a
>>>>>>> first statement in arm_dcaches_flush and was able to see the panic!
>>>>>>
>>>>>> Which Jailhouse revision are you building? Already disassembled
>>>>>> hypervisor.o around arch_entry and arm_dcaches_flush? This is what I
>>>>>> have here for next:
>>>>>>
>>>>> I'm using the next branch too.
>>>>>
>>>>>> 0000ffffc0203efc <arm_dcaches_flush>:
>>>>>> ffffc0203efc: d53b0024 mrs x4, ctr_el0
>>>>>> ffffc0203f00: d3504c84 ubfx x4, x4, #16, #4
>>>>>> ...
>>>>>>
>>>>>> 0000ffffc0204800 <arch_entry>:
>>>>>> ffffc0204800: aa0003f0 mov x16, x0
>>>>>> ffffc0204804: aa1e03f1 mov x17, x30
>>>>>> ...
>>>>>> ffffc0204844: d2800042 mov x2, #0x2
>>>>>> // #2
>>>>>> ffffc0204848: 97fffdad bl ffffc0203efc
>>>>>> <arm_dcaches_flush>
>>>>>>
>>>>>> You could check if there is such a relative call in your case as well.
>>>>> yes it does exist, below is the snippet:
>>>>>
>>>>> 0000ffffc0204000 <arch_entry>:
>>>>> ffffc0204000: aa0003f0 mov x16, x0
>>>>> ffffc0204004: aa1e03f1 mov x17, x30
>>>>> ffffc0204008: 10fdffc0 adr x0, ffffc0200000
>>>>> <hypervisor_header>
>>>>> ffffc020400c: f9402412 ldr x18, [x0, #72]
>>>>> ffffc0204010: 5800fd0f ldr x15, ffffc0205fb0
>>>>> <sdei_handler+0x2c>
>>>>> ffffc0204014: 900000e1 adrp x1, ffffc0220000 <__page_pool>
>>>>> ffffc0204018: 79406002 ldrh w2, [x0, #48]
>>>>> ffffc020401c: d2880003 mov x3, #0x4000
>>>>> // #16384
>>>>> ffffc0204020: 9b030441 madd x1, x2, x3, x1
>>>>> ffffc0204024: f842c02e ldur x14, [x1, #44]
>>>>> ffffc0204028: 5800fc8d ldr x13, ffffc0205fb8
>>>>> <sdei_handler+0x34>
>>>>> ffffc020402c: f840c02c ldur x12, [x1, #12]
>>>>> ffffc0204030: cb0d018b sub x11, x12, x13
>>>>> ffffc0204034: 924051c1 and x1, x14, #0x1fffff
>>>>> ffffc0204038: 8b0101ef add x15, x15, x1
>>>>> ffffc020403c: f9001c0f str x15, [x0, #56]
>>>>> ffffc0204040: f9400401 ldr x1, [x0, #8]
>>>>> ffffc0204044: d2800042 mov x2, #0x2
>>>>> // #2
>>>>> ffffc0204048: 97ffff6c bl ffffc0203df8 <arm_dcaches_flush>
>>>>> ffffc020404c: 5800fba1 ldr x1, ffffc0205fc0
>>>>> <sdei_handler+0x3c>
>>>>> ffffc0204050: 8b0b0021 add x1, x1, x11
>>>>> ffffc0204054: d2800000 mov x0, #0x0
>>>>> // #0
>>>>> ffffc0204058: f100025f cmp x18, #0x0
>>>>> ffffc020405c: 54000041 b.ne ffffc0204064
>>>>> <arch_entry+0x64> // b.any
>>>>> ffffc0204060: d2800020 mov x0, #0x1
>>>>> // #1
>>>>> ffffc0204064: d4000002 hvc #0x0
>>>>> ffffc0204068: d4000002 hvc #0x0
>>>>> ffffc020406c: 14000000 b ffffc020406c <arch_entry+0x6c>
>>>>>
>>>>>> Then you could check, before jumping into arch_entry from the
>>>>>> hypervisor, that the opcode at the actual arm_dcaches_flush address is
>>>>>> as expected. But maybe already the building injects an issue here.
>>>>>>
>>>>> Any pointers on how I could do that?
>>>>>
>>>>
>>>> arm_dcaches_flush is loaded at arch_entry (header->entry +
>>>> hypervisor_mem) - 0x208. Read the u32 at that address and check if it is
>>>> what is in your hypervisor.o (likely also d53b0024).
>>>>
>>>> If that is the case, not the jump but that "mrs x4, ctr_el0" may
>>>> actually be the problem. Check out hypervisor/arch/arm64/caches.S and
>>>> see if that code, specifically dcache_line_size, causes trouble outside
>>>> of Jailhouse as well, maybe as inline assembly in the driver module.
>>>>
>>>
>>> With the below ret added, I get "JAILHOUSE_ENABLE: Success"
>>>
>>> diff --git a/hypervisor/arch/arm64/entry.S b/hypervisor/arch/arm64/entry.S
>>> index a9cabf7f..4e98b292 100644
>>> --- a/hypervisor/arch/arm64/entry.S
>>> +++ b/hypervisor/arch/arm64/entry.S
>>> @@ -96,6 +96,7 @@ arch_entry:
>>> */
>>> ldr x1, [x0, #HEADER_CORE_SIZE]
>>> mov x2, DCACHE_CLEAN_AND_INVALIDATE_ASM
>>> + ret
>>> bl arm_dcaches_flush
>>>
>>> /* install bootstrap_vectors */
>>>
>>> Now when I undo the above and do the below, Im seeing a panic:
>>>
>>> diff --git a/hypervisor/arch/arm64/caches.S b/hypervisor/arch/arm64/caches.S
>>> index 39dad4af..ce29b8e8 100644
>>> --- a/hypervisor/arch/arm64/caches.S
>>> +++ b/hypervisor/arch/arm64/caches.S
>>> @@ -40,6 +40,7 @@
>>> */
>>> .global arm_dcaches_flush
>>> arm_dcaches_flush:
>>> + ret
>>> dcache_line_size x3, x4
>>> add x1, x0, x1
>>> sub x4, x3, #1
>>>
>>> Issue is seen even without dcache_line_size being called. Does that
>>> mean we are not landing in arm_dcaches_flush?
>>
>> Likely. I've never seen such an effect.
>>
>> If you look the reported fault address, when making it relative
>> (subtract hypervisor_mem), is that arm_dcaches_flush (relative to
>> arch_entry)?
>>
> Could you please elaborate on it more. I moved the cache.S code in
> entry.S file but still seeing the same issue.
$ aarch64-linux-gnu-objdump -x hypervisor/hypervisor.o | \
grep "arch_entry\|arm_dcaches_flush"
0000ffffc0203efc g .text 0000000000000000 arm_dcaches_flush
0000ffffc0204800 g .text 0000000000000000 arch_entry
-> delta of 0x904 here
diff --git a/driver/main.c b/driver/main.c
index 64e2b9a4..cb197d77 100644
--- a/driver/main.c
+++ b/driver/main.c
@@ -246,6 +246,8 @@ static void enter_hypervisor(void *info)
entry = header->entry + (unsigned long) hypervisor_mem;
+ printk("obcode @arm_dcaches_flush: %08x\n", *(u32 *)(entry - 0x904));
+
if (cpu < header->max_cpus)
/* either returns 0 or the same error code across all CPUs */
err = entry(cpu);
Untested, though.
>
> In some of the reference platforms LPAE is enabled in the u-boot. Is
> that a strict requirement? Also in the requirement section it's
LPAE is 32-bit arm, your are on 64-bit, no?
> mentioned "Linux is started in HYP mode" does that mean Before loading
> the jailhouse the Linux should be running on EL2? Also to be sure, do
> we need any special configs enabled in TF-A at all?
You need Linux to start in HYP mode so that Linux installs a stub that
KVM (when not using Jailhouse) and Jailhouse can use to take over the
hypervisor role. But your init crashes before arch_entry is able to
issue the related hvc instructions.
TF-A needs to be there in order to have PSCI. Special settings are
usually only related to SDEI, which is optional.
>
> Fyi, I am using arm64_defconfig_5.10 [0] (+ additional configs to
> enable my platform) to build the Linux kernel, should these configs be
> sufficient for Jailhouse?
Yes, at least for the various targets we cover with this so far.
Jan
>
> [0]
> https://github.com/siemens/jailhouse-images/blob/next/recipes-kernel/linux/files/arm64_defconfig_5.10
>
> Cheers,
> Prabhakar
--
Siemens AG, Technology
Competence Center Embedded Linux
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/e0aaf1b9-700d-ba22-270d-049ac10be44c%40siemens.com.
