On 7/4/19 4:39 PM, Jan Kiszka wrote:
> On 04.07.19 15:43, Ralf Ramsauer wrote:
>> Hi,
>>
>> we have some trouble starting non-root Linux on an AMD box. I already
>> tried to narrow things down, but it raised several questions.
>>
>>
>> The main problem is, that non-root Linux tries to write to LVT0, and
>> jailhouse crashes with:
>>
>> FATAL: Setting invalid LVT delivery mode (reg 35, value 00000700)
>>
>>
>> Turns out, in comparison to Intel x86, we don't trap on APIC reads, we
>> only intercept APIC write on AMD (cf. svm.c:338). I thought this would
>> be the issue of this bug, as that's an obvious difference between Intel
>> and AMD: on VMX, we do trap xAPIC reads and writes. However, VMX works
>> slightly different in these regards (side note: [1]).
>>
>> xAPIC reads on AMD systems don't trap the hypervisor, so I intercepted
>> reads (by removing the present bit of the XAPIC_PAGE of the guest), and
>> forwarded the traps to the apic dispatcher (adjusted VMEXIT_NPF).
>>
>> I can confirm that we now trap reads as well as writes. But the non-root
>> Linux still crashes with the same error.
>>
>> Digging a bit deeper, I found out that xAPIC reads are directly
>> forwarded to the hardware, if they were intercepted. So this explains
>> why the bug still remains. This raised another question regarding xAPIC
>> handling on Intel:
>>
>> On AMD, we don't intercept xAPIC reads. On Intel, we do, as we
>> follow the strategy mentioned in [1]… But why?
>
> It accelerates write dispatching at least. I never did the comparison if>
> using a different access scheme would be beneficial because xAPIC is
> practically dead on Intel.
Hmm... The change and benchmark should be pretty easy. Once a bunch of
other issues is solved, I'll maybe have a look at this.
>
>>
>> Wouldn't it be more performant to just trap on xAPIC writes on
>> Intel? This could be done by switching from APIC_ACCESS interception
>> to simple write-only trap & emulate (page faults).
>>
>> However, back to the initial issue. Looks like the difference between
>> Intel and AMD boot is as follows.
>>
>> AMD:
>> [ 0.325578] Switched APIC routing to physical flat.
>> [ 0.366464] enabled ExtINT on CPU#0
>>
>> Intel:
>> [ 0.099486] Switched APIC routing to physical flat.
>> [ 0.113000] masked ExtINT on CPU#0
>>
>>
>> This is why the above-mentioned Jailhouse crash occurs. I tried to find
>> out why Linux takes this decision on AMD. Our victim is in apic.c:1587.
>>
>> On Intel, apic_read(LVT0) & APIC_LVT_MASKED results in 65536, on AMD it
>> is 0. This is why we take a different path.
>>
>> Now the question is simple -- why? :-)
>>
>> Are we just lacking ExtINT delivery mode in Jailhouse, or is anything
>> else odd?
>
> Yes, the ExtINT makes no sense for secondary cells, and it should also
> not be needed for primary ones. Let's dig deeper:
>
> value = apic_read(APIC_LVT0) & APIC_LVT_MASKED;
> if (!cpu && (pic_mode || !value || skip_ioapic_setup)) {
> value = APIC_DM_EXTINT;
> apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n", cpu);
>
> What are the values here, and which are different?
As already mentioned above, only value differs:
>> On Intel, apic_read(LVT0) & APIC_LVT_MASKED results in 65536, on AMD
>> it is 0. This is why we take a different path.
cpu, pic_mode and skip_ioapic_setup is 0 on both machines.
Thanks
Ralf
>
> Jan
>
>>
>> Ralf
>>
>>
>> [1] Regarding Intel VMX, why do we actually need the apic_access_page /
>> SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES dance? On VMX, we do trap on
>> xAPIC reads AND writes. So why do we need virtualisation? Why don't we
>> simply trap on MMIO and handle these situations under
>> EXIT_REASON_EPT_VIOLATION?
>>
>> Is it just to save the dispatching or is there another reason?
>>
>
--
You received this message because you are subscribed to the Google Groups
"Jailhouse" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/jailhouse-dev/c28ae172-5eb8-e9d7-b322-36d7016047dc%40oth-regensburg.de.
For more options, visit https://groups.google.com/d/optout.
