On 3/5/20 7:14 PM, Jan Kiszka wrote: > On 05.03.20 18:26, Andrej Utz wrote: >> Hello everyone, >> >> On 27.02.20 15:46, Ralf Ramsauer wrote: >>> Hi, >>> >>> On 27/02/2020 15:24, [email protected] wrote: >>>> Hi all, >>>> >>>> I'd just like to add that I am experiencing the exact same issue as >>>> described by Ralf on an AMD EPYC 7351P. >>> >>> aah, 'good' to hear! >>> >>>> The stacktrace is the same as well: >>>> >>>> FATAL: Setting invalid LVT delivery mode (reg 35, value 00000700 >>>> >>>> However, on my side I cannot get past this by just handing over the >>>> xAPIC enabled; I get the same trace.. >>>> I added the printk's from Ralf's diff and get the same output: >>>> ... >>>> Before disabling: 10000 >>>> After disabling: 0 >>>> After reenabling: 0 >>> >>> Great. So we definitely have a systematic hardware bug that doesn't only >>> affect our CPU. >>> >>> ... Did I already post my local hacky workaround? >>> >>> diff --git a/hypervisor/arch/x86/apic.c b/hypervisor/arch/x86/apic.c >>> index de691329..7f51b062 100644 >>> --- a/hypervisor/arch/x86/apic.c >>> +++ b/hypervisor/arch/x86/apic.c >>> @@ -340,7 +340,7 @@ void apic_clear(void) >>> >>> /* Finally, reset the TPR again and disable the APIC */ >>> apic_ops.write(APIC_REG_TPR, 0); >>> - apic_ops.write(APIC_REG_SVR, 0xff); >>> + //apic_ops.write(APIC_REG_SVR, 0xff); >>> } >>> >>> static bool apic_valid_ipi_mode(u32 lo_val) >>> >>> >>> Maybe we could try to reach out to AMD via some kernel mailing list? >>> >>>> ... >>>> >>>> I presume that by "hand over the xAPIC enabled" you mean disabling the >>>> write to APIC_REG_SVR? That's what I did but it did not do the trick >>>> unfortunately. >>> >>> Yep. Andrej, did we have to adjust anything else? >>> >>> It's been a while that we've been working on that issue, and I don't >>> have the exact details in mind. >>> >>> However, Andrej wanted to pick up the topic again soon. >>> >>> Thanks, >>> >>> Ralf >>> >> >> From what I've gathered, Jailhouse works as expected. The inmates may >> (or may not) require the APIC, so they also are responsible for >> handling possible hardware bugs. >> >> For this specific case the Linux kernel (inmate) needs a workaround >> during APIC initialization. However it already has a ancient (pre-git) >> quirk[1] to delegate interrupts to 8259A in case LVTs are not masked >> after reset. Nowadays this code seems strange, as the masked state is >> required per APIC specification (for Intel and AMD). >> >> Jan, do you know more about this? > > Nope (despite my age).
:-D > > But keep in mind that the state Linux gets on the BSP is not a reset > state. It is what the BIOS left over (provided masking survives > soft-disabling). Ack, I suspect that this is probably the case. > >> >> Interestingly enough this quirk is also the actual trigger of the >> aforementioned "invalid LVT delivery mode" error in Jailhouse. >> >> For now I've appended a patch with a simple workaround for the kernel. >> Hope that helps. >> >> [1]: >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/x86/kernel/apic/apic.c?h=v5.4.24#n1716 >> > > Can define something that could be proposed to the kernel? Something > that assumes LVTx are unmasked after enabling on CPU types we know of > already. At least we could try to tell the story that way and, at this > chance, pull in some AMD kernel folks to comment on it. Had a discussion on that with Andrej earlier this day. The thing is that upstream is probably not affected: Have a look at this comparison [1]. Apparently, the bug only occurs if smp_processor_id is 0 and the value of LVT0 doesn't match. So actually this bug should also occur during a regular boot of bare-metal Linux, but in fact it doesn't. I assume that CPU 0 is either not affected (unlikely), or, as you already mentioned BIOS/bootloader hands over with a matching value. But we didn't investigate it further yet. However, we'll look for a proper patch that can be used as a basis for a discussion with upstream folks. apic.c is not known for frequent changes... Ralf [1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/x86/kernel/apic/apic.c?h=v5.4.24#n1727 -- You received this message because you are subscribed to the Google Groups "Jailhouse" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jailhouse-dev/d9a9866a-23fe-82e8-bf5a-073ad41a00f1%40oth-regensburg.de.
