Hi Thomas,
I was wondering if you got a chance to take a look at this fix?
I had some mail issues recently and they showed up at lore after 2
days. I wasn't sure if you got the original mail, or maybe it didn't
make it.
If you had a different way to fix it, we can try those out.
On Thu, Aug 20, 2020 at 05:42:03PM -0700, Ashok Raj wrote:
> When offlining CPUs, fixup_irqs() migrates all interrupts away from the
> outgoing CPU to an online CPU. It's always possible the device sent an
> interrupt to the previous CPU destination. Pending interrupt bit in IRR in
> LAPIC identifies such interrupts. apic_soft_disable() will not capture any
> new interrupts in IRR. This causes interrupts from device to be lost during
> CPU offline. The issue was found when explicitly setting MSI affinity to a
> CPU and immediately offlining it. It was simple to recreate with a USB
> ethernet device and doing I/O to it while the CPU is offlined. Lost
> interrupts happen even when Interrupt Remapping is enabled.
>
> Current code does apic_soft_disable() before migrating interrupts.
>
> native_cpu_disable()
> {
> ...
> apic_soft_disable();
> cpu_disable_common();
> --> fixup_irqs(); // Too late to capture anything in IRR.
> }
>
> Just flipping the above call sequence seems to hit the IRR checks
> and the lost interrupt is fixed for both legacy MSI and when
> interrupt remapping is enabled.
On another note, we have tested both with and without the read
after write when programming MSI addr/data on the device. It didn't
seem to change the results. But I think its a useful one to add
for correctness.
https://lore.kernel.org/lkml/[email protected]/
This bug been eluding for a while. Looking for your feedback.
>
> Fixes: 60dcaad5736f ("x86/hotplug: Silence APIC and NMI when CPU is dead")
> Link: https://lore.kernel.org/lkml/[email protected]/
> Reported-by: Evan Green <[email protected]>
> Tested-by: Mathias Nyman <[email protected]>
> Tested-by: Evan Green <[email protected]>
> Reviewed-by: Evan Green <[email protected]>
> Signed-off-by: Ashok Raj <[email protected]>
> ---
> v2:
> - Typos and fixes suggested by Randy Dunlap
>
> To: [email protected]
> To: Thomas Gleixner <[email protected]>
> Cc: Sukumar Ghorai <[email protected]>
> Cc: Srikanth Nandamuri <[email protected]>
> Cc: Evan Green <[email protected]>
> Cc: Mathias Nyman <[email protected]>
> Cc: Bjorn Helgaas <[email protected]>
> Cc: [email protected]
> ---
> arch/x86/kernel/smpboot.c | 11 +++++++++--
> 1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
> index 27aa04a95702..3016c3b627ce 100644
> --- a/arch/x86/kernel/smpboot.c
> +++ b/arch/x86/kernel/smpboot.c
> @@ -1594,13 +1594,20 @@ int native_cpu_disable(void)
> if (ret)
> return ret;
>
> + cpu_disable_common();
> /*
> * Disable the local APIC. Otherwise IPI broadcasts will reach
> * it. It still responds normally to INIT, NMI, SMI, and SIPI
> - * messages.
> + * messages. It's important to do apic_soft_disable() after
> + * fixup_irqs(), because fixup_irqs() called from cpu_disable_common()
> + * depends on IRR being set. After apic_soft_disable() CPU preserves
> + * currently set IRR/ISR but new interrupts will not set IRR.
> + * This causes interrupts sent to outgoing CPU before completion
> + * of IRQ migration to be lost. Check SDM Vol 3 "10.4.7.2 Local
> + * APIC State after It Has been Software Disabled" section for more
> + * details.
> */
> apic_soft_disable();
> - cpu_disable_common();
>
> return 0;
> }
> --
> 2.7.4
>
