On 2014-09-05, 13:03, "Paolo Bonzini" <pbonz...@redhat.com> wrote: >Il 05/09/2014 17:51, Richard Bilson ha scritto: >> Greetings folks, >> >> I've been tracking down certain obscure failures that we see running our >> kernel on qemu x86 smp. Based on a few days diving into qemu internals >>it >> seems that the problems relate to mixing PIC and APIC-generated >> interrupts. I'd appreciate some help in understanding if this is a >> limitation of qemu, or if we are misconfiguring the system in some way. >> >> Scenario: >> - All hardware interrupts come via the PIC, and are delivered by the >>cpu 0 >> LAPIC in ExtINT mode >> - IPIs are delivered by the LAPIC in fixed mode >> >> Failure mode #1: >> - IPI sent to cpu 0 (bit set in APIC irr) >> - IPI accepted by cpu 0 (bit cleared in irr, set in isr) >> - IPI sent to cpu 0 (bit set in both irr and isr) >> - PIC interrupt sent to cpu 0 >> >> The PIC interrupt causes CPU_INTERRUPT_HARD to be set, but >> apic_irq_pending observes that the highest pending APIC interrupt >>priority >> (the IPI) is the same as the processor priority (since the IPI is still >> being handled), so apic_get_interrupt returns a spurious interrupt >>rather >> than the pending PIC interrupt. The result is an endless sequence of >> spurious interrupts, since nothing will clear CPU_INTERRUPT_HARD. > >ExtINT interrupts should have ignored the processor priority, right?
That's my understanding, yes. >> Failure mode #2: >> - cpu 0 masks a particular PIC interrupt >> - IPI sent to cpu 0 (CPU_INTERRUPT_HARD is set) >> - before the IPI is accepted, the masked interrupt line is asserted by >>the >> device >> >> Since the interrupt is masked, apic_deliver_pic_intr will clear >> CPU_INTERRUPT_HARD. The IPI will still be set in the APIC irr, but since >> CPU_INTERRUPT_HARD is not set the cpu will not notice. Depending on the >> scenario this can cause a system hang, i.e. if cpu 0 is expected to >>unmask >> the interrupt. >> >> The commonality here seems to be that the APIC code is attempting to use >> the CPU_INTERRUPT_HARD state for both PIC and APIC interrupts, and one >>of >> these purposes blocks the other. It seems that we should be able to >>solve >> the problem on our side by using a pure-APIC interrupt routing (and >>indeed >> we haven't observed these problems when so configured), but we've never >> noticed these kinds of problems on real hardware using legacy PIC mode. >> >> Any help in confirming or refuting the above analysis is appreciated. > >It sounds plausible. Can you confirm that you are using TCG >(emulation)? Does it work with KVM? What about KVM with -machine >kernel_irqchip=off? I am definitely using TCG. I will try the KVM variants asap. >It would be great if you could reproduce the issue using the harness at >http://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git (it works just >as well without KVM). I'll look into this as well.
