First, about the setup. It's a FreeBSD/amd64 head guest on a FreeBSD/amd64 head
host. The hardware is AMD. The hypervisor is bhyve.
Under a certain specific load, that involves a lot of page faults and IPI-s, I
see the guest system getting stuck. This is pretty consistent. Typically I
find a thread spinning on smp_ipi_mtx. And an owner of the mutex appears to be
in mi_switch() -> sched_switch(). The debugging data that I have is somewhat
flaky, but it seems that the owner is typically in this code path:
smp_targeted_tlb_shootdown -> ipi_send_cpu -> native_lapic_ipi_raw
smp_targeted_tlb_shootdown holds smp_ipi_mtx.
native_lapic_ipi_raw, in this setup, performs the following manipulations:
saveintr = intr_disable();
The interrupts are already disabled when this function is entered, because
smp_ipi_mtx is a spinlock and our spinlock implementation disables interrupts.
So, intr_restore() in this case should be a NOP (BTW, it's implemented via
But what I see suggests that at this point a Local APIC timer interrupt gets
delivered to the thread. And that causes all the mess as the thread holding the
spinlock gets preempted.
Does this ring a bell to anyone?
Is there any suspect code?
It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be
completely virtualized. So, maybe a hardware issue?
firstname.lastname@example.org mailing list
To unsubscribe, send any mail to