On 10/9/2026 8:28 PM, Luigi Rizzo wrote:
On Fri, Oct 9, 2026 at 1:27 PM Luigi Rizzo <[email protected]> wrote:
On Fri, Oct 9, 2026 at 12:17 PM Naman Jain <[email protected]> wrote:
On 10/9/2026 12:21 PM, Naman Jain wrote:
On 10/9/2026 11:24 AM, Michael Kelley wrote:
From: Naman Jain <[email protected]> Sent: Thursday, October
8, 2026 10:06 PM
On systems with several fast NVMe controllers, completion interrupts can
keep returning to the same CPUs faster than scheduled work can run. Each
handler may drain only a small number of completions, but the combined
interrupt stream can still prevent scheduler and watchdog progress.
See this recent proposal [1] that sounds like it is addressing the
same or a
similar issue. And there is this [2] more global approach. It's
worthwhile to read
through the discussion on both threads. I haven't done a detailed
comparison
of either vs. your proposal.
Michael
[1] https://lore.kernel.org/linux-nvme/20260818033846.53790-1-
[email protected]/
[2] https://lore.kernel.org/lkml/20260819124341.4185621-1-
[email protected]/
Thanks for sharing these Michael. I'll check more on these, and try it out.
Regards,
Naman
++ authors of these two series, for awareness and if there is some
configuration in their patches I should be trying to fix these lockup
issues.
Hi Luigi,
Thanks for looking into it.
I see your thorough analysis below, thanks for the details.
Is there any reason why you used nvme.use_threaded_interrupts=0 ?
Setting to 1 moves the work to a kernel thread and seems to be the
best way to prevent too much work in the hardirq and the soft lockup.
I used 0 because the goal was to preserve the existing hardirq fast path
and avoid the throughput cost we had observed with threaded interrupts.
I have both modes on the same kernel image. Threaded interrupts did
prevent the lockup, but at QD256 they reached 3.165M IOPS versus 4.986M
with the IRQ-poll fallback. At QD1 they reached 21.2K versus 25.0K IOPS.
So threaded interrupts are a valid workaround, but on this system they
cost about 15% at QD1 and 37% at QD256 compared with the fallback.
More details in my initial cover letter.
https://lore.kernel.org/all/[email protected]/
I am surprised that GSIM fails to prevent the soft lockup, though:
under high load the sequence of events (starting from unmoderated)
should be the following:
1. HW sends MSIx interrupt, not blocked by anything
2. SW calls handle_fasteoi_irq() --> handle_irq_event() --> ... -->
action->handler() which starts processing
3. HW possibly sends another MSIx interrupt
4. SW handle_irq_event() completes
5. SW irq_start_moderation() starts the timer, __disable_irq() and
sets IRQD_IRQ_INPROGRESS | IRQD_MODERATED
6. if #3 happened, or another HW interrupt comes before the timer expires:
SW handle_fasteoi_irq() finds IRQD_IRQ_INPROGRESS | IRQD_MODERATED both set,
and does not call handle_irq_event(), postponing the call
7. when the timer expires, the pending interrupt is reinjected.
The above suggests that we should see some pauses between runs of
action->handler(),
at least for a single queue per CPU.
Do you have any way to look with perf and bpftrace at the CPU
processing interrupts,
to see what keeps it busy (the pause between 6 and 7 should keep it
well below 100%)
and try to verify whether the calls to hndle_irq_event() are actually spaced
as the sequence above describes?
Having multiple interrupts on the same CPU mean that the gap
between interrupts for one can be used by others, and that may
definitely make the CPU 100% busy in hardirq.
GSIM was creating gaps for individual sources, but the other NVMe
sources sharing that CPU were filling most of them.
I traced irq_handler_entry/irq_handler_exit with trace-cmd for the four
NVMe queue vectors routed to CPU0. The test used GSIM fixed at 10 us
with all 56 data-NVMe I/O vectors opted in.
Without timer push, I recorded 37,218 NVMe handler invocations in three
seconds. The median handler duration was about 2 us, but the median gap
between NVMe handlers was only 4 us. The p99 gap was 872 us.
With timer push enabled, the same three-second CPU0 trace contained only
531 NVMe handler invocations, and the median gap increased from 4 us to
517 us. This confirms your diagnosis, adding one delay per additional
source restores a real CPU-wide gap.
The full QD256 workload then completed without a soft lockup or kernel
taint. There was a significant cost, though. The delayed completions
accumulated into much larger CQ batches:
- median NVMe handler duration increased from 2 us to 3.1 ms
- p99 handler duration increased from 106 us to 7.1 ms
For comparison on the same workload:
fixed 10-us GSIM + timer push: 1.906M IOPS
adaptive GSIM + timer push: 1.712M IOPS
threaded NVMe IRQs: 3.165M IOPS
NVMe IRQ-poll fallback: 4.986M IOPS
So the timer-push idea does prevent the lockup, and the trace confirms
the multi-source gap issue. On this NVMe topology, however, delaying all
sources creates very large completion batches, and the NVMe handler
still drains each batch without a bound in hardirq context. That causes
the large throughput and tail-latency cost above.
Regards,
Naman
