On Fri, Oct 9, 2026 at 1:27 PM Luigi Rizzo <[email protected]> wrote:
>
> On Fri, Oct 9, 2026 at 12:17 PM Naman Jain <[email protected]>
> wrote:
> >
> >
> >
> > On 10/9/2026 12:21 PM, Naman Jain wrote:
> > >
> > >
> > > On 10/9/2026 11:24 AM, Michael Kelley wrote:
> > >> From: Naman Jain <[email protected]> Sent: Thursday, October
> > >> 8, 2026 10:06 PM
> > >>>
> > >>> On systems with several fast NVMe controllers, completion interrupts can
> > >>> keep returning to the same CPUs faster than scheduled work can run. Each
> > >>> handler may drain only a small number of completions, but the combined
> > >>> interrupt stream can still prevent scheduler and watchdog progress.
> > >>
> > >> See this recent proposal [1] that sounds like it is addressing the
> > >> same or a
> > >> similar issue. And there is this [2] more global approach. It's
> > >> worthwhile to read
> > >> through the discussion on both threads. I haven't done a detailed
> > >> comparison
> > >> of either vs. your proposal.
> > >>
> > >> Michael
> > >>
> > >> [1] https://lore.kernel.org/linux-nvme/20260818033846.53790-1-
> > >> [email protected]/
> > >> [2] https://lore.kernel.org/lkml/20260819124341.4185621-1-
> > >> [email protected]/
> > >>
> > >
> > >
> > > Thanks for sharing these Michael. I'll check more on these, and try it
> > > out.
> > >
> > > Regards,
> > > Naman
> >
> > ++ authors of these two series, for awareness and if there is some
> > configuration in their patches I should be trying to fix these lockup
> > issues.
>
> I see your thorough analysis below, thanks for the details.
> Is there any reason why you used nvme.use_threaded_interrupts=0 ?
>
> Setting to 1 moves the work to a kernel thread and seems to be the
> best way to prevent too much work in the hardirq and the soft lockup.
>
> I am surprised that GSIM fails to prevent the soft lockup, though:
> under high load the sequence of events (starting from unmoderated)
> should be the following:
>
> 1. HW sends MSIx interrupt, not blocked by anything
> 2. SW calls handle_fasteoi_irq() --> handle_irq_event() --> ... -->
> action->handler() which starts processing
> 3. HW possibly sends another MSIx interrupt
> 4. SW handle_irq_event() completes
> 5. SW irq_start_moderation() starts the timer, __disable_irq() and
> sets IRQD_IRQ_INPROGRESS | IRQD_MODERATED
> 6. if #3 happened, or another HW interrupt comes before the timer expires:
> SW handle_fasteoi_irq() finds IRQD_IRQ_INPROGRESS | IRQD_MODERATED both set,
> and does not call handle_irq_event(), postponing the call
> 7. when the timer expires, the pending interrupt is reinjected.
>
> The above suggests that we should see some pauses between runs of
> action->handler(),
> at least for a single queue per CPU.
>
> Do you have any way to look with perf and bpftrace at the CPU
> processing interrupts,
> to see what keeps it busy (the pause between 6 and 7 should keep it
> well below 100%)
> and try to verify whether the calls to hndle_irq_event() are actually spaced
> as the sequence above describes?
>
> Having multiple interrupts on the same CPU mean that the gap
> between interrupts for one can be used by others, and that may
> definitely make the CPU 100% busy in hardirq.
Following up on the multiple interrupt sources on one CPU, you could
try something like this:
every time an interrupt source comes while the timer is active,
increment a counter.
When the timer expires and the counter is > 0, add as many "gaps" as
there were sources.
This should restore the invariant that every interrupt source gets one
run per moderation interval,
and the adaptive control loop should adjust the value to hit the target
--- a/kernel/irq/irq_moderation.h
+++ b/kernel/irq/irq_moderation.h
@@ struct irq_mod_state {
+ unsigned int kicked;
unsigned int sleep_ns;
--- a/kernel/irq/irq_moderation.c
+++ b/kernel/irq/irq_moderation.c
@@ bool irq_moderation_do_start(struct irq_desc *desc, struct
irq_mod_state *m)
/* We need moderation, start the timer. */
m->timer_set++;
+ m->kicked = 0;
hrtimer_start_range_ns(&m->timer, ns_to_ktime(m->sleep_ns),
slack_ns, HRTIMER_MODE_REL_PINNED_HARD);
+ } else if (READ_ONCE(irq_mod_params.timer_push)) {
+ /* Timer armed: record the kick, timer_callback() will
push it forward. */
+ m->kicked++;
}
@@ static enum hrtimer_restart timer_callback(struct hrtimer *timer)
lockdep_assert_irqs_disabled();
+ /* Push the timer forward by sleep_ns for each kick received
while armed. */
+ if (m->kicked) {
+ hrtimer_forward_now(timer, ns_to_ktime((u64)m->kicked *
m->sleep_ns));
+ m->kicked = 0;
+ return HRTIMER_RESTART;
+ }
+
cheers
luigi
