On Mon, Oct 12, 2026 at 10:19 AM Naman Jain <[email protected]> wrote: > > > > On 10/9/2026 8:28 PM, Luigi Rizzo wrote: > > On Fri, Oct 9, 2026 at 1:27 PM Luigi Rizzo <[email protected]> wrote: > >> > >> On Fri, Oct 9, 2026 at 12:17 PM Naman Jain <[email protected]> > >> wrote: > >>> > >>> > >>> > >>> On 10/9/2026 12:21 PM, Naman Jain wrote: > >>>> > >>>> > >>>> On 10/9/2026 11:24 AM, Michael Kelley wrote: > >>>>> From: Naman Jain <[email protected]> Sent: Thursday, October > >>>>> 8, 2026 10:06 PM > >>>>>> > >>>>>> On systems with several fast NVMe controllers, completion interrupts > >>>>>> can > >>>>>> keep returning to the same CPUs faster than scheduled work can run. > >>>>>> Each > >>>>>> handler may drain only a small number of completions, but the combined > >>>>>> interrupt stream can still prevent scheduler and watchdog progress. > >>>>> > >>>>> See this recent proposal [1] that sounds like it is addressing the > >>>>> same or a > >>>>> similar issue. And there is this [2] more global approach. It's > >>>>> worthwhile to read > >>>>> through the discussion on both threads. I haven't done a detailed > >>>>> comparison > >>>>> of either vs. your proposal. > >>>>> > >>>>> Michael > >>>>> > >>>>> [1] https://lore.kernel.org/linux-nvme/20260818033846.53790-1- > >>>>> [email protected]/ > >>>>> [2] https://lore.kernel.org/lkml/20260819124341.4185621-1- > >>>>> [email protected]/ > >>>>> > >>>> > >>>> > >>>> Thanks for sharing these Michael. I'll check more on these, and try it > >>>> out. > >>>> > >>>> Regards, > >>>> Naman > >>> > >>> ++ authors of these two series, for awareness and if there is some > >>> configuration in their patches I should be trying to fix these lockup > >>> issues. > >> > > Hi Luigi, > Thanks for looking into it. ... > GSIM was creating gaps for individual sources, but the other NVMe > sources sharing that CPU were filling most of them. > > I traced irq_handler_entry/irq_handler_exit with trace-cmd for the four > NVMe queue vectors routed to CPU0. The test used GSIM fixed at 10 us > with all 56 data-NVMe I/O vectors opted in. > > Without timer push, I recorded 37,218 NVMe handler invocations in three > seconds. The median handler duration was about 2 us, but the median gap > between NVMe handlers was only 4 us. The p99 gap was 872 us.
(medians are not a good metric in an environment where there might be a lot of idle runs because a previous, delayed run already consumed all CQs. You may want to look at the full distribution, I expect to see a step at some point, the exact percentile depends on how many queues share the same CPU) Just a clarification, the pattern I expect on CPU0 is something like this NVME0 NVME1 NVME2 NVME3 TIMER_PUSH NVME0 ... what is "mediam gap between handlers", is it the time between NVME0 and the next NVME0, or between any two NVMEx and NVMEy ? > With timer push enabled, the same three-second CPU0 trace contained only > 531 NVMe handler invocations, and the median gap increased from 4 us to > 517 us. This confirms your diagnosis, adding one delay per additional > source restores a real CPU-wide gap. > > The full QD256 workload then completed without a soft lockup or kernel > taint. There was a significant cost, though. The delayed completions > accumulated into much larger CQ batches: > > - median NVMe handler duration increased from 2 us to 3.1 ms > - p99 handler duration increased from 106 us to 7.1 ms > > For comparison on the same workload: > > fixed 10-us GSIM + timer push: 1.906M IOPS > adaptive GSIM + timer push: 1.712M IOPS > threaded NVMe IRQs: 3.165M IOPS > NVMe IRQ-poll fallback: 4.986M IOPS > > So the timer-push idea does prevent the lockup, and the trace confirms > the multi-source gap issue. On this NVMe topology, however, delaying all > sources creates very large completion batches, and the NVMe handler > still drains each batch without a bound in hardirq context. That causes > the large throughput and tail-latency cost above. Yes, without touching the nvme handler to limit the number of entries processed in each run, there is only so much a generic solution like GSIM can do. This definitely warrants some NVME-specific changes such as those proposed by Fengnan, and they are in line with what NAPI already does by using a bounded batch and rescheduling the handler. I am a bit surprised by your numbers regarding the differences with and without timer push, so I would like to understand the numbers better. (As an aside, 10us is really a bit too short for GSIM delay, because almost 50% of the time gets eaten by the context switching. At least for an initial assessment I would start testing with a fixed 30-50us) Can you clarify the following, specifically for the "full QD256 workload" - how many vectors are sharing one interrupt CPU (the previous test says 4), and how many interrupt CPUs are used in total ? - From my experience (different environments), I have measured < 1 queue - 1 interrupt CPU - 1 user thread > saturating at 400-600KIOPS (I don't remember whether the bottleneck is the user or interrupt CPU). At those rates, 256 outstanding entries may take some 500us to be processed. so it seems strange that the median NVME handler (per queue?) takes 3ms. - these times will be scaled down if each drive uses many queues so the average throughput per queue will be lower, and such to keep the total rate for each interrupt CPU below its 400K cap. My expectations is something along these lines: - the outstanding CQ queue cannot exceed QD ie 256 in your test - worst case each handler should run at most some 500us - four queues will create a total QD=1024 so the interrupt CPU will take at least 2ms to drain them cheers luigi
