Hi Naman: This is an issue I hadn’t noticed before. I'll take a closer look at this issue later.
Regarding testing, there are a few points to note: 1. My patch has been updated to V2, https://lore.kernel.org/linux-nvme/[email protected]/ 2. In my experience, when do performance tests, it’s best to do precondition first and then execute an ABBA test; this ensures reliable results. 3. IIRC, disabling interrupts in the QEMU kernel does not actually disable hardware interrupts; QEMU still handles them, which has a significant impact on performance testing. Perhaps you could test this on a physical machine. Also, I’d like to ask: does this softlockup issue only occur on QEMU+ARM? or does it also occur on x86 physical machines? Have you tried enabling the “Posted Interrupt” feature in VM? Maybe this will solve this problem in VM. Thanks. 在 2026/10/9 18:16, Naman Jain 写道: > > > On 10/9/2026 12:21 PM, Naman Jain wrote: >> >> >> On 10/9/2026 11:24 AM, Michael Kelley wrote: >>> From: Naman Jain <[email protected]> Sent: Thursday, October 8, >>> 2026 10:06 PM >>>> >>>> On systems with several fast NVMe controllers, completion interrupts can >>>> keep returning to the same CPUs faster than scheduled work can run. Each >>>> handler may drain only a small number of completions, but the combined >>>> interrupt stream can still prevent scheduler and watchdog progress. >>> >>> See this recent proposal [1] that sounds like it is addressing the same or a >>> similar issue. And there is this [2] more global approach. It's worthwhile >>> to read >>> through the discussion on both threads. I haven't done a detailed comparison >>> of either vs. your proposal. >>> >>> Michael >>> >>> [1] https://lore.kernel.org/linux-nvme/20260818033846.53790-1- >>> [email protected]/ >>> [2] https://lore.kernel.org/lkml/20260819124341.4185621-1- >>> [email protected]/ >>> >> >> >> Thanks for sharing these Michael. I'll check more on these, and try it out. >> >> Regards, >> Naman > > ++ authors of these two series, for awareness and if there is some > configuration in their patches I should be trying to fix these lockup issues. > > I tested both GSIM v5 and the NVMe adaptive interrupt polling patch on > the ARM64 Azure system where the NVMe hardirq soft lockup is reproducible. > > Test system > =========== > > The VM has: > > - 128 Arm Neoverse-V2 vCPUs > - two 64-CPU sockets / NUMA nodes > - approximately 862 GiB RAM > - four 3.5-TB Microsoft NVMe Direct Disk v2 data devices > - one NVMe OS device and one additional accelerator-facing NVMe > controller > - Hyper-V vPCI with MSI-X > - 14 I/O queues per data controller > > The four data controllers independently map their I/O vectors to the > same 14 CPUs: > > 0, 10, 19, 28, 37, 46, 55, 64, > 74, 83, 92, 101, 110, 119 > > Thus each of these CPUs handles corresponding queues from all four data > controllers. > > The kernel base for both experiments was: > > next-20261006 > eea3fef32a9cf36abcb5975a5a594e4135a6b026 > 7.3.0-rc6-next-20261006 > > The main lockup workload was read-only: > > - 4-KiB random reads > - libaio > - O_DIRECT > - four data devices > - 128 jobs > - iodepth 256 > - 90-second nominal runtime > > NVMe interrupt coalescing was disabled (FID 0x08 = 0), and > nvme.use_threaded_interrupts was zero. > > Without a mitigation, the watchdog reports soft lockups after about > 26 seconds, normally on all 14 CPUs listed above. > > Conclusion > ========== > > On this VM: > > - GSIM v5 did not prevent the lockup with any tested adaptive or fixed > setting, including the documented benchmark settings and the maximum > allowed delay. > - NVMe adaptive polling can improve throughput for sufficiently dense > individual queues, but it did not prevent the production lockup > because the aggregate cross-controller load was spread across enough > queues that most queues did not meet the fixed 10-us admission > threshold. > > The production failure is triggered by aggregate scheduler starvation > from many queues and controllers sharing the same IRQ CPUs. Neither > generic interrupt-rate moderation nor per-queue throughput-based > adaptation directly observes that condition. > > > More details: > > GSIM v5 > ======= > > I tested the complete seven-patch v5 series: > > https://lore.kernel.org/all/[email protected]/ > > The kernel was built with: > > CONFIG_IRQ_SW_MODERATION=y > CONFIG_IRQ_TIME_ACCOUNTING=y > > GSIM is runtime-disabled and per-IRQ opt-in by default. I identified the > 56 dedicated I/O vectors belonging to the four data controllers. All 56 > were eligible and exposed allow_sw_moderation, and only those vectors > were enabled. > > I first verified the same GSIM-patched kernel with runtime moderation > disabled. It reproduced the soft lockup, as expected. > > I then tested the suggested adaptive configuration from the cover letter: > > delay_us=100 > target_intr_rate=1000000 > hardirq_percent=70 > update_ms=5 > > I also tested the exact adaptive configuration used in patch 6's > benchmark section: > > delay_us=200 > target_intr_rate=1000000 > hardirq_percent=70 > update_ms=5 > > In addition, I tested fixed moderation at: > > 10, 25, 50, 75, 100, 200 and 500 us > > The 500-us value is the maximum allowed by the implementation. > > GSIM was definitely active. For the adaptive 100-us test, on the 14 > affected CPUs it selected delays between about 63 and 100 us, set > 323,811 moderation timers, enqueued 506,557 IRQs, and recorded 24,332 > hardirq-over-threshold events. > > However, every full-duration GSIM-only configuration still soft-locked. > > A summary is: > > QD1 IOPS QD256 outcome > > GSIM off 24.2K soft lockup > adaptive 100 us 4.0K soft lockup > adaptive 200 us 3.3K soft lockup / timeout > fixed 10 us 22.1K soft lockup > fixed 200 us 3.2K soft lockup > fixed 500 us 1.7K soft lockup / timeout > > My interpretation is that GSIM reduces how often the NVMe interrupt > handler runs, but it does not bound how much work the handler performs > once entered. Delaying an interrupt permits more CQEs to accumulate, and > the NVMe hardirq still drains the CQ until empty. On this topology that > produces fewer, larger, still-unbounded hardirq executions. > > This does not contradict the reported GSIM benefits for systems limited > by aggregate MSI-X traffic or PCIe/SoC backpressure. It means that on > this VM the limiting issue is scheduler fairness inside the NVMe > completion handler rather than interrupt-delivery overhead. > > NVMe adaptive interrupt polling > =============================== > > I also tested: > > https://lore.kernel.org/linux-nvme/[email protected]/ > > The posted revision also has the irq_poll full-budget bookkeeping issue > reported in the review thread: it can call irq_poll_complete() and still > return the full budget. Since the author acknowledged this and said it > would be fixed in the next revision, I added only the corresponding > one-line fix before boot testing. I did not boot the known-buggy state. > > The tested adaptive algorithm uses fixed constants: > > - 10-us poll period and admission threshold > - 8,192 CQEs per sample/trial window > - irq_poll budget of 64 CQEs > - 64 successful windows per polling episode > - two immediate failed trials followed by a 524,288-CQE backoff > > I tested the same patched kernel with: > > 1. adaptive policy off > 2. adaptive policy enabled at runtime through > /sys/class/nvme/nvmeX/adaptive_irq_polling > 3. nvme.use_adaptive_irq_polling=1 at boot > > The runtime policy was enabled only for the four data controllers. > > For the production lockup workload, both runtime-on and boot-default-on > still soft-locked at about 26 seconds. > > IRQ_POLL increased by only about 520 callbacks during the entire > runtime-on test and by about 539 during the boot-default-on test. > > The reason appears to be the fixed per-queue admission threshold. With > about 5M aggregate IOPS spread across 56 I/O queues: > > 5M / 56 ~= 89K CQEs/s per queue > average interval ~= 11.2 us > > The adaptive patch attempts polling only when the measured per-queue > completion interval is at most 10 us. The workload is intense in > aggregate, but the individual queues are just below the polling > admission threshold. > > Boot-time enablement produced the same result, so runtime sysfs > switching was not the issue. > > I then tested workloads intended to match the patch's expected dense > per-queue case. > > With one SSD, one job: > > IRQ mode adaptive mode > > QD32 328K IOPS 328K IOPS > QD64 335K IOPS 443K IOPS > QD128 406K IOPS 232K IOPS > > At QD64, adaptive polling was clearly active (about 1.83M IRQ_POLL > callbacks) and improved IOPS by about 32% while reducing latency. This > matches the direction reported by the author. > > The results were not stable across repeats, however. A repeated QD64 > run was approximately neutral. QD128 produced both a regression and an > improvement depending on run order/device state. This appears consistent > with the author's comment that some benchmark results were still > affected by drive-state variability. > > I also tested one SSD with 16 deep jobs pinned to one CPU. Adaptive > polling activated, but averaged about 3% fewer IOPS than IRQ mode. > > Thanks, > Naman
