From: Naman Jain <[email protected]> Sent: Thursday, October 8, 2026 10:06 PM > > On systems with several fast NVMe controllers, completion interrupts can > keep returning to the same CPUs faster than scheduled work can run. Each > handler may drain only a small number of completions, but the combined > interrupt stream can still prevent scheduler and watchdog progress.
See this recent proposal [1] that sounds like it is addressing the same or a similar issue. And there is this [2] more global approach. It's worthwhile to read through the discussion on both threads. I haven't done a detailed comparison of either vs. your proposal. Michael [1] https://lore.kernel.org/linux-nvme/[email protected]/ [2] https://lore.kernel.org/lkml/[email protected]/ > > This series keeps the existing hardirq completion path for normal I/O. > For an I/O queue with a dedicated MSI-X vector, if an interrupt arrives > with a completion pending while need_resched() is set, mask that vector > and hand the queue to irq_poll. The poller drains bounded batches and > returns the queue to interrupt mode as soon as it catches up. > > The completion-path transitions are: > > +-----------------+ > | normal IRQ mode | > +-----------------+ > | > CQE pending and need_resched() > | > v > mask vector, schedule irq_poll > | > v > +--------------------------------+ > +---->| drain up to one poll budget | > | +--------------------------------+ > | | | > | full budget short batch > | | | > | v v > | remain on poll list irq_poll_complete() > | | mark handoff allowance > | generic irq_poll clear poll ownership > | invokes next pass enable vector > | | | > +--------------+ v > +-----------------+ > | normal IRQ mode | > +-----------------+ > > A full-budget return deliberately stays on the irq_poll list. The generic > irq_poll core invokes another bounded pass. Eventually a pass consumes > less than its budget, takes the short-batch path, and restores interrupt > mode. > > An MSI-X message may have been latched while the vector was masked, but > not every MSI-X hierarchy can report pending state. The handoff allowance > therefore does not claim that an interrupt is known to be stale. It lets > exactly one empty interrupt after unmasking return IRQ_HANDLED. If a new > completion is present, the normal completion path handles it. > > The core idea is simple: when a hardirq sees need_resched(), it masks the > interrupt and offloads completion processing to irq_poll. The interrupt is > unmasked once the CQ catches up. Everything else handles queue and > controller state changes while irq_poll is active, because its asynchronous > callback can outlive the hardirq that scheduled it. > > The lifecycle transitions are: > > normal IRQ mode / irq_poll > | > | set PAUSED, join IRQ and irq_poll, mask vector > v > +--------+ > | paused |---- temporary ----> resume after controller is ready > +--------+ clear PAUSED, restore IRQ mode > | > | permanent stop > | clear INIT, balance owned mask, synchronize replay > v > free registered IRQ > > INIT means the queue has an initialized irq_poll instance and a dedicated > MSI-X vector. IRQ_POLL owns completion processing, while MASKED records the > matching disable_irq_nosync(). PAUSED blocks IRQ handling and new admission. > HANDOFF allows one empty interrupt after unmasking. A temporary pause keeps > MASKED set for resume; permanent stop keeps PAUSED set through IRQ removal. > > Admin, shared, legacy, and threaded interrupt paths are unchanged. Explicit > polling never enters irq_poll; it only disables bottom halves while holding > the shared CQ polling lock. The series adds no timer, rate sampler, > workqueue, sysfs knob, module parameter, or IRQ thread. > > Why not use nvme.use_threaded_interrupts=1 instead? > > Threaded mode installs nvme_irq_check() as the primary handler and > nvme_irq() as the IRQ thread. When the primary handler finds a pending > completion, it returns IRQ_WAKE_THREAD, so completion queue draining runs > through a schedulable task rather than the normal hardirq fast path. > > That remains a useful opt-in mode and this series does not change it. On > the affected VM it prevented the lockup, but it also reduced peak > throughput. This series has a narrower goal: retain direct hardirq > completion while the CPU is keeping up, and use bounded irq_poll work > only after the scheduler has already asserted need_resched(). > > The distinction is not that irq_poll is always faster than an IRQ thread. > It is that the normal path is left alone, while the fallback creates a > scheduling boundary only after a reschedule request is observed. > > need_resched() is used here as a best-effort pressure signal, not as an > interrupt-flood detector or an elapsed-time guarantee. It is transient, > and irq_poll may execute one bounded callback before the scheduler runs. > The narrower claim is that, once an NVMe hardirq observes an already > pending reschedule request, it stops unbounded CQ draining in hardirq. > > Previous NVMe attempts made the fallback the normal high-load path. An > all-irq_poll RFC in 2016 reported an 8-10% single-core IOPS loss and a > measurable low-queue-depth latency cost. A hardirq-first, threaded > overflow series in 2019 fixed the Azure lockup, but Azure testing > reported a substantial throughput loss. > > Relevant discussions from the past around this problem: > > > https://lore.kernel.org/ > %2Flinux-nvme%2F1475660534-16681-1-git-send-email- > sagi%40grimberg.me%2F&data=05%7C02%7C%7Cbbdfaf4318614114c57808df25c30 > 4c3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639271191764118549 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > CIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=1 > mVDHNgaLqX48N6F4S5ny1KC76E4Yv6CM4k%2BOI7V3ks%3D&reserved=0 > > https://lore.kernel.org/ > %2Flkml%2F1566281669-48212-1-git-send-email- > longli%40linuxonhyperv.com%2F&data=05%7C02%7C%7Cbbdfaf4318614114c57808 > df25c304c3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639271191764 > 147973%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAu > MDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > &sdata=8LgU81UBs5FTTkkc7zbQo7Gx828IjfJO6Un9aebcdkM%3D&reserved=0 > > https://lore.kernel.org/ > %2Flinux-nvme%2F20191209175622.1964-1- > kbusch%40kernel.org%2F&data=05%7C02%7C%7Cbbdfaf4318614114c57808df25c3 > 04c3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C639271191764169164 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwM > CIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=N > IMpebg%2B3xIyJvnIk7FbUJANUxhUSFmljodgIAqswTY%3D&reserved=0 > > The first patch selects IRQ_POLL. The second makes existing task-context > CQ polling safe against the new softirq user. The third adds the NVMe > queue handoff and lifecycle handling. > > The RFC starts with a per-queue weight of 256, matching the existing > global irq_poll budget. This is a starting point, not a claim that 256 is > the optimal NVMe value. > > Testing included: > > - full x86_64 kernel build and strict checkpatch > - four-controller QEMU fio, full-disable S3, controller reset, and > post-reset I/O > - QEMU lockdep, prove-locking, and atomic-context validation > - ARM64 Hyper-V VM with four 14-queue NVMe data controllers > - natural irq_poll activation with threaded interrupts disabled > - matching default-hardirq, IRQ-poll fallback, and threaded-IRQ comparison > - four controller resets followed by successful reads > > Read-only 4 KiB random-read results follow. "Default hardirq" uses the > unpatched base. The other modes use the same kernel built from this series: > "IRQ-poll fallback" uses the default nvme.use_threaded_interrupts=0, while > "threaded IRQs" uses nvme.use_threaded_interrupts=1. Low-load values are > medians of two 30-second runs. High-load fallback and threaded values are > medians of two and three 90-second runs respectively; the default-hardirq > run locked up before it could complete. > > default IRQ-poll threaded > hardirq fallback IRQs > 1 disk, QD1 > IOPS 24.42K 24.96K 21.24K > mean completion latency 38.94 us 38.17 us 44.60 us > p99 completion latency 90.6 us 90.1 us 94.2 us > p99.9 completion latency 94.7 us 94.7 us 103.9 us > > 4 disks, 128 jobs, QD256 > status lockup (26s) pass pass > IOPS N/A 4.986M 3.165M > bandwidth N/A 19.0 GiB/s 12.1 GiB/s > mean completion latency N/A 6.544 ms 10.332 ms > p99 completion latency N/A 14.483 ms 22.151 ms > p99.9 completion latency N/A 22.544 ms 24.773 ms > > IRQ-poll fallback prevents hardirq starvation while avoiding the 15-37% > IOPS loss and higher latency observed with threaded interrupts, by > activating only under scheduler pressure. > > Naman Jain (3): > nvme-pci: select IRQ_POLL > nvme-pci: make completion queue polling softirq-safe > nvme-pci: defer completions when rescheduling is needed > > drivers/nvme/host/Kconfig | 1 + > drivers/nvme/host/pci.c | 295 > ++++++++++++++++++++++++++++++++++++++++++++-- > 2 files changed, 283 insertions(+), 13 deletions(-) > > base-commit: eea3fef32a9cf36abcb5975a5a594e4135a6b026 > -- > 2.43.0
