On 10/12/2026 2:47 PM, Luigi Rizzo wrote:
On Mon, Oct 12, 2026 at 10:19 AM Naman Jain <[email protected]> wrote:
On 10/9/2026 8:28 PM, Luigi Rizzo wrote:
On Fri, Oct 9, 2026 at 1:27 PM Luigi Rizzo <[email protected]> wrote:
On Fri, Oct 9, 2026 at 12:17 PM Naman Jain <[email protected]> wrote:
On 10/9/2026 12:21 PM, Naman Jain wrote:
On 10/9/2026 11:24 AM, Michael Kelley wrote:
From: Naman Jain <[email protected]> Sent: Thursday, October
8, 2026 10:06 PM
On systems with several fast NVMe controllers, completion interrupts can
keep returning to the same CPUs faster than scheduled work can run. Each
handler may drain only a small number of completions, but the combined
interrupt stream can still prevent scheduler and watchdog progress.
See this recent proposal [1] that sounds like it is addressing the
same or a
similar issue. And there is this [2] more global approach. It's
worthwhile to read
through the discussion on both threads. I haven't done a detailed
comparison
of either vs. your proposal.
Michael
[1] https://lore.kernel.org/linux-nvme/20260818033846.53790-1-
[email protected]/
[2] https://lore.kernel.org/lkml/20260819124341.4185621-1-
[email protected]/
Thanks for sharing these Michael. I'll check more on these, and try it out.
Regards,
Naman
++ authors of these two series, for awareness and if there is some
configuration in their patches I should be trying to fix these lockup
issues.
Hi Luigi,
Thanks for looking into it.
...
GSIM was creating gaps for individual sources, but the other NVMe
sources sharing that CPU were filling most of them.
I traced irq_handler_entry/irq_handler_exit with trace-cmd for the four
NVMe queue vectors routed to CPU0. The test used GSIM fixed at 10 us
with all 56 data-NVMe I/O vectors opted in.
Without timer push, I recorded 37,218 NVMe handler invocations in three
seconds. The median handler duration was about 2 us, but the median gap
between NVMe handlers was only 4 us. The p99 gap was 872 us.
(medians are not a good metric in an environment where there might be
a lot of idle runs because a previous, delayed run already consumed all CQs.
You may want to look at the full distribution, I expect to see a step at some
point, the exact percentile depends on how many queues share the same CPU)
Just a clarification, the pattern I expect on CPU0 is something like this
NVME0 NVME1 NVME2 NVME3 TIMER_PUSH NVME0 ...
what is "mediam gap between handlers", is it the time between NVME0 and
the next NVME0, or between any two NVMEx and NVMEy ?
The earlier term was ambiguous. I now use two separate definitions:
any-source gap:
previous NVMe handler exit to the next NVMe handler entry for any of
the four CPU0 NVMe vectors, i.e. NVMEx -> NVMEy
same-source recurrence:
previous exit to the next entry of the same vector, e.g.
NVME0 -> next NVME0
The earlier 4-us push-off and 517-us push-on medians were the any-source
gaps, not same-source recurrence.
I reran the tests, and the distribution confirms two effects:
1. Without push, many handler runs remain very short, but different NVMe
sources can run almost back-to-back.
2. With push, the CPU gets much larger gaps, but CQ work accumulates and
the replayed NVMe hardirq commonly runs for several milliseconds.
The multi source pattern you mentioned, is correct.
NVME0 NVME1 NVME2 NVME3 TIMER_PUSH NVME0 ...
With timer push enabled, the same three-second CPU0 trace contained only
531 NVMe handler invocations, and the median gap increased from 4 us to
517 us. This confirms your diagnosis, adding one delay per additional
source restores a real CPU-wide gap.
The full QD256 workload then completed without a soft lockup or kernel
taint. There was a significant cost, though. The delayed completions
accumulated into much larger CQ batches:
- median NVMe handler duration increased from 2 us to 3.1 ms
- p99 handler duration increased from 106 us to 7.1 ms
For comparison on the same workload:
fixed 10-us GSIM + timer push: 1.906M IOPS
adaptive GSIM + timer push: 1.712M IOPS
threaded NVMe IRQs: 3.165M IOPS
NVMe IRQ-poll fallback: 4.986M IOPS
So the timer-push idea does prevent the lockup, and the trace confirms
the multi-source gap issue. On this NVMe topology, however, delaying all
sources creates very large completion batches, and the NVMe handler
still drains each batch without a bound in hardirq context. That causes
the large throughput and tail-latency cost above.
Yes, without touching the nvme handler to limit the number
of entries processed in each run, there is only so much
a generic solution like GSIM can do.
This definitely warrants some NVME-specific changes
such as those proposed by Fengnan, and they are in line
with what NAPI already does by using a bounded batch
and rescheduling the handler.
Agreed. Timer-push GSIM changes when the NVMe hardirq runs, but it does
not move NVMe completion processing out of hardirq or bound the amount
drained after replay, which is required to avoid lockups.
I am a bit surprised by your numbers regarding the differences with
and without timer push, so I would like to understand
the numbers better.
(As an aside, 10us is really a bit too short for GSIM delay, because
almost 50% of the time gets eaten by the context switching.
At least for an initial assessment I would start testing with a fixed 30-50us)
The push-off/push-on IRQ trace was an earlier short experiment with
fixed 10-us GSIM:
push off push on
short-run IOPS 2.164M 1.944M
CPU0 handlers / 3 seconds 37,219 531
eventual full-test lockup yes no
Those short-run IOPS are not completed 90-second results, as all the
long duration timer push off experiments eventually soft-locked.
The latest controlled three-run matrix used timer_push=1 for every GSIM row.
Can you clarify the following, specifically for the "full QD256 workload"
- how many vectors are sharing one interrupt CPU (the previous test says 4),
and how many interrupt CPUs are used in total ?
There are four data NVMe controllers, each with 14 interrupt-driven I/O
queues.
Corresponding queue vectors from the four controllers are routed to the
same CPU. Therefore:
vectors per affected CPU: 4
affected interrupt CPUs: 14
total data-NVMe I/O vectors: 56
The affected CPUs are:
0, 10, 19, 28, 37, 46, 55,
64, 74, 83, 92, 101, 110, 119
The matched unmodified control reproduced soft lockups on exactly those
14 CPUs. The first watchdog report occurred 25 seconds after fio launch.
The production fio command is equivalent to:
--filename=dev1:dev2:dev3:dev4
--numjobs=128
--iodepth=256
--group_reporting
so the potential total is:
128 x 256 = 32,768 requests
The NVMe hardware queue depth is 1,024, with 1,023 usable entries.
I tested fixed 30 us and 50 us as you suggested.
Every passing mode was tested three times, with a fresh boot before each
sample. The workload uses a 10-second ramp followed by a 90-second
measured runtime.
NVMe fallback: this series
threaded NVMe IRQs: Default kernel with nvme.use_threaded_interrupts=1
High-load results are medians of three runs, you can find complete
minimum-to-maximum range:
IOPS median complete range
NVMe fallback 5.004M 5.002-5.004M
threaded NVMe IRQs 3.140M 3.100-3.169M
GSIM fixed 10 us + push 1.962M 1.935-2.025M
GSIM fixed 30 us + push 1.790M 1.775-1.822M
GSIM fixed 50 us + push 1.744M 1.736-1.746M
Completion latency:
mean p99 p99.9
NVMe fallback 6.533 ms 14.352 ms 16.908 ms
threaded NVMe IRQs 10.421 ms 22.151 ms 24.510 ms
GSIM fixed 10 us + push 16.570 ms 38.011 ms 270.533 ms
GSIM fixed 30 us + push 18.283 ms 40.108 ms 492.831 ms
GSIM fixed 50 us + push 18.743 ms 40.632 ms 541.065 ms
Relative to the fallback median, threaded IRQs are 37.24% lower, while
fixed 10/30/50-us timer-push GSIM is 60.78%/64.22%/65.15% lower.
QD1 results were:
IOPS median complete range
unmodified hardirq 25,518 one run
NVMe fallback 25,520 23,212-25,552
threaded NVMe IRQs 22,253 22,035-22,466
GSIM fixed 10 us + push 21,080 21,032-21,931
GSIM fixed 30 us + push 5,496 5,259-5,608
GSIM fixed 50 us + push 5,492 5,224-5,589
The 30-us and 50-us settings on this VM, do not improve high-load
performance relative to fixed 10 us and they reduce QD1 performance
substantially.
I also traced fixed 30 us plus timer push on CPU0:
handler count: 550 in three seconds
handler busy fraction: 53.05%
handler duration p50: 2.888 ms
handler duration p90: 5.187 ms
handler duration p99: 6.655 ms
handler duration p99.9: 7.461 ms
handler duration maximum: 7.805 ms
any-source gap p50: 439 us
any-source gap p90: 8.073 ms
any-source gap p99: 13.319 ms
same-source recurrence p50: 18.782 ms
same-source recurrence p99: 35.551 ms
- From my experience (different environments), I have measured
< 1 queue - 1 interrupt CPU - 1 user thread > saturating at 400-600KIOPS
(I don't remember whether the bottleneck is the user or interrupt CPU).
At those rates, 256 outstanding entries may take some 500us to be processed.
so it seems strange that the median NVME handler (per queue?) takes 3ms.
- these times will be scaled down if each drive uses many queues so the
average throughput per queue will be lower, and such to keep the total
rate for each interrupt CPU below its 400K cap.
My expectations is something along these lines:
- the outstanding CQ queue cannot exceed QD ie 256 in your test
- worst case each handler should run at most some 500us
- four queues will create a total QD=1024 so the interrupt CPU
will take at least 2ms to drain them
At approximately 5M aggregate IOPS, 56 queues (4 devices), an
even-distribution estimate is:
5,000,000 / 56
= approximately 89,286 completions/s/queue
or:
1,000,000 us / 89,286
= approximately 11.2 us/completion/queue
The measured queue occupancy is more important here i.e. despite the
lower average throughput per hardware queue, timer-push batching allowed
all four traced queues to reach 1,023 outstanding commands.
I also ran a separate experiment matching a true total outstanding depth
of 256:
four jobs
one job per disk
iodepth=64 per job
all jobs pinned to CPU0
Therefore: 4 x 64 = 256 total potential outstanding requests
With fixed 30-us GSIM:
push off push on
total IOPS 15.52K 15.22K
handler duration p50 3 us 3 us
handler duration p99 5 us 5 us
any-source gap p50 267 us 270 us
Timer push had negligible effect in this sparse case because the
workload already had large gaps.
Fixed 30-us moderation heavily throttled that control, so I repeated it
without GSIM moderation. It reached 263.6K IOPS total. Handler duration was:
p50: 1 us
p99: 4 us
p99.9: 5 us
maximum: 106 us
Therefore, your expectation of short handlers applies when total
outstanding depth really is 256. The millisecond handlers in the
production reproducer come from the much deeper 128-job workload and the
accumulation caused by timer push.
Regards,
Naman
