On Mon, Aug 10, 2026 at 05:27:49AM -0700, Puranjay Mohan wrote:
> call_rcu() and call_srcu() only ever touch their per-CPU callback lists
> with interrupts disabled: the enqueue runs under local_irq_save() (and the
> nocb locks when offloaded), and so do callback invocation and grace-period
> work. That is fine as long as call_rcu() itself is invoked with
> interrupts enabled, but it is not always. An NMI handler can call
> call_rcu(), and instrumentation can reenter it. The case that prompted
> this is a BPF program attached to rcu_segcblist_enqueue() that frees an
> object: the free reaches call_rcu_tasks_trace(), which is call_srcu()
> under the hood, back on the same CPU with the srcu_data lock already held,
> and it deadlocks on that lock. Either way, enqueuing directly can corrupt
> the list or deadlock.
Queued for testing and further review, thank you!
Thanx, Paul
> Rather than scatter context checks through the enqueue, make it defer
> whenever interrupts are disabled: stage the callback on a per-CPU lockless
> list and re-issue it from an irq_work once interrupts are back on, going
> straight to the enqueue helper so the re-issue cannot defer again. Only
> the drain side takes a lock; the staging is a bare llist_add() and stays
> safe from NMI. This is behind a new hidden CONFIG_RCU_DEFER, which is set
> wherever a reentrant enqueue is possible (HAVE_NMI, KPROBES,
> FUNCTION_TRACER or TRACEPOINTS); without it call_rcu() enqueues exactly as
> before. The gate is bare irqs_disabled(), so callers that merely hold
> interrupts off are deferred too and pay one irq_work hop.
>
> CPU offline is the awkward part. A callback can be deferred very late in
> the outgoing CPU's teardown -- from do_idle() or cpuhp_ap_report_dead(),
> past the CPUHP_AP_SMPCFD_DYING flush that would otherwise run the irq_work
> -- so the irq_work can no longer run there to re-issue it. rcu_barrier()
> and srcu_barrier() therefore drain every CPU's deferred list themselves
> before they wait. They drain rather than wait the irq_work out because
> irq_work_sync() parks on an rcuwait, which holds a single waiter, so two
> concurrent barriers would clobber each other's wakeup.
> rcutree_migrate_callbacks() drains the outgoing CPU's list
> too, so a late deferral still lands on a callback list even when nobody
> calls a barrier. To keep those drainers from stepping on each other, the
> drain holds a per-CPU raw lock across the llist_del_all() and the
> re-issue, so a drainer never returns having pulled callbacks off the
> deferred list but not yet put them on a callback list. Every lock the
> re-issue touches (nocb, rcu_node, srcu_data) is already raw, so the
> nesting is fine.
>
> The drain re-issues with interrupts disabled, so instrumentation on the
> enqueue path can re-enter call_rcu()/call_srcu() from inside it, stage
> another callback, re-raise the irq_work, and the drain never finishes. A
> per-CPU flag catches that: a deferral that arrives while that CPU is
> inside its own irq_work drain, and is not from an NMI, is dropped rather
> than staged, with a WARN_ONCE() under CONFIG_PROVE_RCU whose backtrace
> names the instrumentation responsible. Dropping leaks that callback, and
> can strand state its caller tied to it, but the alternative is a CPU that
> never leaves the drain, and the producer is a BPF program that emits one
> callback per enqueue, so there is nothing finite to wait for. Only the
> irq_work drain sets the flag: a direct drain from a barrier or from
> CPU-offline re-issues onto the current CPU, so anything staged during it
> is picked up by that CPU's own irq_work rather than feeding the drain in
> progress, and no legitimate callback is dropped. Instrumenting the
> irq_work machinery itself can still loop, as it can for any irq_work user,
> and is not something this series can fix.
>
> The irq_work is IRQ_WORK_INIT_HARD in all four flavors. It is not needed
> for correctness, but a non-HARD irq_work runs from a kthread on
> PREEMPT_RT and can be delayed under load, letting deferred callbacks pile
> up; running the re-issue in hard-irq context keeps that from turning into
> an OOM.
>
> Patches 1 and 2 do Tree and Tiny RCU, 3 and 4 Tree and Tiny SRCU. Patch 5
> teaches rcutorture to issue ->call() from a perf-overflow NMI -- the
> nmi_calls parameter, on by default -- on the flavors that advertise it,
> and checks that every callback issued from NMI is later invoked. Patch 6
> adds the BPF reentry reproducer described above.
>
> Changelog:
> v3: https://lore.kernel.org/rcu/[email protected]/
> Changes in v4:
> - rcu_barrier() and srcu_barrier() no longer skip a CPU whose deferred
> list looks empty. The lockless llist_empty() test added in v3 skipped
> the lock that makes a drain conclusive: a concurrent drainer can already
> have emptied the list without having re-issued yet, so the barrier could
> return before those callbacks reached a callback list.
> - Tree SRCU: drain before cleanup_srcu_struct()'s "just leak it" early
> returns, and in srcu_module_going() before it frees any ->sda. Deferral
> skips check_init_srcu_struct(), so srcu_module_going() could skip
> cleanup_srcu_struct() and free ->sda with a staged srcu_data still
> chained on a per-CPU list.
> - Dropped ->defer_exp: a deferred expedited call_srcu() now completes as a
> normal grace period. The flag lived on the srcu_data rather than on the
> callback, so it expedited whole batches; only srcu_expedite_current()
> with interrupts already disabled can reach this, so the downgrade costs
> a delay and nothing else.
> - rcutorture: use a fixed perf sample period rather than .freq, which sets
> TICK_DEP_BIT_PERF_EVENTS for every CPU and pinned the tick for whole
> runs of TREE04 and TREE07. Also document nmi_calls, and say on the
> console when there is no PMU, since the issued==invoked check then
> compares zero to zero.
> - kasan_record_aux_stack() moved to __call_rcu_common(), so a
> use-after-free report names the caller rather than the irq_work.
> - Known gap, documented in patch 1: a callback deferred past the
> CPUHP_AP_SMPCFD_DYING flush leaves ->defer_work claimed with its
> self-IPI lost. rcutree_migrate_callbacks() still re-issues the
> callback, but the first deferral after that CPU comes back raises no IPI
> and waits for the next irq_work there, or for rcu_barrier().
>
> v2: https://lore.kernel.org/rcu/[email protected]/
> Changes in v3:
> - Barriers no longer call irq_work_sync() on an online CPU's ->defer_work.
> irq_work_sync() waits on an rcuwait, which holds exactly one task, so two
> concurrent rcu_barrier()s syncing the same per-CPU irq_work could lose a
> wakeup and hang. Both flushes now drain every CPU directly, guarded by a
> lockless llist_empty() test so the no-deferrals case stays cheap.
> - The re-entry guard is now set only by the irq_work drain, which always
> runs on the CPU owning the list it drains. In v2 a barrier draining a
> remote CPU set the flag on the draining CPU, so an unrelated irqs-off
> call_rcu() there was dropped and leaked even though it could not have fed
> the drain.
> - The drain clears ->next before re-issuing. A double call_rcu() on a head
> that is already debug-object-active makes llist_add() self-link it, and
> rcu_do_enqueue()'s double-free path returns without clearing ->next, so
> the drain span looped forever with interrupts disabled.
> - An expedited call_srcu() is no longer silently downgraded: ->defer_exp
> records it per srcu_data and the batch is re-issued expedited. Only
> srcu_expedite_current() is affected, __synchronize_srcu() sleeps and so
> is never deferred.
> - The drop is now WARN_ONCE() under CONFIG_PROVE_RCU rather than an
> unconditional WARN, so instrumentation cannot reboot a panic_on_warn
> kernel; the backtrace is what identifies the offending program.
> - Tiny RCU and Tiny SRCU use IRQ_WORK_INIT_HARD like the Tree flavors, and
> READ_ONCE()/WRITE_ONCE() on their draining flags. TINY_SRCU is
> "default y if !SMP" with no PREEMPT_RT dependency, so it really can be
> built on RT where a non-HARD irq_work waits on the irq_workd kthread.
> - Tiny SRCU: cleanup_srcu_struct() drains *and* irq_work_sync()s
> ->defer_iw. That irq_work is embedded in the srcu_struct the caller is
> about to free, unlike the Tree flavors' static per-CPU ones.
> - rcutorture: drive the perf counters from CPU-hotplug callbacks. The
> one-shot for_each_online_cpu() loop lost them at the first CPU offline,
> after which the end-of-test issued==invoked check compared 0 == 0.
> - rcutorture: per-CPU rcu_head instead of one global, so several CPUs can
> race the drain; count the call before issuing it so mid-run stats cannot
> show nmi-cbs > nmi-calls; release the perf events on the
> torture_cleanup_begin() early-return path; print nmi_calls in the module
> banner; report when nmi_calls is set but no NMI ->call() ever happened;
> and drop sample_freq to 100, since 1000 made perf lower the system-wide
> perf_event_max_sample_rate tenfold.
> - selftests/bpf: the old ASSERT_EQ(reentered, 1) could not fail, and an
> atomic allocation failure in the nested task-storage delete made the test
> pass without ever re-entering call_srcu(). It now records and asserts
> the helper return values, probes for rcu_segcblist_enqueue() up front so
> a Tiny kernel skips instead of failing, and restores the CPU affinity it
> changes.
>
> Testing: rcutorture rcu/srcu/srcud/tasks-tracing with nmi_calls, hotplug,
> barriers and nocb toggling issued ~81000 callbacks from NMI with none lost,
> under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING, DEBUG_OBJECTS_RCU_HEAD and
> RCU_LAZY, with no lockdep reports; SRCU-T, SRCU-U, TINY01 and TINY02 pass;
> the BPF reproducer passes.
>
> v1: https://lore.kernel.org/all/[email protected]/
> Changes in v2:
> - Fixed the re-entry livelock Zqiang spotted: a BPF program on the enqueue
> path re-enters call_srcu() from inside srcu_defer_drain(), stages another
> callback and re-raises the irq_work, so the drain never finishes. A
> per-CPU flag now drops such a deferral, with a warning, unless it comes
> from an NMI.
> - cleanup_srcu_struct(): drain the deferred callbacks before syncing
> ->irq_work rather than after, since re-issuing one can start a grace
> period and re-queue that irq_work (Zqiang).
> - Tiny SRCU: sync ->defer_iw in cleanup_srcu_struct() as well, so a
> deferred callback is re-issued onto ->srcu_cb_head where the leak checks
> can see it instead of being stranded on a soon-to-be-freed srcu_struct.
> - Added Kumar's ack to the BPF selftest patch.
>
> Testing: rcutorture rcu, srcu, srcud and tasks-tracing, each with
> nmi_calls, CPU hotplug and barriers, and nocb toggling for rcu, all
> End of test: SUCCESS with the count issued from NMI equal to the count
> invoked (17987, 2924, 2661 and 4412 respectively) and clean consoles,
> under PROVE_LOCKING, PROVE_RAW_LOCK_NESTING and DEBUG_OBJECTS_RCU_HEAD;
> SRCU-T, SRCU-U, TINY01 and TINY02 pass; the BPF reproducer passes.
> Build-tested x86_64 with TREE_RCU+TREE_SRCU, TINY_RCU+TINY_SRCU and
> TREE_RCU+TINY_SRCU, each with CONFIG_RCU_DEFER=y and =n, plus arm64
> TREE_RCU+TREE_SRCU with RCU_NOCB_CPU and RCU_LAZY.
>
> Puranjay Mohan (6):
> rcu: Make call_rcu() safe to call from any context
> rcu: Make Tiny call_rcu() safe to call from any context
> srcu: Make call_srcu() safe to call from any context
> srcu: Make Tiny call_srcu() safe to call from any context
> rcutorture: Exercise ->call() from NMI context
> selftests/bpf: Add a call_srcu() re-entry reproducer
>
> .../admin-guide/kernel-parameters.txt | 7 +
> include/linux/srcutiny.h | 12 +-
> include/linux/srcutree.h | 4 +
> kernel/rcu/Kconfig | 6 +
> kernel/rcu/rcu.h | 14 ++
> kernel/rcu/rcutorture.c | 150 +++++++++++++++-
> kernel/rcu/srcutiny.c | 93 +++++++++-
> kernel/rcu/srcutree.c | 170 +++++++++++++++++-
> kernel/rcu/tiny.c | 127 ++++++++++---
> kernel/rcu/tree.c | 133 ++++++++++++--
> kernel/rcu/tree.h | 6 +
> .../selftests/bpf/prog_tests/rcu_reentry.c | 93 ++++++++++
> .../testing/selftests/bpf/progs/rcu_reentry.c | 51 ++++++
> 13 files changed, 818 insertions(+), 48 deletions(-)
> create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_reentry.c
> create mode 100644 tools/testing/selftests/bpf/progs/rcu_reentry.c
>
> base-commit: 9dc303e69bcd49f9668ca090ae45325269531fbb
> --
> 2.53.0-Meta
>
