On Fri, Sep 18, 2026 at 05:32:52PM -0700, Paul E. McKenney wrote: > From: Puranjay Mohan <[email protected]> >
Hi, Sorry for a bit late repsonse. > RCU's per-CPU callback list is only touched with interrupts disabled: the > enqueue runs under local_irq_save() (and the nocb locks when offloaded), > as do callback invocation and grace-period work. A call_rcu() that > arrives with interrupts already disabled, whether from an NMI or from > instrumentation that re-enters RCU, can interrupt one of those and corrupt > the list or deadlock. > > Defer instead: stage the callback on a per-CPU llist and raise an irq_work > that re-issues it once interrupts are on, straight to the enqueue so it > cannot defer again. The gate is bare irqs_disabled(), so callers that > merely hold interrupts off are deferred too and pay one irq_work hop. > Skip it while the scheduler is down (RCU_SCHEDULER_INACTIVE): irq_work is > not usable that early, rcu_init() already calls call_rcu(), and the per-CPU > deferral state is not initialised until rcu_init_one() runs later in it. > > rcu_barrier() drains every CPU's ->defer_head before it scans the lists, > and rcutree_migrate_callbacks() drains an outgoing CPU's. A drain > re-issues onto the draining CPU, so a barrier moves other CPUs' staged > callbacks onto > its own ->cblist; call_rcu() promises no CPU affinity for invocation. > ->defer_lock is held across llist_del_all() and the whole re-issue so the > drainers > serialize: one that finds the list empty can conclude that everything > staged before it is already on a callback list. Interrupts stay off for > the batch. Where the arch has an irq_work self-IPI that is what one > interrupts-disabled region could stage, normally a single callback; where > arch_irq_work_has_interrupt() is false the drain waits for the tick, so > several regions can accumulate first. > > The drain clears ->next before re-issuing. A double call_rcu() on a head > that is already debug-object-active self-links the staged node, and > rcu_do_enqueue()'s duplicate path returns without clearing it, so the > drain would spin. A re-add behind other staged callbacks makes a longer > cycle, which that does not bound; a double call_rcu() stays undefined. > llist_del_all() yields newest-first, so a batch is re-issued in reverse > call order; nothing depends on call_rcu() ordering. The re-issue drops > the lazy hint, since staging records only ->func, so a deferred callback > loses its batching on CONFIG_RCU_LAZY. kasan_record_aux_stack() moves to I'm not sure this is a good idea, because it effectively remove LAZY support when DEFER is enabled. Since the goal of this patchset supports BPF and NMI, would it be nicer that we skip the whole defer logic if the callback is LAZY? Alternatively, you can have two llist (one for hurry and one for lazy). Thoughts? Regards, Boqun > __call_rcu_common() so a use-after-free report names the caller rather > than the irq_work. > > The re-issue runs with interrupts disabled, so instrumentation on the > enqueue path can re-enter call_rcu(), stage another callback and re-raise > the irq_work, livelocking the drain. A per-CPU flag guards it: a deferral > that arrives while this CPU is draining, and is not from an NMI, is > dropped. The WARN_ONCE() is under CONFIG_PROVE_RCU, so a production kernel > drops it silently. That leaks the callback and can strand state > the caller tied to it, since a one-shot flag only the callback clears > never resets, but the alternative is an unbounded loop. > > A callback deferred past the CPUHP_AP_SMPCFD_DYING irq_work flush leaves > ->defer_work claimed with its self-IPI lost. rcutree_migrate_callbacks() > still re-issues the callback, but the first deferral after that CPU comes > back raises no IPI and waits for the next irq_work there, or for > rcu_barrier(). Unqueueing an irq_work is not something the API offers. > > The irq_work is IRQ_WORK_INIT_HARD so the re-issue stays prompt on > PREEMPT_RT, where a non-HARD irq_work runs in a kthread that can be > delayed under load. A hidden CONFIG_RCU_DEFER gates the deferral code and > its IRQ_WORK dependency, though the rcu_data members are unconditional; > without it call_rcu() enqueues directly as before. Under > CONFIG_PROVE_RCU, warn if the direct path is reached from an NMI. > > Suggested-by: Paul E. McKenney <[email protected]> > Signed-off-by: Puranjay Mohan <[email protected]> > Signed-off-by: Paul E. McKenney <[email protected]> > --- [...]
