On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <[email protected]> wrote: > > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit : > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote: > > > This series lets call_rcu() callbacks be reclaimed as soon as either a > > > normal or an expedited grace period that covers them has elapsed, rather > > > than always waiting for a normal grace period. > > > > > > Motivation > > > ========== > > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast > > > reclaim, but call_rcu() callers never benefit from those same expedited > > > grace periods, even though an expedited GP proves exactly the same thing > > > as a normal one -- all pre-existing readers are done. When expedited GPs > > > are running on the system (driven by other subsystems), call_rcu() > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until > > > the next normal GP. This series treats a grace period as a grace period > > > regardless of how it was driven, so memory is reclaimed sooner. > > > > > > Design > > > ====== > > > Callback segments now record both the normal and expedited grace-period > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a > > > segment as soon as poll_state_synchronize_rcu_full() reports that either > > > has completed. Three notification paths are taught about expedited > > > completion so the advance actually happens: the NOCB rcuog kthreads, > > > the rcu_pending() tick gate, and rcu_core(). > > > > > > Changelog: > > > RFC: > > > https://lore.kernel.org/all/[email protected]/ > > > Changes in v1: > > > - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq > > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic). > > > - The rcu_segcblist segment field stays named gp_seq; only its type > > > changes (Frederic). > > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue > > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf > > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on), > > > and the wait condition only checked the normal ->gp_seq. The rcuog > > > grace-period wait now tracks a struct rcu_gp_seq and is released via > > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf > > > node through the new rcu_nocb_exp_cleanup() (Frederic). > > > - rcu_pending() uses a new memory-ordering-free > > > poll_state_synchronize_rcu_full_unordered() to avoid memory barriers > > > on every tick, leaving the ordering duty to rcu_core() (Frederic). > > > > > > Still open: Frederic asked whether the first smp_mb() in > > > poll_state_synchronize_rcu_full() is needed on the callback-advance path > > > (patch 6). That path still uses the fully ordered helper; only > > > rcu_pending() was switched to the unordered variant. Happy to revisit. > > > > > > Puranjay Mohan (11): > > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq > > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions > > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper > > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq > > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited > > > GPs > > > rcu: Enable RCU callbacks to benefit from expedited grace periods > > > rcu: Update comments for gp_seq and expedited GP tracking > > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion > > > rcu: Detect expedited grace period completion in rcu_pending() > > > rcu: Advance callbacks for expedited GP completion in rcu_core() > > > rcuscale: Add concurrent expedited GP threads for callback scaling > > > tests > > > > I do see the occasional failure when running TREE01, as in a failure or > > four every couple hundred hours of testing. This is the only rcutorture > > scenario that exercises callback (de)offloading, which might (or might > > not) be a clue. > > > > The console log of a failing run is attached, or you can find it here: > > > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing > > > > This is a "??? Writer stall state" failure, where rcutorture never learns > > of grace-period completion. Possibly because no grace period completed. > > But there are no RCU CPU stall warnings. In addition, these two lines > > > > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state > > D cpu 1 > > . . . > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state > > D cpu 1 > > > > show that the grace-period sequence number advanced from 152813 to > > 736160 in about 50 minutes. So grace periods really are completing, > > lots of them. > > > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble: > > > > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116 > > .W120(148380/168068).N295B101 q516 S CPU 16 > > . . . > > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116 > > .W120(148380/168068).N295B101 q516 S CPU 16 > > > > There are 120 callbacks on the WAIT segment, which are waiting on > > grace-period sequence number 148380, which was long gone even back at > > 631.032657 seconds in (g152813). There are 295 more callbacks in the NEXT > > segment, which long ago should have been assigned a grace period and moved > > into the currently empty NEXT_READY segment (which is the "." right before > > the "N295"). There are yet another 101 callbacks on the bypass list. > > > > This suggests that the rcuog kthread isn't being awakened when it > > should be. And that something is failing to advance callbacks out of > > NEXT in a timely fashion. > > > > Thoughts? > > If a grace period has elapsed while in the middle of nocb_gp_wait(), > after the call to rcu_advance_cbs() and before the following polling which > determines if we a grace period must be waited upon, then the kthread > will wait for new callbacks only (and will ignore waiting callbacks that > can now be advanced). > > Does the following help? > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h > index 6da1b8f52476..96d76eb6259d 100644 > --- a/kernel/rcu/tree_nocb.h > +++ b/kernel/rcu/tree_nocb.h > @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct > rcu_data *my_rdp) > > RCU_NEXT_READY_TAIL); > raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */ > } > - // Need to wait on some grace period? > + > WARN_ON_ONCE(wasempty && > !rcu_segcblist_restempty(&rdp->cblist, > RCU_NEXT_READY_TAIL)); > - /* > - * Only request a GP wait if the next pending callback's > - * GP has not already completed (normal or expedited). > - * If poll_state_synchronize_rcu_full() says it completed, > - * then rcu_advance_cbs() above already moved those > - * callbacks to RCU_DONE_TAIL, so there is no GP to wait > - * for. Any remaining callbacks got new (future) GP > - * numbers from rcu_accelerate_cbs() inside > - * rcu_advance_cbs() and will be handled on the next pass. > - */ > - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) && > - !poll_state_synchronize_rcu_full(&cur_gp_seq)) { > + // Need to wait on some grace period? > + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) { > /* > * Track the earliest pending normal and expedited GP > * across the group so the wait below can be released > by
This looks correct to me, this optimization with !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as you correctly explained. Thanks, Puranjay
