On Fri, Jul 24, 2026 at 07:06:42AM -0700, Paul E. McKenney wrote:
> On Fri, Jul 24, 2026 at 02:47:22PM +0100, Puranjay Mohan wrote:
> > On Fri, Jul 24, 2026 at 1:52 PM Puranjay Mohan <[email protected]> wrote:
> > >
> > > On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <[email protected]>
> > > wrote:
> > > >
> > > > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > > > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > > > > This series lets call_rcu() callbacks be reclaimed as soon as
> > > > > > either a
> > > > > > normal or an expedited grace period that covers them has elapsed,
> > > > > > rather
> > > > > > than always waiting for a normal grace period.
> > > > > >
> > > > > > Motivation
> > > > > > ==========
> > > > > > Today there is an asymmetry: synchronize_rcu_expedited() callers
> > > > > > get fast
> > > > > > reclaim, but call_rcu() callers never benefit from those same
> > > > > > expedited
> > > > > > grace periods, even though an expedited GP proves exactly the same
> > > > > > thing
> > > > > > as a normal one -- all pre-existing readers are done. When
> > > > > > expedited GPs
> > > > > > are running on the system (driven by other subsystems), call_rcu()
> > > > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL
> > > > > > until
> > > > > > the next normal GP. This series treats a grace period as a grace
> > > > > > period
> > > > > > regardless of how it was driven, so memory is reclaimed sooner.
> > > > > >
> > > > > > Design
> > > > > > ======
> > > > > > Callback segments now record both the normal and expedited
> > > > > > grace-period
> > > > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases
> > > > > > a
> > > > > > segment as soon as poll_state_synchronize_rcu_full() reports that
> > > > > > either
> > > > > > has completed. Three notification paths are taught about expedited
> > > > > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > > > > the rcu_pending() tick gate, and rcu_core().
> > > > > >
> > > > > > Changelog:
> > > > > > RFC:
> > > > > > https://lore.kernel.org/all/[email protected]/
> > > > > > Changes in v1:
> > > > > > - New prep patch 1 renames struct rcu_gp_oldstate to struct
> > > > > > rcu_gp_seq
> > > > > > and its fields rgos_norm/rgos_exp to norm/exp tree-wide
> > > > > > (Frederic).
> > > > > > - The rcu_segcblist segment field stays named gp_seq; only its type
> > > > > > changes (Frederic).
> > > > > > - Patch 8 (NOCB wake) is reworked. v1 woke the wrong waitqueue
> > > > > > (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > > > > rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps
> > > > > > on),
> > > > > > and the wait condition only checked the normal ->gp_seq. The
> > > > > > rcuog
> > > > > > grace-period wait now tracks a struct rcu_gp_seq and is released
> > > > > > via
> > > > > > poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the
> > > > > > leaf
> > > > > > node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > > > > - rcu_pending() uses a new memory-ordering-free
> > > > > > poll_state_synchronize_rcu_full_unordered() to avoid memory
> > > > > > barriers
> > > > > > on every tick, leaving the ordering duty to rcu_core()
> > > > > > (Frederic).
> > > > > >
> > > > > > Still open: Frederic asked whether the first smp_mb() in
> > > > > > poll_state_synchronize_rcu_full() is needed on the callback-advance
> > > > > > path
> > > > > > (patch 6). That path still uses the fully ordered helper; only
> > > > > > rcu_pending() was switched to the unordered variant. Happy to
> > > > > > revisit.
> > > > > >
> > > > > > Puranjay Mohan (11):
> > > > > > rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > > > > rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > > > > rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > > > > rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > > > > rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without
> > > > > > expedited
> > > > > > GPs
> > > > > > rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > > > > rcu: Update comments for gp_seq and expedited GP tracking
> > > > > > rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > > > > rcu: Detect expedited grace period completion in rcu_pending()
> > > > > > rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > > > > rcuscale: Add concurrent expedited GP threads for callback scaling
> > > > > > tests
> > > > >
> > > > > I do see the occasional failure when running TREE01, as in a failure
> > > > > or
> > > > > four every couple hundred hours of testing. This is the only
> > > > > rcutorture
> > > > > scenario that exercises callback (de)offloading, which might (or might
> > > > > not) be a clue.
> > > > >
> > > > > The console log of a failing run is attached, or you can find it here:
> > > > >
> > > > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> > > > >
> > > > > This is a "??? Writer stall state" failure, where rcutorture never
> > > > > learns
> > > > > of grace-period completion. Possibly because no grace period
> > > > > completed.
> > > > > But there are no RCU CPU stall warnings. In addition, these two lines
> > > > >
> > > > > [ 631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0
> > > > > ->state D cpu 1
> > > > > . . .
> > > > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0
> > > > > ->state D cpu 1
> > > > >
> > > > > show that the grace-period sequence number advanced from 152813 to
> > > > > 736160 in about 50 minutes. So grace periods really are completing,
> > > > > lots of them.
> > > > >
> > > > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> > > > >
> > > > > [ 631.066109] rcu: CB 5^4->4 KblSW F21044 L21044 C116
> > > > > .W120(148380/168068).N295B101 q516 S CPU 16
> > > > > . . .
> > > > > [ 3610.869914] rcu: CB 5^4->4 KblSW F3000848 L3000848 C116
> > > > > .W120(148380/168068).N295B101 q516 S CPU 16
> > > > >
> > > > > There are 120 callbacks on the WAIT segment, which are waiting on
> > > > > grace-period sequence number 148380, which was long gone even back at
> > > > > 631.032657 seconds in (g152813). There are 295 more callbacks in the
> > > > > NEXT
> > > > > segment, which long ago should have been assigned a grace period and
> > > > > moved
> > > > > into the currently empty NEXT_READY segment (which is the "." right
> > > > > before
> > > > > the "N295"). There are yet another 101 callbacks on the bypass list.
> > > > >
> > > > > This suggests that the rcuog kthread isn't being awakened when it
> > > > > should be. And that something is failing to advance callbacks out of
> > > > > NEXT in a timely fashion.
> > > > >
> > > > > Thoughts?
> > > >
> > > > If a grace period has elapsed while in the middle of nocb_gp_wait(),
> > > > after the call to rcu_advance_cbs() and before the following polling
> > > > which
> > > > determines if we a grace period must be waited upon, then the kthread
> > > > will wait for new callbacks only (and will ignore waiting callbacks that
> > > > can now be advanced).
> > > >
> > > > Does the following help?
> > > >
> > > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> > > > index 6da1b8f52476..96d76eb6259d 100644
> > > > --- a/kernel/rcu/tree_nocb.h
> > > > +++ b/kernel/rcu/tree_nocb.h
> > > > @@ -792,22 +792,12 @@ static noinline_for_stack void
> > > > nocb_gp_wait(struct rcu_data *my_rdp)
> > > >
> > > > RCU_NEXT_READY_TAIL);
> > > > raw_spin_unlock_rcu_node(rnp); /* irqs
> > > > disabled. */
> > > > }
> > > > - // Need to wait on some grace period?
> > > > +
> > > > WARN_ON_ONCE(wasempty &&
> > > > !rcu_segcblist_restempty(&rdp->cblist,
> > > >
> > > > RCU_NEXT_READY_TAIL));
> > > > - /*
> > > > - * Only request a GP wait if the next pending callback's
> > > > - * GP has not already completed (normal or expedited).
> > > > - * If poll_state_synchronize_rcu_full() says it
> > > > completed,
> > > > - * then rcu_advance_cbs() above already moved those
> > > > - * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> > > > - * for. Any remaining callbacks got new (future) GP
> > > > - * numbers from rcu_accelerate_cbs() inside
> > > > - * rcu_advance_cbs() and will be handled on the next
> > > > pass.
> > > > - */
> > > > - if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> > > > - !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> > > > + // Need to wait on some grace period?
> > > > + if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> > > > /*
> > > > * Track the earliest pending normal and
> > > > expedited GP
> > > > * across the group so the wait below can be
> > > > released by
> > >
> > >
> > > This looks correct to me, this optimization with
> > > !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
> > > you correctly explained.
> >
> > My local reproducer says that the problem is gone, but let's see what
> > Paul's 100s of hours of testing says.
>
> I will fire it up, and thank you both!!!
>
> And if it passes the 100s of hours (about 600), it will get 1000s of
> hours over the weekend. ;-)
And error-free!!! (Must be some mistake?)
I am starting an eight-hour run for 1600 hours of TREE01-only test time.
Puranjay is also experimenting with ways of reproducing this faster,
which just might find their way into the normal rcutorture testing. ;-)
I will be running a few longer runs of the full set of scenarios over
the weekend.
Thanx, Paul
