On Thu, Jun 11, 2020 at 12:12:46AM +0200, Frederic Weisbecker wrote:
> On Wed, Jun 10, 2020 at 07:02:10AM -0700, Paul E. McKenney wrote:
> > And just to argue against myself...
> >
> > Another approach is to maintain explicit multiple states for each
> > ->cblist, perhaps something like this:
> >
> > 1. In softirq. Transition code advances to next.
> > 2. To no-CB 1. Either GP or CB kthread for the transitioning
> > CPU advances to next. Note that the fact that the
> > transition code runs on the transitioning CPU means that
> > the RCU softirq handler doesn't need to be involved.
> > 3. To no-CB 2. Either GP or CB kthread for the transitioning
> > CPU advances to next.
>
> Just to clarify, if GP has set NO_CB2 in (2), we want CB to set NO_CB3
> in 3), right? OTOH if CB has set NO_CB2 in (2), we want GP to set NO_CB3
> in (3), right?
>
> The point being to make sure that both threads acknowledge the transition?
Exactly!
> > 4. To no-CB 3. Transitioning code advances to next.
> > At this point, the no-CBs setup is fully functional.
>
> And softirq can stop processing callbacks from that point on.
You got it!
> > 5. No-CB. Transitioning code advances to next.
> > Again, the fact that the transitioning code is running
> > on the transitioning CPU with interrupts disabled means
> > that the RCU softirq handler need not be explicitly
> > involved.
> > 6. To softirq 1. The RCU softirq handler for the transitioning
> > CPU advances to next.
> > At this point, the RCU softirq handler is processing callbacks.
> > 7. To softirq 2. Either GP or CB kthread for the transitioning
> > CPU advances to next.
> > At this point, the softirq handler is processing callbacks.
>
> SOFTIRQ2 should be part of what happens in SOFTIRQ1. The transitioning
> CPU sets SOFTIRQ1, which is immediately visible by local softirqs,
> and wakes up CB/GP. CB/GP sets SOFTIRQ2, CB/GP sets SOFTIRQ3 and
> we go back to transitioning code that sets IN_SOFTIRQ.
>
> Or did I miss something?
I was perhaps being overly paranoid. You might well be right.
> > 8. To softirq 3. Either GP or CB kthread for the transitioning
> > CPU advances to next.
> > At this point, the no-CBs setup is fully shut down.
> > 9. To softirq 4. Transitioning code advances to next,
> > which is the first, "In softirq".
> > (This one -might- be unnecessary, but...)
> >
> > All transitions are of course with the ->nocb_lock held.
> >
> > When there is only one CPU during early boot near rcu_init() time,
> > the transition from "In softirq" to "No-CB" can remain be instantaneous.
> >
> > This has the advantage of not slowing things down just because there
> > is an RCU callback flood in progress. It also uses an explicit
> > protocol that should (give or take bugs) maintain full safety both
> > in protection of ->cblist and in dealing with RCU callback floods.
> >
> > Thoughts?
>
> Agreed. And I really like that it details the whole process in a very
> explicit way.
>
> Thanks!
Glad you like it! And of course please adjust it as needed, up to and
including doing something completely different that works better. ;-)
Thanx, Paul
