On Thu, Nov 27, 2025 at 12:43:28AM +0900, Akira Yokosawa wrote:
> On Mon, 24 Nov 2025 09:14:57 -0800, Paul E. McKenney wrote:
> > On Mon, Nov 24, 2025 at 06:10:56PM +0900, Akira Yokosawa wrote:
> >> Hi Paul,
> >>
> >> I see a couple of minor issues in running
> >>
> >> - make cross-klitmus; cd klitmus; make
> >> - make run-herd7
> >>
> >> under CodeSamples/formal/herd/.
> >>
> >> This patch set should resolve those issues.
> >
> > Queued and pushed, thank you!
>
> Paul, this is a side note with regard to klitmus7.
>
> In the old days with Linux kernel <6.0, tests listed in KLITMUS_READY
> took less than a minute.
>
> That have changed a lot. Litmus tests with RCU primitives tend to
> take much longer.
>
> In the list of processes shown by "top", such klitmus threads are
> accompanied by a single (and fairly busy) "rcu_exp_gp_kthread_worker".
> Yes, the frequency RCU primitives are invoked in those litmus tests,
> especially synchronize_rcu(), is unusually high. In realistic workloads,
> such burst of synchronize_rcu()s would indicate you are doing something
> wrong. So the longer time to complete does not mean any performance
> regression.
If I remember correctly, the klitmus tool switch from synchronize_rcu() to
synchronize_rcu_expedited() at about that time. That would definitely
account for the appearance of rcu_exp_gp_kthread_worker(), which
handles expedited grace periods. The motivation for switching klitmus
to synchronize_rcu_expedited() is that the shorter grace periods make
detecting RCU-usage bugs in litmus tests.
Except that your data below shows that downgrading to a pre-v6.0 kernel
reduces the overhead. Hmmm...
> Following is a list of summaries from C-RomanPenyaev-list-rcu-rr runs
> under several VMs:
>
> - Ubuntu 20.04 POWER8 (kernel: 5.4.0-223-generic #243-Ubuntu SMP, 8 CPUs)
>
> Hash=890796f9231134fff935fc64b2b98940
> Observation C-RomanPenyaev-list-rcu-rr Never 0 4000000
> Time C-RomanPenyaev-list-rcu-rr 17.85
>
> no rcu_exp_gp_kthread_worker.
>
> - Ubuntu 24.04 POWER9 (kernel: 6.8.0-88-generic #89-Ubuntu SMP, 8 CPUs)
>
> Hash=890796f9231134fff935fc64b2b98940
> Observation C-RomanPenyaev-list-rcu-rr Never 0 4000000
> Time C-RomanPenyaev-list-rcu-rr 153.49
>
> no rcu_exp_gp_kthread_worker observed.
>
> - Ubuntu 24.04 x86_64 (kernel: 6.8.0-88-generic #89-Ubuntu SMP
> PREEMPT_DYNAMIC, 6 CPUs)
>
> Hash=890796f9231134fff935fc64b2b98940
> Observation C-RomanPenyaev-list-rcu-rr Never 0 3000000
> Time C-RomanPenyaev-list-rcu-rr 60.46
>
> no rcu_exp_gp_kthread_worker.
>
> - Ubuntu 25.10 x86_64 (kernel: 6.17.0-7-generic #7-Ubuntu SMP
> PREEMPT_DYNAMIC, 6 CPUs)
>
> Hash=890796f9231134fff935fc64b2b98940
> Observation C-RomanPenyaev-list-rcu-rr Never 0 3000000
> Time C-RomanPenyaev-list-rcu-rr 311.79
>
> rcu_exp_gp_kthread_worker consumes a lot of time.
>
> - Fedora 43 x86_64 (kernel: 6.17.8-300.fc43.x86_64 #1 SMP PREEMPT_DYNAMIC, 6
> CPUs)
>
> Hash=890796f9231134fff935fc64b2b98940
> Observation C-RomanPenyaev-list-rcu-rr Never 0 3000000
> Time C-RomanPenyaev-list-rcu-rr 326.54
>
> rcu_exp_gp_kthread_worker consumes a lot of time.
How much time? A full CPU's worth through the klitmus run? More than
a CPU? Less?
You mention that this might not be a regression, but then again, it might
well be. ;-)
> Just out of curiosity, is there any knob I can tweak to reduce the time
> to complete, preferably some kernel boot arg?
Well, there is the rcupdate.rcu_normal=1 kernel boot parameter, which
disables expedited grace periods completely. But that cure might be
worse than the disease.
Just to make sure, you are using the exact same version of klitmus
throughout, right?
If you are using the same klitmus throughout, are you comfortable building
and booting kernels on any of these systems? (If so, I would ask you to
bisect, but to focus only on changes affecting kernel/rcu, for example,
"git bisect start v6.17 v6.8 -- kernel/rcu".)
Thanx, Paul
