Re: [PATCH RFC] rcu: torture: shorten the time between forward-progress tests
On Thu, Aug 24, 2023 at 5:15 AM Paul E. McKenney wrote: > > On Tue, May 02, 2023 at 11:06:02PM +0800, zhouzho...@gmail.com wrote: > > From: Zhouyi Zhou > > > > Currently, default time between rcu torture forward-progress tests is 60HZ, > > Under this configuration, false positive caused by __stack_chk_fail [1] is > > difficult to reproduce (needs average 5*420 seconds for SRCU-P), > > which means one has to invoke [2] 5 times in average to make [1] appear. > > > > With time between rcu torture forward-progress tests be 1 HZ, above > > phenomenon will be reproduced within 3 minutes, which means we can > > reproduce [1] everytime we invoke [2]. > > > > Although [1] is a false positive, this change will make possible future > > true bugs easier to be discovered. > > > > [1] Link: > > https://lore.kernel.org/lkml/CAABZP2yS5=zuwezq7ihkv0wdm_hgo8k-teahyjrzhavzkda...@mail.gmail.com/T/ > > [2] tools/testing/selftests/rcutorture/bin/torture.sh > > > > Tested in PPC VM of Opensource Lab of Oregon State Univerisity. > > > > Signed-off-by: Zhouyi Zhou > > Please accept my apologies for being ridiculously slow to reply! Never mind. I have made a lot of self improvement during the study of RCU and RCU torture and your book ;-) > > In recent -rcu, module parameters such as this one that simply set a > value can be overridden on the command line. So you could get the effect > (again, in recent kernels) in your testing by adding: > > --bootargs "rcutorture.fwd_progress_holdoff=1" > > The reason that I am reluctant to accept this patch is that we sometimes > have trouble with this forward-progress testing exhausting memory, and > making in happen could therefore cause trouble with generic rcutorture > testing. Agree, false positives can disrupt our judgement in many cases. Thanx Zhouyi > > Or am I missing the point of this change? > > Thanx, Paul > > > --- > > tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot | 1 + > > tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot | 1 + > > tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot | 1 + > > tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot | 1 + > > tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot | 1 + > > 5 files changed, 5 insertions(+) > > > > diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot > > b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot > > index ce0694fd9b92..982582bff041 100644 > > --- a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot > > +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-N.boot > > @@ -1,2 +1,3 @@ > > rcutorture.torture_type=srcu > > rcutorture.fwd_progress=3 > > +rcutorture.fwd_progress_holdoff=1 > > diff --git a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot > > b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot > > index 2db39f298d18..18f5d7361d8a 100644 > > --- a/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot > > +++ b/tools/testing/selftests/rcutorture/configs/rcu/SRCU-P.boot > > @@ -1,4 +1,5 @@ > > rcutorture.torture_type=srcud > > rcupdate.rcu_self_test=1 > > rcutorture.fwd_progress=3 > > +rcutorture.fwd_progress_holdoff=1 > > srcutree.big_cpu_lim=5 > > diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot > > b/tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot > > index c70b5db6c2ae..b86bc7df7603 100644 > > --- a/tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot > > +++ b/tools/testing/selftests/rcutorture/configs/rcu/TRACE02.boot > > @@ -1,2 +1,3 @@ > > rcutorture.torture_type=tasks-tracing > > rcutorture.fwd_progress=2 > > +rcutorture.fwd_progress_holdoff=1 > > diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot > > b/tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot > > index dd914fa8f690..933302f885df 100644 > > --- a/tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot > > +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE02.boot > > @@ -1 +1,2 @@ > > rcutorture.fwd_progress=2 > > +rcutorture.fwd_progress_holdoff=1 > > diff --git a/tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot > > b/tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot > > index dd914fa8f690..933302f885df 100644 > > --- a/tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot > > +++ b/tools/testing/selftests/rcutorture/configs/rcu/TREE10.boot > > @@ -1 +1,2 @@ > > rcutorture.fwd_progress=2 > > +rcutorture.fwd_progress_holdoff=1 > > -- > > 2.34.1 > >
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
On Thu, Jul 6, 2023 at 3:09 PM Christophe Leroy
wrote:
>
>
>
> Le 21/11/2022 à 04:51, Zhouyi Zhou a écrit :
> > During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to
> > offline tick_do_timer_cpu, the operation will fail because in
> > function tick_nohz_cpu_down:
> > ```
> > if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
> >return -EBUSY;
> > ```
> > Above bug was first discovered in torture tests performed in PPC VM
> > of Open Source Lab of Oregon State University, and reproducable in RISC-V
> > and X86-64 (with additional kernel commandline cpu0_hotplug).
> >
> > In this patch, we avoid offline tick_do_timer_cpu by distribute
> > the offlining cpu among remaining cpus.
> >
> > Signed-off-by: Zhouyi Zhou
> > ---
> > include/linux/tick.h| 1 +
> > kernel/time/tick-common.c | 1 +
> > kernel/time/tick-internal.h | 1 -
> > kernel/torture.c| 10 ++
> > 4 files changed, 12 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/linux/tick.h b/include/linux/tick.h
> > index bfd571f18cfd..23cc0b205853 100644
> > --- a/include/linux/tick.h
> > +++ b/include/linux/tick.h
> > @@ -14,6 +14,7 @@
> > #include
> >
> > #ifdef CONFIG_GENERIC_CLOCKEVENTS
> > +extern int tick_do_timer_cpu __read_mostly;
> > extern void __init tick_init(void);
> > /* Should be core only, but ARM BL switcher requires it */
> > extern void tick_suspend_local(void);
> > diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> > index 46789356f856..87b9b9afa320 100644
> > --- a/kernel/time/tick-common.c
> > +++ b/kernel/time/tick-common.c
> > @@ -48,6 +48,7 @@ ktime_t tick_next_period;
> >*procedure also covers cpu hotplug.
> >*/
> > int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT;
> > +EXPORT_SYMBOL_GPL(tick_do_timer_cpu);
> > #ifdef CONFIG_NO_HZ_FULL
> > /*
> >* tick_do_timer_boot_cpu indicates the boot CPU temporarily owns
> > diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
> > index 649f2b48e8f0..8953dca10fdd 100644
> > --- a/kernel/time/tick-internal.h
> > +++ b/kernel/time/tick-internal.h
> > @@ -15,7 +15,6 @@
> >
> > DECLARE_PER_CPU(struct tick_device, tick_cpu_device);
> > extern ktime_t tick_next_period;
> > -extern int tick_do_timer_cpu __read_mostly;
> >
> > extern void tick_setup_periodic(struct clock_event_device *dev, int
> > broadcast);
> > extern void tick_handle_periodic(struct clock_event_device *dev);
> > diff --git a/kernel/torture.c b/kernel/torture.c
> > index 789aeb0e1159..bccbdd33dda2 100644
> > --- a/kernel/torture.c
> > +++ b/kernel/torture.c
> > @@ -33,6 +33,7 @@
> > #include
> > #include
> > #include
> > +#include
> > #include
> > #include
> > #include
> > @@ -358,7 +359,16 @@ torture_onoff(void *arg)
> > schedule_timeout_interruptible(HZ / 10);
> > continue;
> > }
> > +#ifdef CONFIG_NO_HZ_FULL
> > + /* do not offline tick do timer cpu */
> > + if (tick_nohz_full_running) {
>
> Can you use fonction tick_nohz_full_enabled() instead and avoid the #ifdef ?
Thank Christophe for your wonderful advice, I will follow your advice
next time I prepare a patch.
For this false positive report, 58d766824264 ("tick/nohz: Fix
cpu_is_hotpluggable() by checking with nohz subsystem") has beaten me
in mainline.
Thanks again
Zhouyi
>
> > + cpu = (torture_random() >> 4) % maxcpu;
> > + if (cpu >= tick_do_timer_cpu)
> > + cpu = (cpu + 1) % (maxcpu + 1);
> > + } else
> > +#else
> > cpu = (torture_random() >> 4) % (maxcpu + 1);
> > +#endif
> > if (!torture_offline(cpu,
> >_offline_attempts,
> > _offline_successes,
> >_offline, _offline,
> > _offline))
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Thu, Apr 27, 2023 at 10:13 PM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > On Thu, Apr 27, 2023 at 11:09 AM Michael Ellerman > > wrote: > >> > >> Zhouyi Zhou writes: > >> > On Tue, Apr 25, 2023 at 2:01 PM Zhouyi Zhou wrote: > >> >> On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman > >> >> wrote: > >> ... > >> >> > > >> >> > There's 12.2.0 here: > >> >> > > >> >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > >> >> > > >> >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ > >> > >> > powerpc64le-linux-gnu-gcc-12 cross compiler on my Ubuntu 22.04 does > >> > not seem to have that issue as gcc-10 does > >> > >> OK. So so far it's only that GCC 10 that shows the problem. > >> > >> If you have time, you could use some of the other versions to narrow > >> down which versions show the bug: > >> > >> https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/ > >> > >> There's an 11.0, 11.1 and 11.3 there, as well as 9.5 and so on. > > GCC test results (Tested on PPC VM of Open Source Lab of Oregon State > > University) > > gcc 9.4 (ubuntu native): positive, show bug > > gcc 9.5 (download form [1]): positive, show bug > > gcc 10.1 (download from [1]): positive, show bug > > gcc 10.3 (download from [1]): positive, show bug > > gcc 10.4 (download from [1]): positive, show bug > > > > gcc 11.0 (download from [1]): negative, no bug > > gcc 11.1 (download from [1]): negative, no bug > > gcc 11.3 (download from [1]): negative, no bug > > gcc 12.1 (download from [1]): negative, no bug > > gcc 12.2 (download from [1]): negative, no bug > > Awesome work. Thank you for your encouragement ;-) ;-) > > How are you testing for presence/absence of the bug? By running your > test and seeing if it crashes, or by looking at the generated code? Both I use gdb ./vmlinux; gdb)disassemble srcu_gp_start_if_needed to look up the generated assembly code and I use [1] to see if it crashes, if there is a bug, it always crashes very quickly (within 3 minutes) [1] http://140.211.169.189/0425/whilebash.sh Cheers > > cheers
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Thu, Apr 27, 2023 at 11:09 AM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > On Tue, Apr 25, 2023 at 2:01 PM Zhouyi Zhou wrote: > >> On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman > >> wrote: > ... > >> > > >> > There's 12.2.0 here: > >> > > >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > >> > > >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ > > > powerpc64le-linux-gnu-gcc-12 cross compiler on my Ubuntu 22.04 does > > not seem to have that issue as gcc-10 does > > OK. So so far it's only that GCC 10 that shows the problem. > > If you have time, you could use some of the other versions to narrow > down which versions show the bug: > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/ > > There's an 11.0, 11.1 and 11.3 there, as well as 9.5 and so on. GCC test results (Tested on PPC VM of Open Source Lab of Oregon State University) gcc 9.4 (ubuntu native): positive, show bug gcc 9.5 (download form [1]): positive, show bug gcc 10.1 (download from [1]): positive, show bug gcc 10.3 (download from [1]): positive, show bug gcc 10.4 (download from [1]): positive, show bug gcc 11.0 (download from [1]): negative, no bug gcc 11.1 (download from [1]): negative, no bug gcc 11.3 (download from [1]): negative, no bug gcc 12.1 (download from [1]): negative, no bug gcc 12.2 (download from [1]): negative, no bug [1] https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/ I am very happy to cooperate if there is further need ;-) Cheers Zhouyi > > There's
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Thu, Apr 27, 2023 at 11:09 AM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > On Tue, Apr 25, 2023 at 2:01 PM Zhouyi Zhou wrote: > >> On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman > >> wrote: > ... > >> > > >> > There's 12.2.0 here: > >> > > >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > >> > > >> > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ > > > powerpc64le-linux-gnu-gcc-12 cross compiler on my Ubuntu 22.04 does > > not seem to have that issue as gcc-10 does > > OK. So so far it's only that GCC 10 that shows the problem. The default gcc version 9.4.0 in ubuntu 20.04 (in VM of Oregon State University) also shows the problem. > > If you have time, you could use some of the other versions to narrow > down which versions show the bug: > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/ > > There's an 11.0, 11.1 and 11.3 there, as well as 9.5 and so on. I have time ;-), and am very glad to try the above versions to narrow down which version shows the bug ;-) Cheers Zhouyi > > There's
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Wed, Apr 26, 2023 at 10:15 AM Joel Fernandes wrote: > > Hi Zhouyi, > > On Wed, Apr 26, 2023 at 09:31:17AM +0800, Zhouyi Zhou wrote: > [..] > > Joel makes the learning process easier for me, indeed! > > I know that feeling being a learner myself ;-) > > > One question I have tried very hard to understand, but still confused. > > for now, I know > > r13 is fixed, but r1 is not, why "r9,40(r1)"'s 40(r1) can be assumed > > to be equal to 3192(r10). > > First you have to I guess read up a bit about stack canaries. Google for > "gcc stack protector" and "gcc stack canaries", and the look for basics of > "buffer overflow attacks". That'll explain the concept of stack guards etc > (Sorry if this is too obvious but I did not know how much you knew about it > already). > > 40(r1) is where the canary was stored. In the beginning of the function, you > have this: > > c0226d58: 78 0c 2d e9 ld r9,3192(r13) > c0226d5c: 28 00 21 f9 std r9,40(r1) > > r1 is your stack pointer. 3192(r13) is the canary value. > > 40(r1) is where the canary is stored for later comparison. > > r1 should not change through out the function I believe, because otherwise > you don't know where the stack frame is, right? Thanks Joel's awesome explanation. I can understand the mechanics behind our situation now!! 40(r1) is where the canary is stored for later comparison, this is located on the stack. while 3192(r13) is inside the cpu's PACA. I quote Christophe's note here "in order to be able to have the canary as an offset of a fixed register as expected by GCC, we copy the task canary into the cpu's PACA struct during _switch(): addir6,r4,-THREAD /* Convert THREAD to 'current' */ std r6,PACACURRENT(r13) /* Set new 'current' */ #if defined(CONFIG_STACKPROTECTOR) ld r6, TASK_CANARY(r6) std r6, PACA_CANARY(r13) #endif " > > Later you have this stuff before the function returns which gcc presumably > did due to optimization. That mr means move register and is where the caching > of r13 to r10 happens that Boqun pointed. Thank Boqun and all others' wonderful debugging! Your work confirmed my bug report ;-) > > c0226eb4: 78 6b aa 7d mr r10,r13 > [...] > and then the canary comparison happens: > > c0226ed8: 28 00 21 e9 ld r9,40(r1) > c0226edc: 78 0c 4a e9 ld r10,3192(r10) 3192(r13) is correct because "we copy the task canary into the cpu's PACA struct during _switch():" but 3192(r10) is not correct, because r10 is the old value of r13. > c0226ee0: 79 52 29 7d xor.r9,r9,r10 > c0226ee4: 00 00 40 39 li r10,0 > c0226ee8: b8 03 82 40 bne c02272a0 > > > So looks like for this to blow up, the preemption/migration has to happen > precisely between the mr doing the caching, and the xor doing the comparison, > since that's when the r10 will be stale. Thank Joel and all others for your time ;-) I benefit a lot, and am very glad to do more good work to the community in return ;-) Cheers Zhouyi > > thanks, > > - Joel >
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Wed, Apr 26, 2023 at 8:33 AM Joel Fernandes wrote:
>
> On Tue, Apr 25, 2023 at 9:50 AM Zhouyi Zhou wrote:
> >
> > Hi
> >
> > On Tue, Apr 25, 2023 at 9:40 PM Christophe Leroy
> > wrote:
> > >
> > >
> > >
> > > Le 25/04/2023 à 13:06, Joel Fernandes a écrit :
> > > > On Tue, Apr 25, 2023 at 6:58 AM Zhouyi Zhou
> > > > wrote:
> > > >>
> > > >> hi
> > > >>
> > > >> On Tue, Apr 25, 2023 at 6:13 PM Peter Zijlstra
> > > >> wrote:
> > > >>>
> > > >>> On Mon, Apr 24, 2023 at 02:55:11PM -0400, Joel Fernandes wrote:
> > > >>>> This is amazing debugging Boqun, like a boss! One comment below:
> > > >>>>
> > > >>>>>>> Or something simple I haven't thought of? :)
> > > >>>>>>
> > > >>>>>> At what points can r13 change? Only when some particular
> > > >>>>>> functions are
> > > >>>>>> called?
> > > >>>>>>
> > > >>>>>
> > > >>>>> r13 is the local paca:
> > > >>>>>
> > > >>>>> register struct paca_struct *local_paca asm("r13");
> > > >>>>>
> > > >>>>> , which is a pointer to percpu data.
> > > >>>>>
> > > >>>>> So if a task schedule from one CPU to anotehr CPU, the value gets
> > > >>>>> changed.
> > > >>>>
> > > >>>> It appears the whole issue, per your analysis, is that the stack
> > > >>>> checking code in gcc should not cache or alias r13, and must read its
> > > >>>> most up-to-date value during stack checking, as its value may have
> > > >>>> changed during a migration to a new CPU.
> > > >>>>
> > > >>>> Did I get that right?
> > > >>>>
> > > >>>> IMO, even without a reproducer, gcc on PPC should just not do that,
> > > >>>> that feels terribly broken for the kernel. I wonder what clang does,
> > > >>>> I'll go poke around with compilerexplorer after lunch.
> > > >>>>
> > > >>>> Adding +Peter Zijlstra as well to join the party as I have a feeling
> > > >>>> he'll be interested. ;-)
> > > >>>
> > > >>> I'm a little confused; the way I understand the whole stack protector
> > > >>> thing to work is that we push a canary on the stack at call and on
> > > >>> return check it is still valid. Since in general tasks randomly
> > > >>> migrate,
> > > >>> the per-cpu validation canary should be the same on all CPUs.
> > > >>>
> > > >>> Additionally, the 'new' __srcu_read_{,un}lock_nmisafe() functions use
> > > >>> raw_cpu_ptr() to get 'a' percpu sdp, preferably that of the local cpu,
> > > >>> but no guarantees.
> > > >>>
> > > >>> Both cases use r13 (paca) in a racy manner, and in both cases it
> > > >>> should
> > > >>> be safe.
> > > >> New test results today: both gcc build from git (git clone
> > > >> git://gcc.gnu.org/git/gcc.git) and Ubuntu 22.04 gcc-12.1.0
> > > >> are immune from the above issue. We can see the assembly code on
> > > >> http://140.211.169.189/0425/srcu_gp_start_if_needed-gcc-12.txt
> > > >>
> > > >> while
> > > >> Both native gcc on PPC vm (gcc version 9.4.0), and gcc cross compiler
> > > >> on my x86 laptop (gcc version 10.4.0) will reproduce the bug.
> > > >
> > > > Do you know what fixes the issue? I would not declare victory yet. My
> > > > feeling is something changes in timing, or compiler codegen which
> > > > hides the issue. So the issue is still there but it is just a matter
> > > > of time before someone else reports it.
> > > >
> > > > Out of curiosity for PPC folks, why cannot 64-bit PPC use per-task
> > > > canary? Michael, is this an optimization? Adding Christophe as well
> > > > since it came in a few years ago via the following commit:
> > >
> > > It uses per-task canary. But unlike PPC32, PPC64 doesn't have a fixed
> > > register pointing to 'cu
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
Hi
On Tue, Apr 25, 2023 at 9:40 PM Christophe Leroy
wrote:
>
>
>
> Le 25/04/2023 à 13:06, Joel Fernandes a écrit :
> > On Tue, Apr 25, 2023 at 6:58 AM Zhouyi Zhou wrote:
> >>
> >> hi
> >>
> >> On Tue, Apr 25, 2023 at 6:13 PM Peter Zijlstra
> >> wrote:
> >>>
> >>> On Mon, Apr 24, 2023 at 02:55:11PM -0400, Joel Fernandes wrote:
> >>>> This is amazing debugging Boqun, like a boss! One comment below:
> >>>>
> >>>>>>> Or something simple I haven't thought of? :)
> >>>>>>
> >>>>>> At what points can r13 change? Only when some particular functions are
> >>>>>> called?
> >>>>>>
> >>>>>
> >>>>> r13 is the local paca:
> >>>>>
> >>>>> register struct paca_struct *local_paca asm("r13");
> >>>>>
> >>>>> , which is a pointer to percpu data.
> >>>>>
> >>>>> So if a task schedule from one CPU to anotehr CPU, the value gets
> >>>>> changed.
> >>>>
> >>>> It appears the whole issue, per your analysis, is that the stack
> >>>> checking code in gcc should not cache or alias r13, and must read its
> >>>> most up-to-date value during stack checking, as its value may have
> >>>> changed during a migration to a new CPU.
> >>>>
> >>>> Did I get that right?
> >>>>
> >>>> IMO, even without a reproducer, gcc on PPC should just not do that,
> >>>> that feels terribly broken for the kernel. I wonder what clang does,
> >>>> I'll go poke around with compilerexplorer after lunch.
> >>>>
> >>>> Adding +Peter Zijlstra as well to join the party as I have a feeling
> >>>> he'll be interested. ;-)
> >>>
> >>> I'm a little confused; the way I understand the whole stack protector
> >>> thing to work is that we push a canary on the stack at call and on
> >>> return check it is still valid. Since in general tasks randomly migrate,
> >>> the per-cpu validation canary should be the same on all CPUs.
> >>>
> >>> Additionally, the 'new' __srcu_read_{,un}lock_nmisafe() functions use
> >>> raw_cpu_ptr() to get 'a' percpu sdp, preferably that of the local cpu,
> >>> but no guarantees.
> >>>
> >>> Both cases use r13 (paca) in a racy manner, and in both cases it should
> >>> be safe.
> >> New test results today: both gcc build from git (git clone
> >> git://gcc.gnu.org/git/gcc.git) and Ubuntu 22.04 gcc-12.1.0
> >> are immune from the above issue. We can see the assembly code on
> >> http://140.211.169.189/0425/srcu_gp_start_if_needed-gcc-12.txt
> >>
> >> while
> >> Both native gcc on PPC vm (gcc version 9.4.0), and gcc cross compiler
> >> on my x86 laptop (gcc version 10.4.0) will reproduce the bug.
> >
> > Do you know what fixes the issue? I would not declare victory yet. My
> > feeling is something changes in timing, or compiler codegen which
> > hides the issue. So the issue is still there but it is just a matter
> > of time before someone else reports it.
> >
> > Out of curiosity for PPC folks, why cannot 64-bit PPC use per-task
> > canary? Michael, is this an optimization? Adding Christophe as well
> > since it came in a few years ago via the following commit:
>
> It uses per-task canary. But unlike PPC32, PPC64 doesn't have a fixed
> register pointing to 'current' at all time so the canary is copied into
> a per-cpu struct during _switch().
>
> If GCC keeps an old value of the per-cpu struct pointer, it then gets
> the canary from the wrong CPU struct so from a different task.
This is a fruitful learning process for me!
Christophe:
Do you think there is still a need to bisect GCC ? If so, I am very
glad to continue
Cheers
Zhouyi
>
> Christophe
>
> >
> > commit 06ec27aea9fc84d9c6d879eb64b5bcf28a8a1eb7
> > Author: Christophe Leroy
> > Date: Thu Sep 27 07:05:55 2018 +
> >
> > powerpc/64: add stack protector support
> >
> > On PPC64, as register r13 points to the paca_struct at all time,
> > this patch adds a copy of the canary there, which is copied at
> > task_switch.
> > That new canary is then used by using the following GCC options:
> > -mstack-protector-guard=tls
> > -mstack-protector-guard-reg=r13
> > -mstack-protector-guard-offset=offsetof(struct paca_struct, canary))
> >
> > Signed-off-by: Christophe Leroy
> > Signed-off-by: Michael Ellerman
> >
> > - Joel
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Tue, Apr 25, 2023 at 7:06 PM Joel Fernandes wrote:
>
> On Tue, Apr 25, 2023 at 6:58 AM Zhouyi Zhou wrote:
> >
> > hi
> >
> > On Tue, Apr 25, 2023 at 6:13 PM Peter Zijlstra wrote:
> > >
> > > On Mon, Apr 24, 2023 at 02:55:11PM -0400, Joel Fernandes wrote:
> > > > This is amazing debugging Boqun, like a boss! One comment below:
> > > >
> > > > > > > Or something simple I haven't thought of? :)
> > > > > >
> > > > > > At what points can r13 change? Only when some particular functions
> > > > > > are
> > > > > > called?
> > > > > >
> > > > >
> > > > > r13 is the local paca:
> > > > >
> > > > > register struct paca_struct *local_paca asm("r13");
> > > > >
> > > > > , which is a pointer to percpu data.
> > > > >
> > > > > So if a task schedule from one CPU to anotehr CPU, the value gets
> > > > > changed.
> > > >
> > > > It appears the whole issue, per your analysis, is that the stack
> > > > checking code in gcc should not cache or alias r13, and must read its
> > > > most up-to-date value during stack checking, as its value may have
> > > > changed during a migration to a new CPU.
> > > >
> > > > Did I get that right?
> > > >
> > > > IMO, even without a reproducer, gcc on PPC should just not do that,
> > > > that feels terribly broken for the kernel. I wonder what clang does,
> > > > I'll go poke around with compilerexplorer after lunch.
> > > >
> > > > Adding +Peter Zijlstra as well to join the party as I have a feeling
> > > > he'll be interested. ;-)
> > >
> > > I'm a little confused; the way I understand the whole stack protector
> > > thing to work is that we push a canary on the stack at call and on
> > > return check it is still valid. Since in general tasks randomly migrate,
> > > the per-cpu validation canary should be the same on all CPUs.
> > >
> > > Additionally, the 'new' __srcu_read_{,un}lock_nmisafe() functions use
> > > raw_cpu_ptr() to get 'a' percpu sdp, preferably that of the local cpu,
> > > but no guarantees.
> > >
> > > Both cases use r13 (paca) in a racy manner, and in both cases it should
> > > be safe.
> > New test results today: both gcc build from git (git clone
> > git://gcc.gnu.org/git/gcc.git) and Ubuntu 22.04 gcc-12.1.0
> > are immune from the above issue. We can see the assembly code on
> > http://140.211.169.189/0425/srcu_gp_start_if_needed-gcc-12.txt
> >
> > while
> > Both native gcc on PPC vm (gcc version 9.4.0), and gcc cross compiler
> > on my x86 laptop (gcc version 10.4.0) will reproduce the bug.
>
> Do you know what fixes the issue? I would not declare victory yet. My
> feeling is something changes in timing, or compiler codegen which
> hides the issue. So the issue is still there but it is just a matter
> of time before someone else reports it.
I am going to try bisect on GCC, hope we can find some clue.
>
> Out of curiosity for PPC folks, why cannot 64-bit PPC use per-task
> canary? Michael, is this an optimization? Adding Christophe as well
> since it came in a few years ago via the following commit:
>
> commit 06ec27aea9fc84d9c6d879eb64b5bcf28a8a1eb7
> Author: Christophe Leroy
> Date: Thu Sep 27 07:05:55 2018 +
>
> powerpc/64: add stack protector support
>
> On PPC64, as register r13 points to the paca_struct at all time,
> this patch adds a copy of the canary there, which is copied at
> task_switch.
> That new canary is then used by using the following GCC options:
> -mstack-protector-guard=tls
> -mstack-protector-guard-reg=r13
> -mstack-protector-guard-offset=offsetof(struct paca_struct, canary))
>
> Signed-off-by: Christophe Leroy
> Signed-off-by: Michael Ellerman
>
> - Joel
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
hi
On Tue, Apr 25, 2023 at 6:13 PM Peter Zijlstra wrote:
>
> On Mon, Apr 24, 2023 at 02:55:11PM -0400, Joel Fernandes wrote:
> > This is amazing debugging Boqun, like a boss! One comment below:
> >
> > > > > Or something simple I haven't thought of? :)
> > > >
> > > > At what points can r13 change? Only when some particular functions are
> > > > called?
> > > >
> > >
> > > r13 is the local paca:
> > >
> > > register struct paca_struct *local_paca asm("r13");
> > >
> > > , which is a pointer to percpu data.
> > >
> > > So if a task schedule from one CPU to anotehr CPU, the value gets
> > > changed.
> >
> > It appears the whole issue, per your analysis, is that the stack
> > checking code in gcc should not cache or alias r13, and must read its
> > most up-to-date value during stack checking, as its value may have
> > changed during a migration to a new CPU.
> >
> > Did I get that right?
> >
> > IMO, even without a reproducer, gcc on PPC should just not do that,
> > that feels terribly broken for the kernel. I wonder what clang does,
> > I'll go poke around with compilerexplorer after lunch.
> >
> > Adding +Peter Zijlstra as well to join the party as I have a feeling
> > he'll be interested. ;-)
>
> I'm a little confused; the way I understand the whole stack protector
> thing to work is that we push a canary on the stack at call and on
> return check it is still valid. Since in general tasks randomly migrate,
> the per-cpu validation canary should be the same on all CPUs.
>
> Additionally, the 'new' __srcu_read_{,un}lock_nmisafe() functions use
> raw_cpu_ptr() to get 'a' percpu sdp, preferably that of the local cpu,
> but no guarantees.
>
> Both cases use r13 (paca) in a racy manner, and in both cases it should
> be safe.
New test results today: both gcc build from git (git clone
git://gcc.gnu.org/git/gcc.git) and Ubuntu 22.04 gcc-12.1.0
are immune from the above issue. We can see the assembly code on
http://140.211.169.189/0425/srcu_gp_start_if_needed-gcc-12.txt
while
Both native gcc on PPC vm (gcc version 9.4.0), and gcc cross compiler
on my x86 laptop (gcc version 10.4.0) will reproduce the bug.
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Tue, Apr 25, 2023 at 2:01 PM Zhouyi Zhou wrote: > > hi > > On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman wrote: > > > > Zhouyi Zhou writes: > > > Dear PowerPC and RCU developers: > > > During the RCU torture test on mainline (on the VM of Opensource Lab > > > of Oregon State University), SRCU-P failed with __stack_chk_fail: > > ... > > > by debugging, I see the r10 is assigned with r13 on c0226eb4, > > > but if there is a context-switch before c0226edc, a false > > > positive will be reported. > > > > > > [1] http://154.220.3.115/logs/0422/configformainline.txt > > > > Says: > > > > CONFIG_CC_VERSION_TEXT="powerpc64le-linux-gnu-gcc-10 (Ubuntu > > 10.4.0-4ubuntu1~22.04) 10.4.0" > > > > Do you see the same issue with a newer GCC? > On PPC vm of Oregon State University (I can grant rsa-pub key ssh > access if you are interested), I > build and install the gcc from git, then use the newly built gcc to > compile the kernel, the bug disappears, > I don't know why. Following is what is do: > > 1) git clone git://gcc.gnu.org/git/gcc.git > git rev-parse --short HEAD > f0eabc52c9a > 2) mkdir gcc/build > 3) cd gcc/build > 4) ../configure --disable-bootstrap --prefix=/home/ubuntu/gcc-install > 5) make -j 4 //my VM has limited memory ;-) > 6) make install > 7) cd linux-dir > git rev-parse --short HEAD > 61d325dcbc05 > 8) export PATH=/home/ubuntu/gcc-install/bin:$PATH > 9) make vmlinux -j 8 > 10) ./whilebash.sh [1] > > From the assembly code of srcu_gp_start_if_needed [2], I found stack protector > is operated directly on r13: > > c0225098: 30 00 0d e9 ld r8,48(r13) > c022509c: 08 00 3c e9 ld r9,8(r28) > c02250a0: 14 42 29 7d add r9,r9,r8 > c02250a4: ac 04 00 7c hwsync > c02250a8: 10 00 7b 3b addir27,r27,16 > c02250ac: 14 da 29 7d add r9,r9,r27 > c02250b0: a8 48 00 7d ldarx r8,0,r9 > c02250b4: 01 00 08 31 addic r8,r8,1 > c02250b8: ad 49 00 7d stdcx. r8,0,r9 > c02250bc: f4 ff c2 40 bne-c02250b0 > > c02250c0: 28 00 01 e9 ld r8,40(r1) > c02250c4: 78 0c 2d e9 ld r9,3192(r13) > c02250c8: 79 4a 08 7d xor.r8,r8,r9 > c02250cc: 00 00 20 39 li r9,0 > c02250d0: 90 03 82 40 bne c0225460 > > > console.log is attached at [3]. > > [1] 140.211.169.189/0425/whilebash.sh > [2] http://140.211.169.189/0425/srcu_gp_start_if_needed.txt > [3] http://140.211.169.189/0425/console.log > > I am very glad to cooperate if there is anything else I can do ;-) > > Cheers > Zhouyi > > > > There's 12.2.0 here: > > > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > > > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ powerpc64le-linux-gnu-gcc-12 cross compiler on my Ubuntu 22.04 does not seem to have that issue as gcc-10 does [4] http://140.211.169.189/0425/srcu_gp_start_if_needed-gcc-12.txt > > > > Or if you can build in a Fedora 38 system or container, it has GCC 13. > > > > cheers
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
hi On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > Dear PowerPC and RCU developers: > > During the RCU torture test on mainline (on the VM of Opensource Lab > > of Oregon State University), SRCU-P failed with __stack_chk_fail: > ... > > by debugging, I see the r10 is assigned with r13 on c0226eb4, > > but if there is a context-switch before c0226edc, a false > > positive will be reported. > > > > [1] http://154.220.3.115/logs/0422/configformainline.txt > > Says: > > CONFIG_CC_VERSION_TEXT="powerpc64le-linux-gnu-gcc-10 (Ubuntu > 10.4.0-4ubuntu1~22.04) 10.4.0" > > Do you see the same issue with a newer GCC? On PPC vm of Oregon State University (I can grant rsa-pub key ssh access if you are interested), I build and install the gcc from git, then use the newly built gcc to compile the kernel, the bug disappears, I don't know why. Following is what is do: 1) git clone git://gcc.gnu.org/git/gcc.git git rev-parse --short HEAD f0eabc52c9a 2) mkdir gcc/build 3) cd gcc/build 4) ../configure --disable-bootstrap --prefix=/home/ubuntu/gcc-install 5) make -j 4 //my VM has limited memory ;-) 6) make install 7) cd linux-dir git rev-parse --short HEAD 61d325dcbc05 8) export PATH=/home/ubuntu/gcc-install/bin:$PATH 9) make vmlinux -j 8 10) ./whilebash.sh [1] >From the assembly code of srcu_gp_start_if_needed [2], I found stack protector is operated directly on r13: c0225098: 30 00 0d e9 ld r8,48(r13) c022509c: 08 00 3c e9 ld r9,8(r28) c02250a0: 14 42 29 7d add r9,r9,r8 c02250a4: ac 04 00 7c hwsync c02250a8: 10 00 7b 3b addir27,r27,16 c02250ac: 14 da 29 7d add r9,r9,r27 c02250b0: a8 48 00 7d ldarx r8,0,r9 c02250b4: 01 00 08 31 addic r8,r8,1 c02250b8: ad 49 00 7d stdcx. r8,0,r9 c02250bc: f4 ff c2 40 bne-c02250b0 c02250c0: 28 00 01 e9 ld r8,40(r1) c02250c4: 78 0c 2d e9 ld r9,3192(r13) c02250c8: 79 4a 08 7d xor.r8,r8,r9 c02250cc: 00 00 20 39 li r9,0 c02250d0: 90 03 82 40 bne c0225460 console.log is attached at [3]. [1] 140.211.169.189/0425/whilebash.sh [2] http://140.211.169.189/0425/srcu_gp_start_if_needed.txt [3] http://140.211.169.189/0425/console.log I am very glad to cooperate if there is anything else I can do ;-) Cheers Zhouyi > > There's 12.2.0 here: > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ > > Or if you can build in a Fedora 38 system or container, it has GCC 13. > > cheers
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Tue, Apr 25, 2023 at 6:07 AM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > Dear PowerPC and RCU developers: > > During the RCU torture test on mainline (on the VM of Opensource Lab > > of Oregon State University), SRCU-P failed with __stack_chk_fail: > ... > > by debugging, I see the r10 is assigned with r13 on c0226eb4, > > but if there is a context-switch before c0226edc, a false > > positive will be reported. > > > > [1] http://154.220.3.115/logs/0422/configformainline.txt > > Says: > > CONFIG_CC_VERSION_TEXT="powerpc64le-linux-gnu-gcc-10 (Ubuntu > 10.4.0-4ubuntu1~22.04) 10.4.0" > > Do you see the same issue with a newer GCC? I would like to try that in the newest GCC ;-), please give me about a day's time because I am going to compile the gcc ;-) > > There's 12.2.0 here: > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/12.2.0/ > > https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/ppc64le/12.2.0/ > > Or if you can build in a Fedora 38 system or container, it has GCC 13. OK, I will try it or similar This is a very learningful process for me, thank you all ;-) Cheers > > cheers
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
Thank Boqun for your wonderful analysis!
On Mon, Apr 24, 2023 at 8:33 AM Boqun Feng wrote:
>
> On Sat, Apr 22, 2023 at 09:28:39PM +0200, Joel Fernandes wrote:
> > On Sat, Apr 22, 2023 at 2:47 PM Zhouyi Zhou wrote:
> > >
> > > Dear PowerPC and RCU developers:
> > > During the RCU torture test on mainline (on the VM of Opensource Lab
> > > of Oregon State University), SRCU-P failed with __stack_chk_fail:
> > > [ 264.381952][ T99] [c6c7bab0] [c10c67c0]
> > > dump_stack_lvl+0x94/0xd8 (unreliable)
> > > [ 264.383786][ T99] [c6c7bae0] [c014fc94]
> > > panic+0x19c/0x468
> > > [ 264.385128][ T99] [c6c7bb80] [c10fca24]
> > > __stack_chk_fail+0x24/0x30
> > > [ 264.386610][ T99] [c6c7bbe0] [c02293b4]
> > > srcu_gp_start_if_needed+0x5c4/0x5d0
> > > [ 264.388188][ T99] [c6c7bc70] [c022f7f4]
> > > srcu_torture_call+0x34/0x50
> > > [ 264.389611][ T99] [c6c7bc90] [c022b5e8]
> > > rcu_torture_fwd_prog+0x8c8/0xa60
> > > [ 264.391439][ T99] [c6c7be00] [c018e37c]
> > > kthread+0x15c/0x170
> > > [ 264.392792][ T99] [c6c7be50] [c000df94]
> > > ret_from_kernel_thread+0x5c/0x64
> > > The kernel config file can be found in [1].
> > > And I write a bash script to accelerate the bug reproducing [2].
> > > After a week's debugging, I found the cause of the bug is because the
> > > register r10 used to judge for stack overflow is not constant between
> > > context switches.
> > > The assembly code for srcu_gp_start_if_needed is located at [3]:
> > > c0226eb4: 78 6b aa 7d mr r10,r13
> > > c0226eb8: 14 42 29 7d add r9,r9,r8
> > > c0226ebc: ac 04 00 7c hwsync
> > > c0226ec0: 10 00 7b 3b addir27,r27,16
> > > c0226ec4: 14 da 29 7d add r9,r9,r27
> > > c0226ec8: a8 48 00 7d ldarx r8,0,r9
> > > c0226ecc: 01 00 08 31 addic r8,r8,1
> > > c0226ed0: ad 49 00 7d stdcx. r8,0,r9
> > > c0226ed4: f4 ff c2 40 bne-c0226ec8
> > >
> > > c0226ed8: 28 00 21 e9 ld r9,40(r1)
> > > c0226edc: 78 0c 4a e9 ld r10,3192(r10)
> > > c0226ee0: 79 52 29 7d xor.r9,r9,r10
> > > c0226ee4: 00 00 40 39 li r10,0
> > > c0226ee8: b8 03 82 40 bne c02272a0
> > >
> > > by debugging, I see the r10 is assigned with r13 on c0226eb4,
> > > but if there is a context-switch before c0226edc, a false
> > > positive will be reported.
> > >
> > > [1] http://154.220.3.115/logs/0422/configformainline.txt
> > > [2] 154.220.3.115/logs/0422/whilebash.sh
> > > [3] http://154.220.3.115/logs/0422/srcu_gp_start_if_needed.txt
> > >
> > > My analysis and debugging may not be correct, but the bug is easily
> > > reproducible.
> >
> > If this is a bug in the stack smashing protection as you seem to hint,
> > I wonder if you see the issue with a specific gcc version and is a
> > compiler-specific issue. It's hard to say, but considering this I
>
> Very likely, more asm code from Zhouyi's link:
>
> This is the __srcu_read_unlock_nmisafe(), since "hwsync" is
> smp_mb__{after,before}_atomic(), and the following code is first
> barrier then atomic, so it's the unlock.
>
> c0226eb4: 78 6b aa 7d mr r10,r13
>
> ^ r13 is the pointer to percpu data on PPC64 kernel, and it's also
> the pointer to TLS data for userspace code.
>
> c0226eb8: 14 42 29 7d add r9,r9,r8
> c0226ebc: ac 04 00 7c hwsync
> c0226ec0: 10 00 7b 3b addir27,r27,16
> c0226ec4: 14 da 29 7d add r9,r9,r27
> c0226ec8: a8 48 00 7d ldarx r8,0,r9
> c0226ecc: 01 00 08 31 addic r8,r8,1
> c0226ed0: ad 49 00 7d stdcx. r8,0,r9
> c0226ed4: f4 ff c2 40 bne-c0226ec8
>
> c0226ed8: 28 00 21 e9 ld r9,40(r1)
> c0226edc: 78 0c 4a e9 ld r10,3192(r10)
>
> here I think that the compiler is using r10 as an alias to r13, since
> for userspace program, it's safe to assume the TLS pointer doesn't
> change. However this is not true fo
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Sun, Apr 23, 2023 at 9:37 AM Zhouyi Zhou wrote: > > On Sun, Apr 23, 2023 at 3:19 AM Joel Fernandes wrote: > > > > Hi Zhouyi, > Thank Joel for your quick response ;-) > I will gradually provide all the necessary information to facilitate > our chasing. Please do not hesitate email me > if I have ignored any ;-) > > > > On Sat, Apr 22, 2023 at 2:47 PM Zhouyi Zhou wrote: > > > > > > Dear PowerPC and RCU developers: > > > During the RCU torture test on mainline (on the VM of Opensource Lab > > > of Oregon State University), SRCU-P failed with __stack_chk_fail: > > > [ 264.381952][ T99] [c6c7bab0] [c10c67c0] > > > dump_stack_lvl+0x94/0xd8 (unreliable) > > > [ 264.383786][ T99] [c6c7bae0] [c014fc94] > > > panic+0x19c/0x468 > > > [ 264.385128][ T99] [c6c7bb80] [c10fca24] > > > __stack_chk_fail+0x24/0x30 > > > [ 264.386610][ T99] [c6c7bbe0] [c02293b4] > > > srcu_gp_start_if_needed+0x5c4/0x5d0 > > > [ 264.388188][ T99] [c6c7bc70] [c022f7f4] > > > srcu_torture_call+0x34/0x50 > > > [ 264.389611][ T99] [c6c7bc90] [c022b5e8] > > > rcu_torture_fwd_prog+0x8c8/0xa60 > > > [ 264.391439][ T99] [c6c7be00] [c018e37c] > > > kthread+0x15c/0x170 > > > [ 264.392792][ T99] [c6c7be50] [c000df94] > > > ret_from_kernel_thread+0x5c/0x64 > > > The kernel config file can be found in [1]. > > > And I write a bash script to accelerate the bug reproducing [2]. > > > After a week's debugging, I found the cause of the bug is because the > > > register r10 used to judge for stack overflow is not constant between > > > context switches. > > > The assembly code for srcu_gp_start_if_needed is located at [3]: > > > c0226eb4: 78 6b aa 7d mr r10,r13 > > > c0226eb8: 14 42 29 7d add r9,r9,r8 > > > c0226ebc: ac 04 00 7c hwsync > > > c0226ec0: 10 00 7b 3b addir27,r27,16 > > > c0226ec4: 14 da 29 7d add r9,r9,r27 > > > c0226ec8: a8 48 00 7d ldarx r8,0,r9 > > > c0226ecc: 01 00 08 31 addic r8,r8,1 > > > c0226ed0: ad 49 00 7d stdcx. r8,0,r9 > > > c0226ed4: f4 ff c2 40 bne-c0226ec8 > > > > > > c0226ed8: 28 00 21 e9 ld r9,40(r1) > > > c0226edc: 78 0c 4a e9 ld r10,3192(r10) > > > c0226ee0: 79 52 29 7d xor.r9,r9,r10 > > > c0226ee4: 00 00 40 39 li r10,0 > > > c0226ee8: b8 03 82 40 bne c02272a0 > > > > > > by debugging, I see the r10 is assigned with r13 on c0226eb4, > > > but if there is a context-switch before c0226edc, a false > > > positive will be reported. > > > > > > [1] http://154.220.3.115/logs/0422/configformainline.txt > > > [2] 154.220.3.115/logs/0422/whilebash.sh > > > [3] http://154.220.3.115/logs/0422/srcu_gp_start_if_needed.txt > > > > > > My analysis and debugging may not be correct, but the bug is easily > > > reproducible. > > > > Could you provide the full kernel log? It is not clear exactly from > > your attachments, but I think this is a stack overflow issue as > > implied by the mention of __stack_chk_fail. One trick might be to turn > > on any available stack debug kernel config options, or check the > > kernel logs for any messages related to shortage of remaining stack > > space. > The full kernel log is [1] > [1] http://154.220.3.115/logs/0422/console.log > > > > Additionally, you could also find out where the kernel crash happened > > in C code following the below notes [1] I wrote (see section "Figuring > > out where kernel crashes happen in C code"). The notes are > > x86-specific but should be generally applicable (In the off chance > > you'd like to improve the notes, feel free to share them ;-)). > Fantastic article!!!, I benefit a lot from reading it. Because we can > reproduce it so easily on powerpc VM, > I can even use gdb to debug it, following is my debug process on > 2e83b879fb91dafe995967b46a1d38a5b0889242(srcu: Create an > srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe()). > > [2] http://154.220.3.115/logs/0422/gdb.txt > > > > Lastly, is it a specific kernel release from which you start seeing > > this issue? You should try git bisect if it is easil
Re: BUG : PowerPC RCU: torture test failed with __stack_chk_fail
On Sun, Apr 23, 2023 at 3:19 AM Joel Fernandes wrote: > > Hi Zhouyi, Thank Joel for your quick response ;-) I will gradually provide all the necessary information to facilitate our chasing. Please do not hesitate email me if I have ignored any ;-) > > On Sat, Apr 22, 2023 at 2:47 PM Zhouyi Zhou wrote: > > > > Dear PowerPC and RCU developers: > > During the RCU torture test on mainline (on the VM of Opensource Lab > > of Oregon State University), SRCU-P failed with __stack_chk_fail: > > [ 264.381952][ T99] [c6c7bab0] [c10c67c0] > > dump_stack_lvl+0x94/0xd8 (unreliable) > > [ 264.383786][ T99] [c6c7bae0] [c014fc94] > > panic+0x19c/0x468 > > [ 264.385128][ T99] [c6c7bb80] [c10fca24] > > __stack_chk_fail+0x24/0x30 > > [ 264.386610][ T99] [c6c7bbe0] [c02293b4] > > srcu_gp_start_if_needed+0x5c4/0x5d0 > > [ 264.388188][ T99] [c6c7bc70] [c022f7f4] > > srcu_torture_call+0x34/0x50 > > [ 264.389611][ T99] [c6c7bc90] [c022b5e8] > > rcu_torture_fwd_prog+0x8c8/0xa60 > > [ 264.391439][ T99] [c6c7be00] [c018e37c] > > kthread+0x15c/0x170 > > [ 264.392792][ T99] [c6c7be50] [c000df94] > > ret_from_kernel_thread+0x5c/0x64 > > The kernel config file can be found in [1]. > > And I write a bash script to accelerate the bug reproducing [2]. > > After a week's debugging, I found the cause of the bug is because the > > register r10 used to judge for stack overflow is not constant between > > context switches. > > The assembly code for srcu_gp_start_if_needed is located at [3]: > > c0226eb4: 78 6b aa 7d mr r10,r13 > > c0226eb8: 14 42 29 7d add r9,r9,r8 > > c0226ebc: ac 04 00 7c hwsync > > c0226ec0: 10 00 7b 3b addir27,r27,16 > > c0226ec4: 14 da 29 7d add r9,r9,r27 > > c0226ec8: a8 48 00 7d ldarx r8,0,r9 > > c0226ecc: 01 00 08 31 addic r8,r8,1 > > c0226ed0: ad 49 00 7d stdcx. r8,0,r9 > > c0226ed4: f4 ff c2 40 bne-c0226ec8 > > > > c0226ed8: 28 00 21 e9 ld r9,40(r1) > > c0226edc: 78 0c 4a e9 ld r10,3192(r10) > > c0226ee0: 79 52 29 7d xor.r9,r9,r10 > > c0226ee4: 00 00 40 39 li r10,0 > > c0226ee8: b8 03 82 40 bne c02272a0 > > > > by debugging, I see the r10 is assigned with r13 on c0226eb4, > > but if there is a context-switch before c0226edc, a false > > positive will be reported. > > > > [1] http://154.220.3.115/logs/0422/configformainline.txt > > [2] 154.220.3.115/logs/0422/whilebash.sh > > [3] http://154.220.3.115/logs/0422/srcu_gp_start_if_needed.txt > > > > My analysis and debugging may not be correct, but the bug is easily > > reproducible. > > Could you provide the full kernel log? It is not clear exactly from > your attachments, but I think this is a stack overflow issue as > implied by the mention of __stack_chk_fail. One trick might be to turn > on any available stack debug kernel config options, or check the > kernel logs for any messages related to shortage of remaining stack > space. The full kernel log is [1] [1] http://154.220.3.115/logs/0422/console.log > > Additionally, you could also find out where the kernel crash happened > in C code following the below notes [1] I wrote (see section "Figuring > out where kernel crashes happen in C code"). The notes are > x86-specific but should be generally applicable (In the off chance > you'd like to improve the notes, feel free to share them ;-)). Fantastic article!!!, I benefit a lot from reading it. Because we can reproduce it so easily on powerpc VM, I can even use gdb to debug it, following is my debug process on 2e83b879fb91dafe995967b46a1d38a5b0889242(srcu: Create an srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe()). [2] http://154.220.3.115/logs/0422/gdb.txt > > Lastly, is it a specific kernel release from which you start seeing > this issue? You should try git bisect if it is easily reproducible in > a newer release, but goes away in an older one. I did bisect on powerpc VM, the problem begin to appear on 2e83b879fb91dafe995967b46a1d38a5b0889242(srcu: Create an srcu_read_lock_nmisafe() and srcu_read_unlock_nmisafe()). The kernel is good at 5d0f5953b60f5f7a278085b55ddc73e2932f4c33(srcu: Convert ->srcu_lock_count and ->srcu_unlock_count to atomic) But if I apply the following patch [3] to 5d0f5953b60f5f7a278085b55ddc73e2932f4c33, the bug appears
BUG : PowerPC RCU: torture test failed with __stack_chk_fail
Dear PowerPC and RCU developers: During the RCU torture test on mainline (on the VM of Opensource Lab of Oregon State University), SRCU-P failed with __stack_chk_fail: [ 264.381952][ T99] [c6c7bab0] [c10c67c0] dump_stack_lvl+0x94/0xd8 (unreliable) [ 264.383786][ T99] [c6c7bae0] [c014fc94] panic+0x19c/0x468 [ 264.385128][ T99] [c6c7bb80] [c10fca24] __stack_chk_fail+0x24/0x30 [ 264.386610][ T99] [c6c7bbe0] [c02293b4] srcu_gp_start_if_needed+0x5c4/0x5d0 [ 264.388188][ T99] [c6c7bc70] [c022f7f4] srcu_torture_call+0x34/0x50 [ 264.389611][ T99] [c6c7bc90] [c022b5e8] rcu_torture_fwd_prog+0x8c8/0xa60 [ 264.391439][ T99] [c6c7be00] [c018e37c] kthread+0x15c/0x170 [ 264.392792][ T99] [c6c7be50] [c000df94] ret_from_kernel_thread+0x5c/0x64 The kernel config file can be found in [1]. And I write a bash script to accelerate the bug reproducing [2]. After a week's debugging, I found the cause of the bug is because the register r10 used to judge for stack overflow is not constant between context switches. The assembly code for srcu_gp_start_if_needed is located at [3]: c0226eb4: 78 6b aa 7d mr r10,r13 c0226eb8: 14 42 29 7d add r9,r9,r8 c0226ebc: ac 04 00 7c hwsync c0226ec0: 10 00 7b 3b addir27,r27,16 c0226ec4: 14 da 29 7d add r9,r9,r27 c0226ec8: a8 48 00 7d ldarx r8,0,r9 c0226ecc: 01 00 08 31 addic r8,r8,1 c0226ed0: ad 49 00 7d stdcx. r8,0,r9 c0226ed4: f4 ff c2 40 bne-c0226ec8 c0226ed8: 28 00 21 e9 ld r9,40(r1) c0226edc: 78 0c 4a e9 ld r10,3192(r10) c0226ee0: 79 52 29 7d xor.r9,r9,r10 c0226ee4: 00 00 40 39 li r10,0 c0226ee8: b8 03 82 40 bne c02272a0 by debugging, I see the r10 is assigned with r13 on c0226eb4, but if there is a context-switch before c0226edc, a false positive will be reported. [1] http://154.220.3.115/logs/0422/configformainline.txt [2] 154.220.3.115/logs/0422/whilebash.sh [3] http://154.220.3.115/logs/0422/srcu_gp_start_if_needed.txt My analysis and debugging may not be correct, but the bug is easily reproducible. Thanks Zhouyi
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
Thank you all for your guidance and encouragement! I learn how to construct commit message properly and learn how important the role that the torture test framework plays for the Linux kernel. Hope I can be of benefit to the community by my work. I am going to continue to study this topic and study the torture test framework, and wait for your further instructions. Best Regards Zhouyi On Mon, Nov 28, 2022 at 1:53 AM Paul E. McKenney wrote: > > On Sun, Nov 27, 2022 at 01:40:28PM +0100, Thomas Gleixner wrote: > > [ . . . ] > > > >> No. We are not exporting this just to make a bogus test case happy. > > >> > > >> Fix the torture code to handle -EBUSY correctly. > > > I am going to do a study on this, for now, I do a grep in the kernel tree: > > > find . -name "*.c"|xargs grep cpuhp_setup_state|wc -l > > > The result of the grep command shows that there are 268 > > > cpuhp_setup_state* cases. > > > which may make our task more complicated. > > > > Why? The whole point of this torture thing is to stress the > > infrastructure. > > Indeed. > > > There are quite some reasons why a CPU-hotplug or a hot-unplug operation > > can fail, which is not a fatal problem, really. > > > > So if a CPU hotplug operation fails, then why can't the torture test > > just move on and validate that the system still behaves correctly? > > > > That gives us more coverage than just testing the good case and giving > > up when something unexpected happens. > > Agreed, with access to a function like the tick_nohz_full_timekeeper() > suggested earlier in this email thread, then yes, it would make sense to > try to offline the CPU anyway, then forgive the failure in cases where > the CPU matches that indicated by tick_nohz_full_timekeeper(). > > > I even argue that the torture test should inject random failures into > > the hotplug state machine to achieve extended code coverage. > > I could imagine torture_onoff() telling various CPU-hotplug notifiers > to refuse the transition using some TBD interface. That would better > test the CPU-hotplug common code's ability to deal with failures. > > Or did you have something else/additional in mind? > > Thanx, Paul
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
Thank Thomas for your guidance On Sun, Nov 27, 2022 at 1:05 AM Thomas Gleixner wrote: > > On Mon, Nov 21 2022 at 11:51, Zhouyi Zhou wrote: > > During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to > > offline tick_do_timer_cpu, the operation will fail because in > > function tick_nohz_cpu_down: > > ``` > > if (tick_nohz_full_running && tick_do_timer_cpu == cpu) > > return -EBUSY; > > ``` > > Above bug was first discovered in torture tests performed in PPC VM > > How is this a bug? Yes, this is a false positive instead. > > > of Open Source Lab of Oregon State University, and reproducable in RISC-V > > and X86-64 (with additional kernel commandline cpu0_hotplug). > > > > In this patch, we avoid offline tick_do_timer_cpu by distribute > > the offlining cpu among remaining cpus. > > Please read Documentation/process. Search for 'this patch'... Documentation/process/submitting-patches.rst says: "Describe your changes in imperative mood, e.g. "make xyzzy do frotz" instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour." So, I should construct my patch as: We avoid ... by ... > > > > > Signed-off-by: Zhouyi Zhou > > --- > > include/linux/tick.h| 1 + > > kernel/time/tick-common.c | 1 + > > kernel/time/tick-internal.h | 1 - > > kernel/torture.c| 10 ++ > > 4 files changed, 12 insertions(+), 1 deletion(-) > > > > diff --git a/include/linux/tick.h b/include/linux/tick.h > > index bfd571f18cfd..23cc0b205853 100644 > > --- a/include/linux/tick.h > > +++ b/include/linux/tick.h > > @@ -14,6 +14,7 @@ > > #include > > > > #ifdef CONFIG_GENERIC_CLOCKEVENTS > > +extern int tick_do_timer_cpu __read_mostly; > > extern void __init tick_init(void); > > /* Should be core only, but ARM BL switcher requires it */ > > extern void tick_suspend_local(void); > > diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c > > index 46789356f856..87b9b9afa320 100644 > > --- a/kernel/time/tick-common.c > > +++ b/kernel/time/tick-common.c > > @@ -48,6 +48,7 @@ ktime_t tick_next_period; > > *procedure also covers cpu hotplug. > > */ > > int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT; > > +EXPORT_SYMBOL_GPL(tick_do_timer_cpu); > > No. We are not exporting this just to make a bogus test case happy. > > Fix the torture code to handle -EBUSY correctly. I am going to do a study on this, for now, I do a grep in the kernel tree: find . -name "*.c"|xargs grep cpuhp_setup_state|wc -l The result of the grep command shows that there are 268 cpuhp_setup_state* cases. which may make our task more complicated. After my study, should we also take Frederic's proposal as a possible option? (construct a function for this) https://lore.kernel.org/lkml/20221123223658.GC1395324@lothringen/ I learned a lot during this process Many thanks Zhouyi > > Thanks, > > tglx
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
On Thu, Nov 24, 2022 at 2:49 AM Paul E. McKenney wrote:
>
> On Wed, Nov 23, 2022 at 10:23:11AM +0800, Zhouyi Zhou wrote:
> > On Tue, Nov 22, 2022 at 9:37 AM Paul E. McKenney wrote:
> > >
> > > On Mon, Nov 21, 2022 at 11:51:40AM +0800, Zhouyi Zhou wrote:
> > > > During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to
> > > > offline tick_do_timer_cpu, the operation will fail because in
> > > > function tick_nohz_cpu_down:
> > > > ```
> > > > if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
> > > > return -EBUSY;
> > > > ```
> > > > Above bug was first discovered in torture tests performed in PPC VM
> > > > of Open Source Lab of Oregon State University, and reproducable in
> > > > RISC-V
> > > > and X86-64 (with additional kernel commandline cpu0_hotplug).
> > > >
> > > > In this patch, we avoid offline tick_do_timer_cpu by distribute
> > > > the offlining cpu among remaining cpus.
> > > >
> > > > Signed-off-by: Zhouyi Zhou
> > >
> > > Good show chasing this down!
> > Thank Paul for your guidance and encouragement!
> > >
> > > A couple of questions below.
> > The answers below.
> > >
> > > > ---
> > > > include/linux/tick.h| 1 +
> > > > kernel/time/tick-common.c | 1 +
> > > > kernel/time/tick-internal.h | 1 -
> > > > kernel/torture.c| 10 ++
> > > > 4 files changed, 12 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/include/linux/tick.h b/include/linux/tick.h
> > > > index bfd571f18cfd..23cc0b205853 100644
> > > > --- a/include/linux/tick.h
> > > > +++ b/include/linux/tick.h
> > > > @@ -14,6 +14,7 @@
> > > > #include
> > > >
> > > > #ifdef CONFIG_GENERIC_CLOCKEVENTS
> > > > +extern int tick_do_timer_cpu __read_mostly;
> > > > extern void __init tick_init(void);
> > > > /* Should be core only, but ARM BL switcher requires it */
> > > > extern void tick_suspend_local(void);
> > > > diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> > > > index 46789356f856..87b9b9afa320 100644
> > > > --- a/kernel/time/tick-common.c
> > > > +++ b/kernel/time/tick-common.c
> > > > @@ -48,6 +48,7 @@ ktime_t tick_next_period;
> > > > *procedure also covers cpu hotplug.
> > > > */
> > > > int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT;
> > > > +EXPORT_SYMBOL_GPL(tick_do_timer_cpu);
> > > > #ifdef CONFIG_NO_HZ_FULL
> > > > /*
> > > > * tick_do_timer_boot_cpu indicates the boot CPU temporarily owns
> > > > diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
> > > > index 649f2b48e8f0..8953dca10fdd 100644
> > > > --- a/kernel/time/tick-internal.h
> > > > +++ b/kernel/time/tick-internal.h
> > > > @@ -15,7 +15,6 @@
> > > >
> > > > DECLARE_PER_CPU(struct tick_device, tick_cpu_device);
> > > > extern ktime_t tick_next_period;
> > > > -extern int tick_do_timer_cpu __read_mostly;
> > > >
> > > > extern void tick_setup_periodic(struct clock_event_device *dev, int
> > > > broadcast);
> > > > extern void tick_handle_periodic(struct clock_event_device *dev);
> > > > diff --git a/kernel/torture.c b/kernel/torture.c
> > > > index 789aeb0e1159..bccbdd33dda2 100644
> > > > --- a/kernel/torture.c
> > > > +++ b/kernel/torture.c
> > > > @@ -33,6 +33,7 @@
> > > > #include
> > > > #include
> > > > #include
> > > > +#include
> > > > #include
> > > > #include
> > > > #include
> > > > @@ -358,7 +359,16 @@ torture_onoff(void *arg)
> > > > schedule_timeout_interruptible(HZ / 10);
> > > > continue;
> > > > }
> > > > +#ifdef CONFIG_NO_HZ_FULL
> > > > + /* do not offline tick do timer cpu */
> > > > + if (tick_nohz_full_running) {
> > > > + cpu = (torture_random() >> 4) % maxcpu;
by examine the beginning code of torture_onoff, I see if we has 8
cpus, then maxcpu = 7 (not 8) here,
then cpu is distributed evenly among 0, 1, 2, 3, 4, 5, 6
if we happens to get 6, then cp
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
On Thu, Nov 24, 2022 at 6:37 AM Frederic Weisbecker wrote:
>
> On Mon, Nov 21, 2022 at 11:51:40AM +0800, Zhouyi Zhou wrote:
> > During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to
> > offline tick_do_timer_cpu, the operation will fail because in
> > function tick_nohz_cpu_down:
> > ```
> > if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
> > return -EBUSY;
> > ```
> > Above bug was first discovered in torture tests performed in PPC VM
> > of Open Source Lab of Oregon State University, and reproducable in RISC-V
> > and X86-64 (with additional kernel commandline cpu0_hotplug).
> >
> > In this patch, we avoid offline tick_do_timer_cpu by distribute
> > the offlining cpu among remaining cpus.
> >
> > Signed-off-by: Zhouyi Zhou
> > ---
> > include/linux/tick.h| 1 +
> > kernel/time/tick-common.c | 1 +
> > kernel/time/tick-internal.h | 1 -
> > kernel/torture.c| 10 ++
> > 4 files changed, 12 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/linux/tick.h b/include/linux/tick.h
> > index bfd571f18cfd..23cc0b205853 100644
> > --- a/include/linux/tick.h
> > +++ b/include/linux/tick.h
> > @@ -14,6 +14,7 @@
> > #include
> >
> > #ifdef CONFIG_GENERIC_CLOCKEVENTS
> > +extern int tick_do_timer_cpu __read_mostly;
> > extern void __init tick_init(void);
> > /* Should be core only, but ARM BL switcher requires it */
> > extern void tick_suspend_local(void);
> > diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> > index 46789356f856..87b9b9afa320 100644
> > --- a/kernel/time/tick-common.c
> > +++ b/kernel/time/tick-common.c
> > @@ -48,6 +48,7 @@ ktime_t tick_next_period;
> > *procedure also covers cpu hotplug.
> > */
> > int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT;
> > +EXPORT_SYMBOL_GPL(tick_do_timer_cpu);
>
> Please rather make a function for this. This is an internal value
> that we don't want to expose to modules.
>
> This can be:
>
> int tick_nohz_full_timekeeper(void)
> {
> if (tick_nohz_full_enabled() && tick_do_timer_cpu >= 0)
> return tick_do_timer_cpu;
> else
> return nr_cpu_ids;
> }
>
> And then just check if the value is below nr_cpu_ids.
Thank Paul and Frederic both for your guidance!
Things are much easier;-) and I will do it.
Cheers
Zhouyi
>
> Thanks.
Re: [PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
On Tue, Nov 22, 2022 at 9:37 AM Paul E. McKenney wrote:
>
> On Mon, Nov 21, 2022 at 11:51:40AM +0800, Zhouyi Zhou wrote:
> > During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to
> > offline tick_do_timer_cpu, the operation will fail because in
> > function tick_nohz_cpu_down:
> > ```
> > if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
> > return -EBUSY;
> > ```
> > Above bug was first discovered in torture tests performed in PPC VM
> > of Open Source Lab of Oregon State University, and reproducable in RISC-V
> > and X86-64 (with additional kernel commandline cpu0_hotplug).
> >
> > In this patch, we avoid offline tick_do_timer_cpu by distribute
> > the offlining cpu among remaining cpus.
> >
> > Signed-off-by: Zhouyi Zhou
>
> Good show chasing this down!
Thank Paul for your guidance and encouragement!
>
> A couple of questions below.
The answers below.
>
> > ---
> > include/linux/tick.h| 1 +
> > kernel/time/tick-common.c | 1 +
> > kernel/time/tick-internal.h | 1 -
> > kernel/torture.c| 10 ++
> > 4 files changed, 12 insertions(+), 1 deletion(-)
> >
> > diff --git a/include/linux/tick.h b/include/linux/tick.h
> > index bfd571f18cfd..23cc0b205853 100644
> > --- a/include/linux/tick.h
> > +++ b/include/linux/tick.h
> > @@ -14,6 +14,7 @@
> > #include
> >
> > #ifdef CONFIG_GENERIC_CLOCKEVENTS
> > +extern int tick_do_timer_cpu __read_mostly;
> > extern void __init tick_init(void);
> > /* Should be core only, but ARM BL switcher requires it */
> > extern void tick_suspend_local(void);
> > diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
> > index 46789356f856..87b9b9afa320 100644
> > --- a/kernel/time/tick-common.c
> > +++ b/kernel/time/tick-common.c
> > @@ -48,6 +48,7 @@ ktime_t tick_next_period;
> > *procedure also covers cpu hotplug.
> > */
> > int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT;
> > +EXPORT_SYMBOL_GPL(tick_do_timer_cpu);
> > #ifdef CONFIG_NO_HZ_FULL
> > /*
> > * tick_do_timer_boot_cpu indicates the boot CPU temporarily owns
> > diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
> > index 649f2b48e8f0..8953dca10fdd 100644
> > --- a/kernel/time/tick-internal.h
> > +++ b/kernel/time/tick-internal.h
> > @@ -15,7 +15,6 @@
> >
> > DECLARE_PER_CPU(struct tick_device, tick_cpu_device);
> > extern ktime_t tick_next_period;
> > -extern int tick_do_timer_cpu __read_mostly;
> >
> > extern void tick_setup_periodic(struct clock_event_device *dev, int
> > broadcast);
> > extern void tick_handle_periodic(struct clock_event_device *dev);
> > diff --git a/kernel/torture.c b/kernel/torture.c
> > index 789aeb0e1159..bccbdd33dda2 100644
> > --- a/kernel/torture.c
> > +++ b/kernel/torture.c
> > @@ -33,6 +33,7 @@
> > #include
> > #include
> > #include
> > +#include
> > #include
> > #include
> > #include
> > @@ -358,7 +359,16 @@ torture_onoff(void *arg)
> > schedule_timeout_interruptible(HZ / 10);
> > continue;
> > }
> > +#ifdef CONFIG_NO_HZ_FULL
> > + /* do not offline tick do timer cpu */
> > + if (tick_nohz_full_running) {
> > + cpu = (torture_random() >> 4) % maxcpu;
> > + if (cpu >= tick_do_timer_cpu)
>
> Why is this ">=" instead of "=="?
I use probability theory here to let the remaining cpu distribute evenly.
Example:
we have cpus: 0 1 2 3 4 5 6 7
maxcpu = 7
tick_do_timer_cpu = 2
remaining cpus are: 0 1 3 4 5 6 7
if the offline cpu candidate is 2, then the result cpu is 2+1
else if the offline cpu candidate is 3, then the result cpu is 3+1
...
else if the offline cpu candidate is 6, then the result cpu is 6+1
>
> > + cpu = (cpu + 1) % (maxcpu + 1);
we could just use cpu = cpu + 1 here
> > + } else
> > +#else
> > cpu = (torture_random() >> 4) % (maxcpu + 1);
> > +#endif
>
> What happens if the value of tick_do_timer_cpu changes between the time of
> the check above and the call to torture_offline() below? Alternatively,
> how is such a change in value prevented?
I did a preliminary research about the above question, this is quite
complicated for me
(because I think I must not bring locks to kernel just because our
test frame need them),
Please give me some days to perform intensive research.
Thanks again
Cheers
Zhouyi
>
> Thanx, Paul
>
> > if (!torture_offline(cpu,
> >_offline_attempts,
> > _offline_successes,
> >_offline, _offline,
> > _offline))
> > --
> > 2.34.1
> >
[PATCH linux-next][RFC]torture: avoid offline tick_do_timer_cpu
During CPU-hotplug torture (CONFIG_NO_HZ_FULL=y), if we try to
offline tick_do_timer_cpu, the operation will fail because in
function tick_nohz_cpu_down:
```
if (tick_nohz_full_running && tick_do_timer_cpu == cpu)
return -EBUSY;
```
Above bug was first discovered in torture tests performed in PPC VM
of Open Source Lab of Oregon State University, and reproducable in RISC-V
and X86-64 (with additional kernel commandline cpu0_hotplug).
In this patch, we avoid offline tick_do_timer_cpu by distribute
the offlining cpu among remaining cpus.
Signed-off-by: Zhouyi Zhou
---
include/linux/tick.h| 1 +
kernel/time/tick-common.c | 1 +
kernel/time/tick-internal.h | 1 -
kernel/torture.c| 10 ++
4 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/include/linux/tick.h b/include/linux/tick.h
index bfd571f18cfd..23cc0b205853 100644
--- a/include/linux/tick.h
+++ b/include/linux/tick.h
@@ -14,6 +14,7 @@
#include
#ifdef CONFIG_GENERIC_CLOCKEVENTS
+extern int tick_do_timer_cpu __read_mostly;
extern void __init tick_init(void);
/* Should be core only, but ARM BL switcher requires it */
extern void tick_suspend_local(void);
diff --git a/kernel/time/tick-common.c b/kernel/time/tick-common.c
index 46789356f856..87b9b9afa320 100644
--- a/kernel/time/tick-common.c
+++ b/kernel/time/tick-common.c
@@ -48,6 +48,7 @@ ktime_t tick_next_period;
*procedure also covers cpu hotplug.
*/
int tick_do_timer_cpu __read_mostly = TICK_DO_TIMER_BOOT;
+EXPORT_SYMBOL_GPL(tick_do_timer_cpu);
#ifdef CONFIG_NO_HZ_FULL
/*
* tick_do_timer_boot_cpu indicates the boot CPU temporarily owns
diff --git a/kernel/time/tick-internal.h b/kernel/time/tick-internal.h
index 649f2b48e8f0..8953dca10fdd 100644
--- a/kernel/time/tick-internal.h
+++ b/kernel/time/tick-internal.h
@@ -15,7 +15,6 @@
DECLARE_PER_CPU(struct tick_device, tick_cpu_device);
extern ktime_t tick_next_period;
-extern int tick_do_timer_cpu __read_mostly;
extern void tick_setup_periodic(struct clock_event_device *dev, int broadcast);
extern void tick_handle_periodic(struct clock_event_device *dev);
diff --git a/kernel/torture.c b/kernel/torture.c
index 789aeb0e1159..bccbdd33dda2 100644
--- a/kernel/torture.c
+++ b/kernel/torture.c
@@ -33,6 +33,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -358,7 +359,16 @@ torture_onoff(void *arg)
schedule_timeout_interruptible(HZ / 10);
continue;
}
+#ifdef CONFIG_NO_HZ_FULL
+ /* do not offline tick do timer cpu */
+ if (tick_nohz_full_running) {
+ cpu = (torture_random() >> 4) % maxcpu;
+ if (cpu >= tick_do_timer_cpu)
+ cpu = (cpu + 1) % (maxcpu + 1);
+ } else
+#else
cpu = (torture_random() >> 4) % (maxcpu + 1);
+#endif
if (!torture_offline(cpu,
_offline_attempts, _offline_successes,
_offline, _offline, _offline))
--
2.34.1
Re: [PATCH linux-next][RFC] powerpc: fix HOTPLUG error in rcutorture
Hi, I also reappear the same phenomenon in RISC-V: [ 120.156380] scftorture: --- End of test: LOCK_HOTPLUG So I guess it is not the arch's responsibility. I am very interested in it ;-) Thank you both for your guidance! Cheers Zhouyi On Tue, Oct 11, 2022 at 9:59 AM Zhouyi Zhou wrote: > > Thanks Michael for reviewing my patch > > On Mon, Oct 10, 2022 at 7:21 PM Michael Ellerman wrote: > > > > Zhouyi Zhou writes: > > > I think we should avoid torture offline the cpu who do tick timer > > > when nohz full is running. > > > > Can you tell us what the bug you're fixing is? > > > > Did you see a crash/oops/hang etc? Or are you just proposing this as > > something that would be a good idea? > Sorry for the trouble and inconvenience that I bring to the community. > I haven't made myself clear in my patch. > The ins and outs are as follows: > 1) cd linux-next > 2) ./tools/testing/selftests/rcutorture/bin/torture.sh > after 19 hours ;-) > 3) tail > ./tools/testing/selftests/rcutorture/res/2022.09.30-01.06.22-torture/results-scftorture/NOPREEMPT/console.log > > [ 121.449268][ T57] scftorture: scf_invoked_count VER: 2415215 > resched: 697463 single: 619512/619760 single_ofl: 255751/256554 > single_rpc: 620692 single_rpc_ofl: 0 many: 155476/154658 all: > 77282/76988 onoff: 3/3:5/6 18,25:9,28 63:93 (HZ=100) ste: 0 stnmie: 0 > stnmoe: 0 staf: 0 > [ 121.454485][ T57] scftorture: --- End of test: LOCK_HOTPLUG: > verbose=1 holdoff=10 longwait=0 nthreads=4 onoff_holdoff=30 > onoff_interval=1000 shutdown_secs=1 stat_interval=15 stutter=5 > use_cpus_read_lock=0, weight_resched=-1, weight_single=-1, > weight_single_rpc=-1, weight_single_wait=-1, weight_many=-1, > weight_many_wait=-1, weight_all=-1, weight_all_wait=-1 > [ 121.469305][ T57] reboot: Power down > > I see "End of test: LOCK_HOTPLUG", which means the function > torture_offline in kernel torture.c failed to bring down the cpu. > 4) Then I chase the reason down to tick_nohz_cpu_down: > if (tick_nohz_full_running && tick_do_timer_cpu == cpu) > return -EBUSY; > 5) I create above patch > > > > > Tested on PPC VM of Open Source Lab of Oregon State University. > > > The test results show that after the fix, the success rate of > > > rcutorture is improved. > > > After: > > > Successes: 40 Failures: 9 > > > Before: > > > Successes: 38 Failures: 11 > > > > > > I examined the console.log and Make.out files one by one, no new > > > compile error or test error is introduced by above fix. > > > > > > Signed-off-by: Zhouyi Zhou > > > --- > > > Dear PPC developers > > > > > > I found this bug when trying to do rcutorture tests in ppc VM of > > > Open Source Lab of Oregon State University: > > > > > > ubuntu@ubuntu:~/linux-next/tools/testing/selftests/rcutorture/res/2022.09.30-01.06.22-torture$ > > > find . -name "console.log.diags"|xargs grep HOTPLUG > > > ./results-scftorture/NOPREEMPT/console.log.diags:WARNING: HOTPLUG > > > FAILURES NOPREEMPT > > > ./results-rcutorture/TASKS03/console.log.diags:WARNING: HOTPLUG FAILURES > > > TASKS03 > > > ./results-rcutorture/TREE04/console.log.diags:WARNING: HOTPLUG FAILURES > > > TREE04 > > > ./results-scftorture-kasan/NOPREEMPT/console.log.diags:WARNING: HOTPLUG > > > FAILURES NOPREEMPT > > > ./results-rcutorture-kasan/TASKS03/console.log.diags:WARNING: HOTPLUG > > > FAILURES TASKS03 > > > ./results-rcutorture-kasan/TREE04/console.log.diags:WARNING: HOTPLUG > > > FAILURES TREE04 > > > > > > I tried to fix this bug. > > > > > > Thanks for your patience and guidance ;-) > > > > > > Thanks > > > Zhouyi > > > -- > > > arch/powerpc/kernel/sysfs.c | 8 +++- > > > 1 file changed, 7 insertions(+), 1 deletion(-) > > > > > > diff --git a/arch/powerpc/kernel/sysfs.c b/arch/powerpc/kernel/sysfs.c > > > index ef9a61718940..be9c0e45337e 100644 > > > --- a/arch/powerpc/kernel/sysfs.c > > > +++ b/arch/powerpc/kernel/sysfs.c > > > @@ -4,6 +4,7 @@ > > > #include > > > #include > > > #include > > > +#include > > > #include > > > #include > > > #include > > > @@ -21,6 +22,7 @@ > > > #include > > > #include > > > #include > > > +#include "../../../kernel/time/tick-internal.h" > > > > Needing to include this internal header is a sign that we are using the > >
[PATCH linux-next][RFC]powerpc: move pseries interrupt cleanup code to under __cpu_disable
According to
Link:
https://lore.kernel.org/lkml/20220914021528.15946-1-zhouzho...@gmail.com/T/
Link: https://lore.kernel.org/lkml/CN6WCMKCWHOG.LT2QV3910UJ2@bobo/
During the cpu offlining, the sub functions of xive_teardown_cpu will
call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will
travel RCU protected list, so "WARNING: suspicious RCU usage" will be
triggered.
According to Documentation/core-api/cpu_hotplug.rst:
__cpu_disable should shut down the interrupt handler.
This patch move pseries interrupt cleanup code to under __cpu_disable.
RCU torture tested in ppc VM of Open Source Lab of Oregon State University,
by comparing the test results, our fix don't introduce new bugs and fixed
previous "WARNING: suspicious RCU usage" bugs.
Suggested-by: Nicholas Piggin
Signed-off-by: Zhouyi Zhou
---
Dear PPC and RCU developers
I tries do move pseries interrupt cleanup code to under __cpu_disable.
During this process, I learn the XIVE interrupt controller architecture
by debugging the qemu virtual machine, and reading the following documents:
Link: https://www.qemu.org/docs/master/specs/ppc-spapr-xive.html
Link: https://www.qemu.org/docs/master/specs/ppc-xive.html
It is a fruitful journey, thank you for you guidance ;-)
I also tested the patch in ppc VM of Open Source Lab of Oregon
State University by invoking
tools/testing/selftests/rcutorture/bin/torture.sh
Thanks again
Zhouyi
--
arch/powerpc/platforms/pseries/hotplug-cpu.c | 12 +++-
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c
b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index e0a7ac5db15d..c7c86ea0a74d 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -62,12 +62,7 @@ static void pseries_cpu_offline_self(void)
{
unsigned int hwcpu = hard_smp_processor_id();
- local_irq_disable();
idle_task_exit();
- if (xive_enabled())
- xive_teardown_cpu();
- else
- xics_teardown_cpu();
unregister_slb_shadow(hwcpu);
rtas_stop_self();
@@ -96,3 +91,10 @@ static int pseries_cpu_disable(void)
cleanup_cpu_mmu_context();
+ local_irq_disable();
+
+ if (xive_enabled())
+ xive_teardown_cpu();
+ else
+ xics_teardown_cpu();
+
return 0;
}
--
2.25.1
Re: [PATCH linux-next][RFC] powerpc: fix HOTPLUG error in rcutorture
Thanks Michael for reviewing my patch On Mon, Oct 10, 2022 at 7:21 PM Michael Ellerman wrote: > > Zhouyi Zhou writes: > > I think we should avoid torture offline the cpu who do tick timer > > when nohz full is running. > > Can you tell us what the bug you're fixing is? > > Did you see a crash/oops/hang etc? Or are you just proposing this as > something that would be a good idea? Sorry for the trouble and inconvenience that I bring to the community. I haven't made myself clear in my patch. The ins and outs are as follows: 1) cd linux-next 2) ./tools/testing/selftests/rcutorture/bin/torture.sh after 19 hours ;-) 3) tail ./tools/testing/selftests/rcutorture/res/2022.09.30-01.06.22-torture/results-scftorture/NOPREEMPT/console.log [ 121.449268][ T57] scftorture: scf_invoked_count VER: 2415215 resched: 697463 single: 619512/619760 single_ofl: 255751/256554 single_rpc: 620692 single_rpc_ofl: 0 many: 155476/154658 all: 77282/76988 onoff: 3/3:5/6 18,25:9,28 63:93 (HZ=100) ste: 0 stnmie: 0 stnmoe: 0 staf: 0 [ 121.454485][ T57] scftorture: --- End of test: LOCK_HOTPLUG: verbose=1 holdoff=10 longwait=0 nthreads=4 onoff_holdoff=30 onoff_interval=1000 shutdown_secs=1 stat_interval=15 stutter=5 use_cpus_read_lock=0, weight_resched=-1, weight_single=-1, weight_single_rpc=-1, weight_single_wait=-1, weight_many=-1, weight_many_wait=-1, weight_all=-1, weight_all_wait=-1 [ 121.469305][ T57] reboot: Power down I see "End of test: LOCK_HOTPLUG", which means the function torture_offline in kernel torture.c failed to bring down the cpu. 4) Then I chase the reason down to tick_nohz_cpu_down: if (tick_nohz_full_running && tick_do_timer_cpu == cpu) return -EBUSY; 5) I create above patch > > > Tested on PPC VM of Open Source Lab of Oregon State University. > > The test results show that after the fix, the success rate of > > rcutorture is improved. > > After: > > Successes: 40 Failures: 9 > > Before: > > Successes: 38 Failures: 11 > > > > I examined the console.log and Make.out files one by one, no new > > compile error or test error is introduced by above fix. > > > > Signed-off-by: Zhouyi Zhou > > --- > > Dear PPC developers > > > > I found this bug when trying to do rcutorture tests in ppc VM of > > Open Source Lab of Oregon State University: > > > > ubuntu@ubuntu:~/linux-next/tools/testing/selftests/rcutorture/res/2022.09.30-01.06.22-torture$ > > find . -name "console.log.diags"|xargs grep HOTPLUG > > ./results-scftorture/NOPREEMPT/console.log.diags:WARNING: HOTPLUG FAILURES > > NOPREEMPT > > ./results-rcutorture/TASKS03/console.log.diags:WARNING: HOTPLUG FAILURES > > TASKS03 > > ./results-rcutorture/TREE04/console.log.diags:WARNING: HOTPLUG FAILURES > > TREE04 > > ./results-scftorture-kasan/NOPREEMPT/console.log.diags:WARNING: HOTPLUG > > FAILURES NOPREEMPT > > ./results-rcutorture-kasan/TASKS03/console.log.diags:WARNING: HOTPLUG > > FAILURES TASKS03 > > ./results-rcutorture-kasan/TREE04/console.log.diags:WARNING: HOTPLUG > > FAILURES TREE04 > > > > I tried to fix this bug. > > > > Thanks for your patience and guidance ;-) > > > > Thanks > > Zhouyi > > -- > > arch/powerpc/kernel/sysfs.c | 8 +++- > > 1 file changed, 7 insertions(+), 1 deletion(-) > > > > diff --git a/arch/powerpc/kernel/sysfs.c b/arch/powerpc/kernel/sysfs.c > > index ef9a61718940..be9c0e45337e 100644 > > --- a/arch/powerpc/kernel/sysfs.c > > +++ b/arch/powerpc/kernel/sysfs.c > > @@ -4,6 +4,7 @@ > > #include > > #include > > #include > > +#include > > #include > > #include > > #include > > @@ -21,6 +22,7 @@ > > #include > > #include > > #include > > +#include "../../../kernel/time/tick-internal.h" > > Needing to include this internal header is a sign that we are using the > wrong API or otherwise using time keeping internals we shouldn't be. Yes, when I do this, I guess there is something wrong in my patch. > > > #include "cacheinfo.h" > > #include "setup.h" > > @@ -1151,7 +1153,11 @@ static int __init topology_init(void) > >* CPU. For instance, the boot cpu might never be valid > >* for hotplugging. > >*/ > > - if (smp_ops && smp_ops->cpu_offline_self) > > + if (smp_ops && smp_ops->cpu_offline_self > > +#ifdef CONFIG_NO_HZ_FULL > > + && !(tick_nohz_full_running && tick_do_timer_cpu == cpu) > > +#endif > > + ) > >
Re: [PATCH linux-next][RFC] powerpc: avoid lockdep when we are offline
On Mon, Oct 10, 2022 at 11:49 AM Nicholas Piggin wrote: > > On Thu Sep 29, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > On Wed, Sep 28, 2022 at 10:51 AM Nicholas Piggin wrote: > > > > > > On Wed Sep 28, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > > > Thank Nick for reviewing my patch > > > > > > > > On Tue, Sep 27, 2022 at 12:25 PM Nicholas Piggin > > > > wrote: > > > > > > > > > > On Tue Sep 27, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > > > > > This is second version of my fix to PPC's "WARNING: suspicious RCU > > > > > > usage", > > > > > > I improved my fix under Paul E. McKenney's guidance: > > > > > > Link: > > > > > > https://lore.kernel.org/lkml/20220914021528.15946-1-zhouzho...@gmail.com/T/ > > > > > > > > > > > > During the cpu offlining, the sub functions of xive_teardown_cpu > > > > > > will > > > > > > call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will > > > > > > travel RCU protected list, so "WARNING: suspicious RCU usage" will > > > > > > be > > > > > > triggered. > > > > > > > > > > > > Avoid lockdep when we are offline. > > > > > > > > > > I don't see how this is safe. If RCU is no longer watching the CPU > > > > > then > > > > > the memory it is accessing here could be concurrently freed. I think > > > > > the > > > > > warning is valid. > > > > Agree > > > > > > > > > > powerpc's problem is that cpuhp_report_idle_dead() is called before > > > > > arch_cpu_idle_dead(), so it must not rely on any RCU protection there. > > > > > I would say xive cleanup just needs to be done earlier. I wonder why > > > > > it > > > > > is not done in __cpu_disable or thereabouts, that's where the > > > > > interrupt > > > > > controller is supposed to be stopped. > > > > Yes, I learn flowing events sequence from kgdb debugging > > > > __cpu_disable -> pseries_cpu_disable -> set_cpu_online(cpu, false) = > > > > leads to => do_idle: if (cpu_is_offline(cpu) -> arch_cpu_idle_dead > > > > so xive cleanup should be done in pseries_cpu_disable. > > > > > > It's a good catch and a reasonable approach to the problem. > > Thank Nick for your encouragement ;-) > > > > > > > But as a beginner, I afraid that I am incompetent to do above > > > > sophisticated work without error although I am very like to, > > > > Could any expert do this for us? > > > > > > This will be difficult for anybody, it's tricky code. I'm not an > > > expert at it. > > > > > > It looks like the interrupt controller disable split has been there > > > since long before xive. I would try just move them together than see > > > if that works. > > Yes, I use "git blame" (I learned "git blame" from Paul E. McKenny ;-) > > ) to see the same. > > and anticipate your great works! > > I was thinking you could try it and see if it works and what you find. > If you are interested and have time to look into it? I am interested! and I have time ;-) Thank Nick for your trust in me! I am going to submit my babyish work in about a month (counting the rcutoture tests time), and thank you in advance for your patience. Cheers Zhouyi > > Thanks, > Nick
[PATCH linux-next][RFC] powerpc: fix HOTPLUG error in rcutorture
I think we should avoid torture offline the cpu who do tick timer when nohz full is running. Tested on PPC VM of Open Source Lab of Oregon State University. The test results show that after the fix, the success rate of rcutorture is improved. After: Successes: 40 Failures: 9 Before: Successes: 38 Failures: 11 I examined the console.log and Make.out files one by one, no new compile error or test error is introduced by above fix. Signed-off-by: Zhouyi Zhou --- Dear PPC developers I found this bug when trying to do rcutorture tests in ppc VM of Open Source Lab of Oregon State University: ubuntu@ubuntu:~/linux-next/tools/testing/selftests/rcutorture/res/2022.09.30-01.06.22-torture$ find . -name "console.log.diags"|xargs grep HOTPLUG ./results-scftorture/NOPREEMPT/console.log.diags:WARNING: HOTPLUG FAILURES NOPREEMPT ./results-rcutorture/TASKS03/console.log.diags:WARNING: HOTPLUG FAILURES TASKS03 ./results-rcutorture/TREE04/console.log.diags:WARNING: HOTPLUG FAILURES TREE04 ./results-scftorture-kasan/NOPREEMPT/console.log.diags:WARNING: HOTPLUG FAILURES NOPREEMPT ./results-rcutorture-kasan/TASKS03/console.log.diags:WARNING: HOTPLUG FAILURES TASKS03 ./results-rcutorture-kasan/TREE04/console.log.diags:WARNING: HOTPLUG FAILURES TREE04 I tried to fix this bug. Thanks for your patience and guidance ;-) Thanks Zhouyi -- arch/powerpc/kernel/sysfs.c | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/sysfs.c b/arch/powerpc/kernel/sysfs.c index ef9a61718940..be9c0e45337e 100644 --- a/arch/powerpc/kernel/sysfs.c +++ b/arch/powerpc/kernel/sysfs.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,7 @@ #include #include #include +#include "../../../kernel/time/tick-internal.h" #include "cacheinfo.h" #include "setup.h" @@ -1151,7 +1153,11 @@ static int __init topology_init(void) * CPU. For instance, the boot cpu might never be valid * for hotplugging. */ - if (smp_ops && smp_ops->cpu_offline_self) + if (smp_ops && smp_ops->cpu_offline_self +#ifdef CONFIG_NO_HZ_FULL + && !(tick_nohz_full_running && tick_do_timer_cpu == cpu) +#endif + ) c->hotpluggable = 1; #endif -- 2.25.1
Re: [PATCH linux-next][RFC] powerpc: avoid lockdep when we are offline
On Wed, Sep 28, 2022 at 10:51 AM Nicholas Piggin wrote: > > On Wed Sep 28, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > Thank Nick for reviewing my patch > > > > On Tue, Sep 27, 2022 at 12:25 PM Nicholas Piggin wrote: > > > > > > On Tue Sep 27, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > > > This is second version of my fix to PPC's "WARNING: suspicious RCU > > > > usage", > > > > I improved my fix under Paul E. McKenney's guidance: > > > > Link: > > > > https://lore.kernel.org/lkml/20220914021528.15946-1-zhouzho...@gmail.com/T/ > > > > > > > > During the cpu offlining, the sub functions of xive_teardown_cpu will > > > > call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will > > > > travel RCU protected list, so "WARNING: suspicious RCU usage" will be > > > > triggered. > > > > > > > > Avoid lockdep when we are offline. > > > > > > I don't see how this is safe. If RCU is no longer watching the CPU then > > > the memory it is accessing here could be concurrently freed. I think the > > > warning is valid. > > Agree > > > > > > powerpc's problem is that cpuhp_report_idle_dead() is called before > > > arch_cpu_idle_dead(), so it must not rely on any RCU protection there. > > > I would say xive cleanup just needs to be done earlier. I wonder why it > > > is not done in __cpu_disable or thereabouts, that's where the interrupt > > > controller is supposed to be stopped. > > Yes, I learn flowing events sequence from kgdb debugging > > __cpu_disable -> pseries_cpu_disable -> set_cpu_online(cpu, false) = > > leads to => do_idle: if (cpu_is_offline(cpu) -> arch_cpu_idle_dead > > so xive cleanup should be done in pseries_cpu_disable. > > It's a good catch and a reasonable approach to the problem. Thank Nick for your encouragement ;-) > > > But as a beginner, I afraid that I am incompetent to do above > > sophisticated work without error although I am very like to, > > Could any expert do this for us? > > This will be difficult for anybody, it's tricky code. I'm not an > expert at it. > > It looks like the interrupt controller disable split has been there > since long before xive. I would try just move them together than see > if that works. Yes, I use "git blame" (I learned "git blame" from Paul E. McKenny ;-) ) to see the same. and anticipate your great works! > > Documentation/core-api/cpu_hotplug.rst says that __cpu_disable should > shut down the interrupt handler. So if there is a complication it > would probably be from powerpc-specific CPU hotplug or interrupt > code. Thank Nick for your guidance! I studied Documentation/core-api/cpu_hotplug.rst this morning. I also found X86 shut down the interrupt handler in __cpu_disable according to above document. Many Thanks Zhouyi > > Thanks, > Nick >
Re: [PATCH linux-next][RFC] powerpc: avoid lockdep when we are offline
Thank Nick for reviewing my patch On Tue, Sep 27, 2022 at 12:25 PM Nicholas Piggin wrote: > > On Tue Sep 27, 2022 at 11:48 AM AEST, Zhouyi Zhou wrote: > > This is second version of my fix to PPC's "WARNING: suspicious RCU usage", > > I improved my fix under Paul E. McKenney's guidance: > > Link: > > https://lore.kernel.org/lkml/20220914021528.15946-1-zhouzho...@gmail.com/T/ > > > > During the cpu offlining, the sub functions of xive_teardown_cpu will > > call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will > > travel RCU protected list, so "WARNING: suspicious RCU usage" will be > > triggered. > > > > Avoid lockdep when we are offline. > > I don't see how this is safe. If RCU is no longer watching the CPU then > the memory it is accessing here could be concurrently freed. I think the > warning is valid. Agree > > powerpc's problem is that cpuhp_report_idle_dead() is called before > arch_cpu_idle_dead(), so it must not rely on any RCU protection there. > I would say xive cleanup just needs to be done earlier. I wonder why it > is not done in __cpu_disable or thereabouts, that's where the interrupt > controller is supposed to be stopped. Yes, I learn flowing events sequence from kgdb debugging __cpu_disable -> pseries_cpu_disable -> set_cpu_online(cpu, false) = leads to => do_idle: if (cpu_is_offline(cpu) -> arch_cpu_idle_dead so xive cleanup should be done in pseries_cpu_disable. But as a beginner, I afraid that I am incompetent to do above sophisticated work without error although I am very like to, Could any expert do this for us? Thanks a lot Cheers Zhouyi > > Thanks, > Nick > > > > > Signed-off-by: Zhouyi Zhou > > --- > > Dear PPC and RCU developers > > > > I found this bug when trying to do rcutorture tests in ppc VM of > > Open Source Lab of Oregon State University > > > > console.log report following bug: > > [ 37.635545][T0] WARNING: suspicious RCU usage^M > > [ 37.636409][T0] 6.0.0-rc4-next-20220907-dirty #8 Not tainted^M > > [ 37.637575][T0] -^M > > [ 37.638306][T0] kernel/locking/lockdep.c:3723 RCU-list traversed in > > non-reader section!!^M > > [ 37.639651][T0] ^M > > [ 37.639651][T0] other info that might help us debug this:^M > > [ 37.639651][T0] ^M > > [ 37.641381][T0] ^M > > [ 37.641381][T0] RCU used illegally from offline CPU!^M > > [ 37.641381][T0] rcu_scheduler_active = 2, debug_locks = 1^M > > [ 37.667170][T0] no locks held by swapper/6/0.^M > > [ 37.668328][T0] ^M > > [ 37.668328][T0] stack backtrace:^M > > [ 37.669995][T0] CPU: 6 PID: 0 Comm: swapper/6 Not tainted > > 6.0.0-rc4-next-20220907-dirty #8^M > > [ 37.672777][T0] Call Trace:^M > > [ 37.673729][T0] [c4653920] [c097f9b4] > > dump_stack_lvl+0x98/0xe0 (unreliable)^M > > [ 37.678579][T0] [c4653960] [c01f2eb8] > > lockdep_rcu_suspicious+0x148/0x16c^M > > [ 37.680425][T0] [c46539f0] [c01ed9b4] > > __lock_acquire+0x10f4/0x26e0^M > > [ 37.682450][T0] [c4653b30] [c01efc2c] > > lock_acquire+0x12c/0x420^M > > [ 37.684113][T0] [c4653c20] [c10d704c] > > _raw_spin_lock_irqsave+0x6c/0xc0^M > > [ 37.686154][T0] [c4653c60] [c00c7b4c] > > xive_spapr_put_ipi+0xcc/0x150^M > > [ 37.687879][T0] [c4653ca0] [c10c72a8] > > xive_cleanup_cpu_ipi+0xc8/0xf0^M > > [ 37.689856][T0] [c4653cf0] [c10c7370] > > xive_teardown_cpu+0xa0/0xf0^M > > [ 37.691877][T0] [c4653d30] [c00fba5c] > > pseries_cpu_offline_self+0x5c/0x100^M > > [ 37.693882][T0] [c4653da0] [c005d2c4] > > arch_cpu_idle_dead+0x44/0x60^M > > [ 37.695739][T0] [c4653dc0] [c01c740c] > > do_idle+0x16c/0x3d0^M > > [ 37.697536][T0] [c4653e70] [c01c7a1c] > > cpu_startup_entry+0x3c/0x40^M > > [ 37.699694][T0] [c4653ea0] [c005ca20] > > start_secondary+0x6c0/0xb50^M > > [ 37.701742][T0] [c4653f90] [c000d054] > > start_secondary_prolog+0x10/0x14^M > > > > > > Tested on PPC VM of Open Source Lab of Oregon State University. > > Test results show that although "WARNING: suspicious RCU usage" has gone, > > and there are less "BUG: soft lockup" reports than the original kernel > > (9 vs 13), which sounds good ;-) > > > > But af
[PATCH linux-next][RFC] powerpc: avoid lockdep when we are offline
This is second version of my fix to PPC's "WARNING: suspicious RCU usage", I improved my fix under Paul E. McKenney's guidance: Link: https://lore.kernel.org/lkml/20220914021528.15946-1-zhouzho...@gmail.com/T/ During the cpu offlining, the sub functions of xive_teardown_cpu will call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will travel RCU protected list, so "WARNING: suspicious RCU usage" will be triggered. Avoid lockdep when we are offline. Signed-off-by: Zhouyi Zhou --- Dear PPC and RCU developers I found this bug when trying to do rcutorture tests in ppc VM of Open Source Lab of Oregon State University console.log report following bug: [ 37.635545][T0] WARNING: suspicious RCU usage^M [ 37.636409][T0] 6.0.0-rc4-next-20220907-dirty #8 Not tainted^M [ 37.637575][T0] -^M [ 37.638306][T0] kernel/locking/lockdep.c:3723 RCU-list traversed in non-reader section!!^M [ 37.639651][T0] ^M [ 37.639651][T0] other info that might help us debug this:^M [ 37.639651][T0] ^M [ 37.641381][T0] ^M [ 37.641381][T0] RCU used illegally from offline CPU!^M [ 37.641381][T0] rcu_scheduler_active = 2, debug_locks = 1^M [ 37.667170][T0] no locks held by swapper/6/0.^M [ 37.668328][T0] ^M [ 37.668328][T0] stack backtrace:^M [ 37.669995][T0] CPU: 6 PID: 0 Comm: swapper/6 Not tainted 6.0.0-rc4-next-20220907-dirty #8^M [ 37.672777][T0] Call Trace:^M [ 37.673729][T0] [c4653920] [c097f9b4] dump_stack_lvl+0x98/0xe0 (unreliable)^M [ 37.678579][T0] [c4653960] [c01f2eb8] lockdep_rcu_suspicious+0x148/0x16c^M [ 37.680425][T0] [c46539f0] [c01ed9b4] __lock_acquire+0x10f4/0x26e0^M [ 37.682450][T0] [c4653b30] [c01efc2c] lock_acquire+0x12c/0x420^M [ 37.684113][T0] [c4653c20] [c10d704c] _raw_spin_lock_irqsave+0x6c/0xc0^M [ 37.686154][T0] [c4653c60] [c00c7b4c] xive_spapr_put_ipi+0xcc/0x150^M [ 37.687879][T0] [c4653ca0] [c10c72a8] xive_cleanup_cpu_ipi+0xc8/0xf0^M [ 37.689856][T0] [c4653cf0] [c10c7370] xive_teardown_cpu+0xa0/0xf0^M [ 37.691877][T0] [c4653d30] [c00fba5c] pseries_cpu_offline_self+0x5c/0x100^M [ 37.693882][T0] [c4653da0] [c005d2c4] arch_cpu_idle_dead+0x44/0x60^M [ 37.695739][T0] [c4653dc0] [c01c740c] do_idle+0x16c/0x3d0^M [ 37.697536][T0] [c4653e70] [c01c7a1c] cpu_startup_entry+0x3c/0x40^M [ 37.699694][T0] [c4653ea0] [c005ca20] start_secondary+0x6c0/0xb50^M [ 37.701742][T0] [c4653f90] [c000d054] start_secondary_prolog+0x10/0x14^M Tested on PPC VM of Open Source Lab of Oregon State University. Test results show that although "WARNING: suspicious RCU usage" has gone, and there are less "BUG: soft lockup" reports than the original kernel (9 vs 13), which sounds good ;-) But after my modification, results-rcutorture-kasan/SRCU-P/console.log.diags shows a new warning: [ 222.289242][ T110] WARNING: CPU: 6 PID: 110 at kernel/rcu/rcutorture.c:2806 rcu_torture_fwd_prog+0xc88/0xdd0 I guess above new warning also exits in original kernel, so I write a tiny test script as follows: #!/bin/sh COUNTER=0 while [ $COUNTER -lt 1000 ] ; do qemu-system-ppc64 -nographic -smp cores=8,threads=1 -net none -M pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log -m 2G -kernel /tmp/vmlinux -append "debug_boot_weak_hash panic=-1 console=ttyS0 rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot rcupdate.rcu_task_stall_timeout=3 rcutorture.torture_type=srcud rcupdate.rcu_self_test=1 rcutorture.fwd_progress=3 srcutree.big_cpu_lim=5 rcutorture.onoff_interval=1000 rcutorture.onoff_holdoff=30 rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 rcutorture.shutdown_secs=420 rcutorture.test_no_idle_hz=1 rcutorture.verbose=1"& qemu_pid=$! cd ~/next1/linux-next make clean #I use "make vmlinux -j 8" to create heavy background jitter make vmlinux -j 8 > /dev/null 2>&1 make_pid=$! wait $qemu_pid kill $qemu_pid kill $make_id if grep -q WARN /tmp/console.log; then echo $COUNTER > /tmp/counter exit fi COUNTER=$(($COUNTER+1)) done Above test shows that original kernel also warn about "WARNING: CPU: 6 PID: 110 at kernel/rcu/rcutorture.c:2806 rcu_torture_fwd_prog+0xc88/0xdd0" But I am not very sure about my results, so I still add a [RFC] to my subject line. Thank all of you for your guidance and encouragement ;-) Cheers Zhouyi -- arch/powerpc/platforms/pseries/hotplug-cpu.c | 5 + 1 file changed, 5 insertions(+) diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c b/arch/powerpc/platforms/pse
Re: [PATCH linux-next][RFC] powerpc: protect cpu offlining by RCU offline lock
On Wed, Sep 14, 2022 at 8:17 PM Paul E. McKenney wrote: > > On Wed, Sep 14, 2022 at 10:15:28AM +0800, Zhouyi Zhou wrote: > > During the cpu offlining, the sub functions of xive_teardown_cpu will > > call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will > > travel RCU protected list, so "WARNING: suspicious RCU usage" will be > > triggered. > > > > Try to protect cpu offlining by RCU offline lock. > Thank Paul for your guidance! > Rather than acquiring the RCU lock, why not change the functions called > by xive_teardown_cpu() to avoid calling __lock_acquire()? For example, > a call to spin_lock() could be changed to arch_spin_lock(). Great idea! I will take a try, and perform new rounds of rcutorture tests. I will submit a new version next week. Also thank PPC developers for your patience on me ;-) Cheers Zhouyi > > Thanx, Paul > > > Tested on PPC VM of Open Source Lab of Oregon State University. > > (Each round of tests takes about 19 hours to finish) > > Test results show that although "WARNING: suspicious RCU usage" has gone, > > but there are more "BUG: soft lockup" reports than the original kernel > > (10 vs 6), so I add a [RFC] to my subject line. > > > > Signed-off-by: Zhouyi Zhou > > --- > > [it seems that there are some delivery problem in my previous email, > > so I send again via gmail, sorry for the trouble] > > > > Dear PPC and RCU developers > > > > I found this bug when trying to do rcutorture tests in ppc VM of > > Open Source Lab of Oregon State University. > > > > console.log report following bug: > > [ 37.635545][T0] WARNING: suspicious RCU usage^M > > [ 37.636409][T0] 6.0.0-rc4-next-20220907-dirty #8 Not tainted^M > > [ 37.637575][T0] -^M > > [ 37.638306][T0] kernel/locking/lockdep.c:3723 RCU-list traversed in > > non-reader section!!^M > > [ 37.639651][T0] ^M > > [ 37.639651][T0] other info that might help us debug this:^M > > [ 37.639651][T0] ^M > > [ 37.641381][T0] ^M > > [ 37.641381][T0] RCU used illegally from offline CPU!^M > > [ 37.641381][T0] rcu_scheduler_active = 2, debug_locks = 1^M > > [ 37.667170][T0] no locks held by swapper/6/0.^M > > [ 37.668328][T0] ^M > > [ 37.668328][T0] stack backtrace:^M > > [ 37.669995][T0] CPU: 6 PID: 0 Comm: swapper/6 Not tainted > > 6.0.0-rc4-next-20220907-dirty #8^M > > [ 37.672777][T0] Call Trace:^M > > [ 37.673729][T0] [c4653920] [c097f9b4] > > dump_stack_lvl+0x98/0xe0 (unreliable)^M > > [ 37.678579][T0] [c4653960] [c01f2eb8] > > lockdep_rcu_suspicious+0x148/0x16c^M > > [ 37.680425][T0] [c46539f0] [c01ed9b4] > > __lock_acquire+0x10f4/0x26e0^M > > [ 37.682450][T0] [c4653b30] [c01efc2c] > > lock_acquire+0x12c/0x420^M > > [ 37.684113][T0] [c4653c20] [c10d704c] > > _raw_spin_lock_irqsave+0x6c/0xc0^M > > [ 37.686154][T0] [c4653c60] [c00c7b4c] > > xive_spapr_put_ipi+0xcc/0x150^M > > [ 37.687879][T0] [c4653ca0] [c10c72a8] > > xive_cleanup_cpu_ipi+0xc8/0xf0^M > > [ 37.689856][T0] [c4653cf0] [c10c7370] > > xive_teardown_cpu+0xa0/0xf0^M > > [ 37.691877][T0] [c4653d30] [c00fba5c] > > pseries_cpu_offline_self+0x5c/0x100^M > > [ 37.693882][T0] [c4653da0] [c005d2c4] > > arch_cpu_idle_dead+0x44/0x60^M > > [ 37.695739][T0] [c4653dc0] [c01c740c] > > do_idle+0x16c/0x3d0^M > > [ 37.697536][T0] [c4653e70] [c01c7a1c] > > cpu_startup_entry+0x3c/0x40^M > > [ 37.699694][T0] [c4653ea0] [c005ca20] > > start_secondary+0x6c0/0xb50^M > > [ 37.701742][T0] [c4653f90] [c000d054] > > start_secondary_prolog+0x10/0x14^M > > > > > > I am a beginner, hope I can be of some beneficial to the community ;-) > > > > Thanks > > Zhouyi > > -- > > arch/powerpc/platforms/pseries/hotplug-cpu.c | 5 - > > include/linux/rcupdate.h | 3 ++- > > kernel/rcu/tree.c| 10 ++ > > 3 files changed, 16 insertions(+), 2 deletions(-) > > > > diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c > > b/arch/powerpc/platforms/pseries/hotplug-cpu.c > > index 0f8cd8b06432..ddf66a253
[PATCH linux-next][RFC] powerpc: protect cpu offlining by RCU offline lock
From: Zhouyi Zhou
During the cpu offlining, the sub functions of xive_teardown_cpu will
call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will
travel RCU protected list, so "WARNING: suspicious RCU usage" will be
triggered.
Try to protect cpu offlining by RCU offline lock.
Tested on PPC VM of Open Source Lab of Oregon State University.
(Each round of tests takes about 19 hours to finish)
Test results show that although "WARNING: suspicious RCU usage" has gone,
but there are more "BUG: soft lockup" reports than the original kernel
(10 vs 6), so I add a [RFC] to my subject line.
Signed-off-by: Zhouyi Zhou
---
Dear PPC and RCU developers
I found this bug when trying to do rcutorture tests in ppc VM of
Open Source Lab of Oregon State University.
console.log report following bug:
[ 37.635545][T0] WARNING: suspicious RCU usage^M
[ 37.636409][T0] 6.0.0-rc4-next-20220907-dirty #8 Not tainted^M
[ 37.637575][T0] -^M
[ 37.638306][T0] kernel/locking/lockdep.c:3723 RCU-list traversed in
non-reader section!!^M
[ 37.639651][T0] ^M
[ 37.639651][T0] other info that might help us debug this:^M
[ 37.639651][T0] ^M
[ 37.641381][T0] ^M
[ 37.641381][T0] RCU used illegally from offline CPU!^M
[ 37.641381][T0] rcu_scheduler_active = 2, debug_locks = 1^M
[ 37.667170][T0] no locks held by swapper/6/0.^M
[ 37.668328][T0] ^M
[ 37.668328][T0] stack backtrace:^M
[ 37.669995][T0] CPU: 6 PID: 0 Comm: swapper/6 Not tainted
6.0.0-rc4-next-20220907-dirty #8^M
[ 37.672777][T0] Call Trace:^M
[ 37.673729][T0] [c4653920] [c097f9b4]
dump_stack_lvl+0x98/0xe0 (unreliable)^M
[ 37.678579][T0] [c4653960] [c01f2eb8]
lockdep_rcu_suspicious+0x148/0x16c^M
[ 37.680425][T0] [c46539f0] [c01ed9b4]
__lock_acquire+0x10f4/0x26e0^M
[ 37.682450][T0] [c4653b30] [c01efc2c]
lock_acquire+0x12c/0x420^M
[ 37.684113][T0] [c4653c20] [c10d704c]
_raw_spin_lock_irqsave+0x6c/0xc0^M
[ 37.686154][T0] [c4653c60] [c00c7b4c]
xive_spapr_put_ipi+0xcc/0x150^M
[ 37.687879][T0] [c4653ca0] [c10c72a8]
xive_cleanup_cpu_ipi+0xc8/0xf0^M
[ 37.689856][T0] [c4653cf0] [c10c7370]
xive_teardown_cpu+0xa0/0xf0^M
[ 37.691877][T0] [c4653d30] [c00fba5c]
pseries_cpu_offline_self+0x5c/0x100^M
[ 37.693882][T0] [c4653da0] [c005d2c4]
arch_cpu_idle_dead+0x44/0x60^M
[ 37.695739][T0] [c4653dc0] [c01c740c]
do_idle+0x16c/0x3d0^M
[ 37.697536][T0] [c4653e70] [c01c7a1c]
cpu_startup_entry+0x3c/0x40^M
[ 37.699694][T0] [c4653ea0] [c005ca20]
start_secondary+0x6c0/0xb50^M
[ 37.701742][T0] [c4653f90] [c000d054]
start_secondary_prolog+0x10/0x14^M
I am a beginner, hope I can be of some beneficial to the community ;-)
Thanks
Zhouyi
--
arch/powerpc/platforms/pseries/hotplug-cpu.c | 5 -
include/linux/rcupdate.h | 3 ++-
kernel/rcu/tree.c| 10 ++
3 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c
b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index 0f8cd8b06432..ddf66a253c70 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -64,11 +64,14 @@ static void pseries_cpu_offline_self(void)
local_irq_disable();
idle_task_exit();
+
+ /* Because the cpu is now offline, let rcu know that */
+ rcu_state_ofl_lock();
if (xive_enabled())
xive_teardown_cpu();
else
xics_teardown_cpu();
-
+ rcu_state_ofl_unlock();
unregister_slb_shadow(hwcpu);
rtas_stop_self();
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 63d2e6a60ad7..d857955a02ba 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1034,5 +1034,6 @@ rcu_head_after_call_rcu(struct rcu_head *rhp,
rcu_callback_t f)
/* kernel/ksysfs.c definitions */
extern int rcu_expedited;
extern int rcu_normal;
-
+void rcu_state_ofl_lock(void);
+void rcu_state_ofl_unlock(void);
#endif /* __LINUX_RCUPDATE_H */
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6bb8e72bc815..3282725f1054 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4796,6 +4796,16 @@ void __init rcu_init(void)
(void)start_poll_synchronize_rcu_expedited();
}
+void rcu_state_ofl_lock(void)
+{
+ arch_spin_lock(_state.ofl_lock);
+}
+
+void rcu_state_ofl_unlock(void)
+{
+ arch_spin_unlock(_state.ofl_lock);
+}
+
#include "tree_stall.h"
#include "tree_exp.h"
#include "tree_nocb.h"
--
2.34.1
[PATCH linux-next][RFC] powerpc: protect cpu offlining by RCU offline lock
During the cpu offlining, the sub functions of xive_teardown_cpu will
call __lock_acquire when CONFIG_LOCKDEP=y. The latter function will
travel RCU protected list, so "WARNING: suspicious RCU usage" will be
triggered.
Try to protect cpu offlining by RCU offline lock.
Tested on PPC VM of Open Source Lab of Oregon State University.
(Each round of tests takes about 19 hours to finish)
Test results show that although "WARNING: suspicious RCU usage" has gone,
but there are more "BUG: soft lockup" reports than the original kernel
(10 vs 6), so I add a [RFC] to my subject line.
Signed-off-by: Zhouyi Zhou
---
[it seems that there are some delivery problem in my previous email,
so I send again via gmail, sorry for the trouble]
Dear PPC and RCU developers
I found this bug when trying to do rcutorture tests in ppc VM of
Open Source Lab of Oregon State University.
console.log report following bug:
[ 37.635545][T0] WARNING: suspicious RCU usage^M
[ 37.636409][T0] 6.0.0-rc4-next-20220907-dirty #8 Not tainted^M
[ 37.637575][T0] -^M
[ 37.638306][T0] kernel/locking/lockdep.c:3723 RCU-list traversed in
non-reader section!!^M
[ 37.639651][T0] ^M
[ 37.639651][T0] other info that might help us debug this:^M
[ 37.639651][T0] ^M
[ 37.641381][T0] ^M
[ 37.641381][T0] RCU used illegally from offline CPU!^M
[ 37.641381][T0] rcu_scheduler_active = 2, debug_locks = 1^M
[ 37.667170][T0] no locks held by swapper/6/0.^M
[ 37.668328][T0] ^M
[ 37.668328][T0] stack backtrace:^M
[ 37.669995][T0] CPU: 6 PID: 0 Comm: swapper/6 Not tainted
6.0.0-rc4-next-20220907-dirty #8^M
[ 37.672777][T0] Call Trace:^M
[ 37.673729][T0] [c4653920] [c097f9b4]
dump_stack_lvl+0x98/0xe0 (unreliable)^M
[ 37.678579][T0] [c4653960] [c01f2eb8]
lockdep_rcu_suspicious+0x148/0x16c^M
[ 37.680425][T0] [c46539f0] [c01ed9b4]
__lock_acquire+0x10f4/0x26e0^M
[ 37.682450][T0] [c4653b30] [c01efc2c]
lock_acquire+0x12c/0x420^M
[ 37.684113][T0] [c4653c20] [c10d704c]
_raw_spin_lock_irqsave+0x6c/0xc0^M
[ 37.686154][T0] [c4653c60] [c00c7b4c]
xive_spapr_put_ipi+0xcc/0x150^M
[ 37.687879][T0] [c4653ca0] [c10c72a8]
xive_cleanup_cpu_ipi+0xc8/0xf0^M
[ 37.689856][T0] [c4653cf0] [c10c7370]
xive_teardown_cpu+0xa0/0xf0^M
[ 37.691877][T0] [c4653d30] [c00fba5c]
pseries_cpu_offline_self+0x5c/0x100^M
[ 37.693882][T0] [c4653da0] [c005d2c4]
arch_cpu_idle_dead+0x44/0x60^M
[ 37.695739][T0] [c4653dc0] [c01c740c]
do_idle+0x16c/0x3d0^M
[ 37.697536][T0] [c4653e70] [c01c7a1c]
cpu_startup_entry+0x3c/0x40^M
[ 37.699694][T0] [c4653ea0] [c005ca20]
start_secondary+0x6c0/0xb50^M
[ 37.701742][T0] [c4653f90] [c000d054]
start_secondary_prolog+0x10/0x14^M
I am a beginner, hope I can be of some beneficial to the community ;-)
Thanks
Zhouyi
--
arch/powerpc/platforms/pseries/hotplug-cpu.c | 5 -
include/linux/rcupdate.h | 3 ++-
kernel/rcu/tree.c| 10 ++
3 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/pseries/hotplug-cpu.c
b/arch/powerpc/platforms/pseries/hotplug-cpu.c
index 0f8cd8b06432..ddf66a253c70 100644
--- a/arch/powerpc/platforms/pseries/hotplug-cpu.c
+++ b/arch/powerpc/platforms/pseries/hotplug-cpu.c
@@ -64,11 +64,14 @@ static void pseries_cpu_offline_self(void)
local_irq_disable();
idle_task_exit();
+
+ /* Because the cpu is now offline, let rcu know that */
+ rcu_state_ofl_lock();
if (xive_enabled())
xive_teardown_cpu();
else
xics_teardown_cpu();
-
+ rcu_state_ofl_unlock();
unregister_slb_shadow(hwcpu);
rtas_stop_self();
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 63d2e6a60ad7..d857955a02ba 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -1034,5 +1034,6 @@ rcu_head_after_call_rcu(struct rcu_head *rhp,
rcu_callback_t f)
/* kernel/ksysfs.c definitions */
extern int rcu_expedited;
extern int rcu_normal;
-
+void rcu_state_ofl_lock(void);
+void rcu_state_ofl_unlock(void);
#endif /* __LINUX_RCUPDATE_H */
diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6bb8e72bc815..3282725f1054 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -4796,6 +4796,16 @@ void __init rcu_init(void)
(void)start_poll_synchronize_rcu_expedited();
}
+void rcu_state_ofl_lock(void)
+{
+ arch_spin_lock(_state.ofl_lock);
+}
+
+void rcu_state_ofl_unlock(void)
+{
+ arch_spin_unlock(_state.ofl_lock);
+}
+
#include "tree_stall.h"
#include "tree_exp.h"
#include "tree_nocb.h"
--
2.34.1
Re: [PATCH linux-next] powerpc: disable sanitizer in irq_soft_mask_set
On Wed, Aug 24, 2022 at 12:50 AM Christophe Leroy
wrote:
>
>
>
> Le 23/08/2022 à 10:47, Christophe Leroy a écrit :
> >
> >
> > Le 23/08/2022 à 10:33, Michael Ellerman a écrit :
> >> Zhouyi Zhou writes:
> >>
> >> My worry is that this will force irq_soft_mask_set() out of line, which
> >> we would rather avoid. It's meant to be a fast path.
> >>
> >> In fact with this applied I see nearly 300 out-of-line copies of the
> >> function when building a defconfig, and ~1700 calls to it.
> >>
> >> Normally it is inlined at every call site.
> >>
> >>
> >> So I think I'm inclined to revert ef5b570d3700 ("powerpc/irq: Don't open
> >> code irq_soft_mask helpers").
> >
> > Could you revert it only partially ? In extenso, revert the
> > READ/WRITE_ONCE and bring back the inline asm in irq_soft_mask_return()
> > and irq_soft_mask_set(), but keep other changes.
>
> I sent a patch doing that.
Thank Christophe for the fix. I am very glad to be of benefit to the
community ;-)
Also thank Michael and Paul for your constant encouragement and
guidance, I learned to use objdump to count the number of failed
inline function calls today ;-)
By the way, from my experiments, both gcc-11 and clang-14 behave the
same as Michael has described.
Cheers
Zhouyi
>
> Christophe
Re: [PATCH linux-next] powerpc: disable sanitizer in irq_soft_mask_set
On Mon, Aug 22, 2022 at 2:04 PM Christophe Leroy
wrote:
>
>
>
> Le 21/08/2022 à 03:00, Zhouyi Zhou a écrit :
> > In ppc, compiler based sanitizer will generate instrument instructions
> > around statement WRITE_ONCE(local_paca->irq_soft_mask, mask):
> >
> > 0xc0295cb0 <+0>: addis r2,r12,774
> > 0xc0295cb4 <+4>: addir2,r2,16464
> > 0xc0295cb8 <+8>: mflrr0
> > 0xc0295cbc <+12>: bl 0xc008bb4c
> > 0xc0295cc0 <+16>: mflrr0
> > 0xc0295cc4 <+20>: std r31,-8(r1)
> > 0xc0295cc8 <+24>: addir3,r13,2354
> > 0xc0295ccc <+28>: mr r31,r13
> > 0xc0295cd0 <+32>: std r0,16(r1)
> > 0xc0295cd4 <+36>: stdur1,-48(r1)
> > 0xc0295cd8 <+40>: bl 0xc0609b98 <__asan_store1+8>
> > 0xc0295cdc <+44>: nop
> > 0xc0295ce0 <+48>: li r9,1
> > 0xc0295ce4 <+52>: stb r9,2354(r31)
> > 0xc0295ce8 <+56>: addir1,r1,48
> > 0xc0295cec <+60>: ld r0,16(r1)
> > 0xc0295cf0 <+64>: ld r31,-8(r1)
> > 0xc0295cf4 <+68>: mtlrr0
> >
> > If there is a context switch before "stb r9,2354(r31)", r31 may
> > not equal to r13, in such case, irq soft mask will not work.
> >
> > This patch disable sanitizer in irq_soft_mask_set.
>
> Well spotted, thanks.
Thank Christophe for reviewing my patch and your guidance!
>
> You should add:
>
> Fixes: ef5b570d3700 ("powerpc/irq: Don't open code irq_soft_mask helpers")
OK, I will do it!
>
> By the way, I think the READ_ONCE() likely has the same issue so you
> should fix irq_soft_mask_return() at the same time.
Yes, after disassembling irq_soft_mask_return, she has the same issue
as irq_soft_mask_set.
In addition, I read hw_irq.h by naked eye to search for removed inline
assembly one by one,
I found another place that we could possible enhance (Thank Paul E.
McKenny for teaching me use git blame ;-)):
077fc62b2b66a("powerpc/irq: remove inline assembly in hard_irq_disable macro")
--- a/arch/powerpc/include/asm/hw_irq.h
+++ b/arch/powerpc/include/asm/hw_irq.h
@@ -282,9 +282,7 @@ static inline bool pmi_irq_pending(void)
flags = irq_soft_mask_set_return(IRQS_ALL_DISABLED);\
local_paca->irq_happened |= PACA_IRQ_HARD_DIS; \
if (!arch_irqs_disabled_flags(flags)) { \
- asm ("stdx %%r1, 0, %1 ;" \
-: "=m" (local_paca->saved_r1) \
-: "b" (_paca->saved_r1));\
+ WRITE_ONCE(local_paca->saved_r1, current_stack_pointer);\
trace_hardirqs_off(); \
} \
} while(0)
I wrap the macro hard_irq_disable into a test function and disassemble
it, she has the above issue too:
(gdb) disassemble test002
Dump of assembler code for function test002:
0xc0295db0 <+0>:addis r2,r12,774
0xc0295db4 <+4>:addir2,r2,16464
0xc0295db8 <+8>:mflrr0
0xc0295dbc <+12>:bl 0xc008bacc
0xc0295dc0 <+16>:mflrr0
0xc0295dc4 <+20>:std r30,-16(r1)
0xc0295dc8 <+24>:std r31,-8(r1)
0xc0295dcc <+28>:li r9,2
0xc0295dd0 <+32>:std r0,16(r1)
0xc0295dd4 <+36>:stdur1,-48(r1)
0xc0295dd8 <+40>:mtmsrd r9,1
0xc0295ddc <+44>:addir3,r13,2354
0xc0295de0 <+48>:mr r31,r13
0xc0295de4 <+52>:bl 0xc0609838 <__asan_load1+8>
0xc0295de8 <+56>:nop
0xc0295dec <+60>:li r3,3
0xc0295df0 <+64>:lbz r30,2354(r31)
0xc0295df4 <+68>:bl 0xc028de90
0xc0295df8 <+72>:mr r31,r13
0xc0295dfc <+76>:addir3,r13,2355
0xc0295e00 <+80>:bl 0xc0609838 <__asan_load1+8>
0xc0295e04 <+84>:nop
0xc0295e08 <+88>:lbz r9,2355(r31)
0xc0295e0c <+92>:andi. r10,r30,1
0xc0295e10 <+96>:ori r9,r9,1
0xc0295e14 <+100>:stb r9,2355(r31)
0xc0295e18 <+10
[PATCH linux-next] powerpc: disable sanitizer in irq_soft_mask_set
In ppc, compiler based sanitizer will generate instrument instructions
around statement WRITE_ONCE(local_paca->irq_soft_mask, mask):
0xc0295cb0 <+0>: addis r2,r12,774
0xc0295cb4 <+4>: addir2,r2,16464
0xc0295cb8 <+8>: mflrr0
0xc0295cbc <+12>:bl 0xc008bb4c
0xc0295cc0 <+16>:mflrr0
0xc0295cc4 <+20>:std r31,-8(r1)
0xc0295cc8 <+24>:addir3,r13,2354
0xc0295ccc <+28>:mr r31,r13
0xc0295cd0 <+32>:std r0,16(r1)
0xc0295cd4 <+36>:stdur1,-48(r1)
0xc0295cd8 <+40>:bl 0xc0609b98 <__asan_store1+8>
0xc0295cdc <+44>:nop
0xc0295ce0 <+48>:li r9,1
0xc0295ce4 <+52>:stb r9,2354(r31)
0xc0295ce8 <+56>:addir1,r1,48
0xc0295cec <+60>:ld r0,16(r1)
0xc0295cf0 <+64>:ld r31,-8(r1)
0xc0295cf4 <+68>:mtlrr0
If there is a context switch before "stb r9,2354(r31)", r31 may
not equal to r13, in such case, irq soft mask will not work.
This patch disable sanitizer in irq_soft_mask_set.
Signed-off-by: Zhouyi Zhou
---
Dear PPC developers
I found this bug when trying to do rcutorture tests in ppc VM of
Open Source Lab of Oregon State University following Paul E. McKenny's guidance.
console.log report following bug:
[ 346.527467][ T100] BUG: using smp_processor_id() in preemptible []
code: rcu_torture_rea/100^M
[ 346.529416][ T100] caller is rcu_preempt_deferred_qs_irqrestore+0x74/0xed0^M
[ 346.531157][ T100] CPU: 4 PID: 100 Comm: rcu_torture_rea Tainted: G
W 5.19.0-rc5-next-20220708-dirty #253^M
[ 346.533620][ T100] Call Trace:^M
[ 346.534449][ T100] [c94876c0] [c0ce2b68]
dump_stack_lvl+0xbc/0x108 (unreliable)^M
[ 346.536632][ T100] [c9487710] [c1712954]
check_preemption_disabled+0x154/0x160^M
[ 346.538665][ T100] [c94877a0] [c02ce2d4]
rcu_preempt_deferred_qs_irqrestore+0x74/0xed0^M
[ 346.540830][ T100] [c94878b0] [c02cf3c0]
__rcu_read_unlock+0x290/0x3b0^M
[ 346.542746][ T100] [c9487910] [c02bb330]
rcu_torture_read_unlock+0x30/0xb0^M
[ 346.544779][ T100] [c9487930] [c02b7ff8]
rcutorture_one_extend+0x198/0x810^M
[ 346.546851][ T100] [c9487a10] [c02b8bfc]
rcu_torture_one_read+0x58c/0xc90^M
[ 346.548844][ T100] [c9487ca0] [c02b942c]
rcu_torture_reader+0x12c/0x360^M
[ 346.550784][ T100] [c9487db0] [c01de978]
kthread+0x1e8/0x220^M
[ 346.552555][ T100] [c9487e10] [c000cd54]
ret_from_kernel_thread+0x5c/0x64^M
After 12 days debugging, I finally narrow the problem to irq_soft_mask_set.
I am a beginner, hope I can be of some beneficial to the community ;-)
Thanks
Zhouyi
--
arch/powerpc/include/asm/hw_irq.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/include/asm/hw_irq.h
b/arch/powerpc/include/asm/hw_irq.h
index 26ede09c521d..a5ae8d82cc9d 100644
--- a/arch/powerpc/include/asm/hw_irq.h
+++ b/arch/powerpc/include/asm/hw_irq.h
@@ -121,7 +121,7 @@ static inline notrace unsigned long
irq_soft_mask_return(void)
* for the critical section and as a clobber because
* we changed paca->irq_soft_mask
*/
-static inline notrace void irq_soft_mask_set(unsigned long mask)
+static inline notrace __no_kcsan __no_sanitize_address void
irq_soft_mask_set(unsigned long mask)
{
/*
* The irq mask must always include the STD bit if any are set.
--
2.34.1
Re: [PATCH linux-next] powerpc: init jump label early in ppc 64
On Mon, Jul 25, 2022 at 3:55 PM Michael Ellerman wrote: > > zhouzho...@gmail.com writes: > > From: Zhouyi Zhou > > > > In ppc 64, invoke jump_label_init in setup_feature_keys is too late > > because static key will be used in subroutine of early_init_devtree. > > > > So we can invoke jump_label_init earlier in early_setup. > > We can not move setup_feature_keys backward because its subroutine > > cpu_feature_keys_init depend on data structures initialized in > > early_init_devtree. > > > > Signed-off-by: Zhouyi Zhou > > --- > > Dear PPC developers > > > > I found this bug when trying to do rcutorture tests in ppc VM of > > Open Source Lab of Oregon State University. > > > > qemu-system-ppc64 -nographic -smp cores=8,threads=1 -net none -M pseries > > -nodefaults -device spapr-vscsi -serial > > file:/home/ubuntu/linux-next/tools/testing/selftests/rcutorture/res/2022.07.19-01.18.42-torture/results-rcutorture/TREE03/console.log > > -m 512 -kernel > > /home/ubuntu/linux-next/tools/testing/selftests/rcutorture/res/2022.07.19-01.18.42-torture/results-rcutorture/TREE03/vmlinux > > -append "debug_boot_weak_hash panic=-1 console=ttyS0 > > rcupdate.rcu_cpu_stall_suppress_at_boot=1 torture.disable_onoff_at_boot > > rcupdate.rcu_task_stall_timeout=3 rcutorture.onoff_interval=200 > > rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12 > > rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcutree.kthread_prio=2 > > threadirqs tree.use_softirq=0 rcutorture.n_barrier_cbs=4 > > rcutorture.stat_interval=15 rcutorture.shutdown_secs=420 > > rcutorture.test_no_idle_hz=1 rcutorture.verbose=1" > > > > console.log report following WARN: > > [0.00][T0] static_key_enable_cpuslocked(): static key > > '0xc2953260' used before call to jump_label_init()^M > > [0.00][T0] WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:166 > > static_key_enable_cpuslocked+0xfc/0x120^M > > [0.00][T0] Modules linked in:^M > > [0.00][T0] CPU: 0 PID: 0 Comm: swapper Not tainted > > 5.19.0-rc5-next-20220708-dirty #131^M > > [0.00][T0] NIP: c038068c LR: c0380688 CTR: > > c0186ac0^M > > [0.00][T0] REGS: c2867930 TRAP: 0700 Not tainted > > (5.19.0-rc5-next-20220708-dirty)^M > > [0.00][T0] MSR: 80022003 CR: 24282224 > > XER: 2004^M > > [0.00][T0] CFAR: 0730 IRQMASK: 1 ^M > > [0.00][T0] GPR00: c0380688 c2867bd0 > > c2868d00 0065 ^M > > [0.00][T0] GPR04: 0001 > > 0080 000d ^M > > [0.00][T0] GPR08: > > c27fd000 000f ^M > > [0.00][T0] GPR12: c0186ac0 c2082280 > > 0003 000d ^M > > [0.00][T0] GPR16: 02cc00d0 > > c2082280 0001 ^M > > [0.00][T0] GPR20: c2080942 > > ^M > > [0.00][T0] GPR24: c10d6168 > > c20034c8 ^M > > [0.00][T0] GPR28: 0028 > > c2080942 c2953260 ^M > > [0.00][T0] NIP [c038068c] > > static_key_enable_cpuslocked+0xfc/0x120^M > > [0.00][T0] LR [c0380688] > > static_key_enable_cpuslocked+0xf8/0x120^M > > [0.00][T0] Call Trace:^M > > [0.00][T0] [c2867bd0] [c0380688] > > static_key_enable_cpuslocked+0xf8/0x120 (unreliable)^M > > [0.00][T0] [c2867c40] [c0380810] > > static_key_enable+0x30/0x50^M > > [0.00][T0] [c2867c70] [c2030314] > > setup_forced_irqthreads+0x28/0x40^M > > [0.00][T0] [c2867c90] [c2003568] > > do_early_param+0xa0/0x108^M > > [0.00][T0] [c2867d10] [c0175340] > > parse_args+0x290/0x4e0^M > > [0.00][T0] [c2867e10] [c2003c74] > > parse_early_options+0x48/0x5c^M > > [0.00][T0] [c2867e30] [c2003ce0] > > parse_early_param+0x58/0x84^M > > [0.00][T0] [c2867e60] [c2009878] > > early_init_devtree+0xd4/0x518^M > > [0.00][T0] [c2867f10] [c200aee0] > > early
Re: [PATCH linux-next] powerpc: use raw_smp_processor_id in arch_touch_nmi_watchdog
Thank John for correcting me ;-)
On Thu, Jul 14, 2022 at 5:25 PM John Ogness wrote:
>
> On 2022-07-14, Zhouyi Zhou wrote:
> > use raw_smp_processor_id() in arch_touch_nmi_watchdog
> > because when called from watchdog, the cpu is preemptible.
>
> I would expect the correct solution is to make it a non-migration
> section. Something like the below (untested) patch.
I applied your patch (I have made a tiny modification by removing the
return statement after "goto out;") and
passed the test in the ppc VM of Open Source Lab of Oregon State University.
Tested-by: Zhouyi Zhou
Many Thanks
Kindly Regards
Zhouyi
>
> John Ogness
>
> diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c
> index bfc27496fe7e..9d34aa809241 100644
> --- a/arch/powerpc/kernel/watchdog.c
> +++ b/arch/powerpc/kernel/watchdog.c
> @@ -450,17 +450,23 @@ static enum hrtimer_restart watchdog_timer_fn(struct
> hrtimer *hrtimer)
> void arch_touch_nmi_watchdog(void)
> {
> unsigned long ticks = tb_ticks_per_usec * wd_timer_period_ms * 1000;
> - int cpu = smp_processor_id();
> + int cpu;
> u64 tb;
>
> - if (!cpumask_test_cpu(cpu, _cpumask))
> + cpu = get_cpu();
> +
> + if (!cpumask_test_cpu(cpu, _cpumask)) {
> + goto out;
> return;
I think we should remove the return statement here.
> + }
>
> tb = get_tb();
> if (tb - per_cpu(wd_timer_tb, cpu) >= ticks) {
> per_cpu(wd_timer_tb, cpu) = tb;
> wd_smp_clear_cpu_pending(cpu);
> }
> +out:
> + put_cpu();
> }
> EXPORT_SYMBOL(arch_touch_nmi_watchdog);
[PATCH linux-next] powerpc: use raw_smp_processor_id in arch_touch_nmi_watchdog
use raw_smp_processor_id() in arch_touch_nmi_watchdog
because when called from watchdog, the cpu is preemptible.
Signed-off-by: Zhouyi Zhou
---
Dear PPC developers
I found this bug when trying to do rcutorture tests in ppc VM of
Open Source Lab of Oregon State University.
qemu-system-ppc64 -nographic -smp cores=4,threads=1 -net none -M pseries
-nodefaults -device spapr-vscsi -serial file:/tmp/console.log -m 2G -kernel
/home/ubuntu/linux-next/tools/testing/selftests/rcutorture/res/2022.07.08-22.36.11-torture/results-rcuscale-kvfree/TREE/vmlinux
-append "debug_boot_weak_hash panic=-1 console=ttyS0 rcuscale.kfree_rcu_test=1
rcuscale.kfree_nthreads=16 rcuscale.holdoff=20 rcuscale.kfree_loops=1
torture.disable_onoff_at_boot rcuscale.shutdown=1 rcuscale.verbose=0"
tail /tmp/console.log
[ 1232.433552][ T41] BUG: using smp_processor_id() in preemptible []
code: khungtaskd/41
[ 1232.439751][ T41] caller is arch_touch_nmi_watchdog+0x34/0xd0
[ 1232.440934][ T41] CPU: 3 PID: 41 Comm: khungtaskd Not tainted
5.19.0-rc5-next-20220708-dirty #106
[ 1232.442684][ T41] Call Trace:
[ 1232.443343][ T41] [c29cbbb0] [c06df360]
dump_stack_lvl+0x74/0xa8 (unreliable)
[ 1232.445237][ T41] [c29cbbf0] [c0d04f30]
check_preemption_disabled+0x150/0x160
[ 1232.446926][ T41] [c29cbc80] [c0035584]
arch_touch_nmi_watchdog+0x34/0xd0
[ 1232.448532][ T41] [c29cbcb0] [c02068ac]
watchdog+0x40c/0x5b0
[ 1232.451449][ T41] [c29cbdc0] [c0139df4] kthread+0x144/0x170
[ 1232.452896][ T41] [c29cbe10] [c000cd54]
ret_from_kernel_thread+0x5c/0x64
After this fix, "BUG: using smp_processor_id() in preemptible [] code:
khungtaskd/41" does not
appear again.
I also examined other places in watchdog.c where smp_processor_id() are used,
but they are well protected by preempt
disable.
Kind Regards
Zhouyi
--
arch/powerpc/kernel/watchdog.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/watchdog.c b/arch/powerpc/kernel/watchdog.c
index 7d28b9553654..ab6b84e00311 100644
--- a/arch/powerpc/kernel/watchdog.c
+++ b/arch/powerpc/kernel/watchdog.c
@@ -450,7 +450,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct
hrtimer *hrtimer)
void arch_touch_nmi_watchdog(void)
{
unsigned long ticks = tb_ticks_per_usec * wd_timer_period_ms * 1000;
- int cpu = smp_processor_id();
+ int cpu = raw_smp_processor_id();
u64 tb;
if (!cpumask_test_cpu(cpu, _cpumask))
--
2.25.1
Re: rcu_sched self-detected stall on CPU
On Fri, Apr 8, 2022 at 10:07 PM Paul E. McKenney wrote:
>
> On Fri, Apr 08, 2022 at 06:02:19PM +0800, Zhouyi Zhou wrote:
> > On Fri, Apr 8, 2022 at 3:23 PM Michael Ellerman wrote:
> > >
> > > "Paul E. McKenney" writes:
> > > > On Wed, Apr 06, 2022 at 05:31:10PM +0800, Zhouyi Zhou wrote:
> > > >> Hi
> > > >>
> > > >> I can reproduce it in a ppc virtual cloud server provided by Oregon
> > > >> State University. Following is what I do:
> > > >> 1) curl -l
> > > >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/snapshot/linux-5.18-rc1.tar.gz
> > > >> -o linux-5.18-rc1.tar.gz
> > > >> 2) tar zxf linux-5.18-rc1.tar.gz
> > > >> 3) cp config linux-5.18-rc1/.config
> > > >> 4) cd linux-5.18-rc1
> > > >> 5) make vmlinux -j 8
> > > >> 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot
> > > >> -smp 2 (QEMU 4.2.1)
> > > >> 7) after 12 rounds, the bug got reproduced:
> > > >> (http://154.223.142.244/logs/20220406/qemu.log.txt)
> > > >
> > > > Just to make sure, are you both seeing the same thing? Last I knew,
> > > > Zhouyi was chasing an RCU-tasks issue that appears only in kernels
> > > > built with CONFIG_PROVE_RCU=y, which Miguel does not have set. Or did
> > > > I miss something?
> > > >
> > > > Miguel is instead seeing an RCU CPU stall warning where RCU's
> > > > grace-period
> > > > kthread slept for three milliseconds, but did not wake up for more than
> > > > 20 seconds. This kthread would normally have awakened on CPU 1, but
> > > > CPU 1 looks to me to be very unhealthy, as can be seen in your console
> > > > output below (but maybe my idea of what is healthy for powerpc systems
> > > > is outdated). Please see also the inline annotations.
> > > >
> > > > Thoughts from the PPC guys?
> > >
> > > I haven't seen it in my testing. But using Miguel's config I can
> > > reproduce it seemingly on every boot.
> > >
> > > For me it bisects to:
> > >
> > > 35de589cb879 ("powerpc/time: improve decrementer clockevent processing")
> > >
> > > Which seems plausible.
> > I also bisect to 35de589cb879 ("powerpc/time: improve decrementer
> > clockevent processing")
>
> Very good! Thank you all!!!
You are very welcome ;-) and Thank you all
>
> Thanx, Paul
>
> > > Reverting that on mainline makes the bug go away.
> > I also revert that on the mainline, and am currently doing a pressure
> > test (by repeatedly invoking qemu and checking the console.log) on PPC
> > VM in Oregon State University.
After 306 rounds of stress test on mainline without triggering the bug
(last for 4 hours and 27 minutes), I think the bug is indeed caused by
35de589cb879 ("powerpc/time: improve decrementer clockevent
processing") and stop the test for now.
Thanks ;-)
Zhouyi
> > >
> > > I don't see an obvious bug in the diff, but I could be wrong, or the old
> > > code was papering over an existing bug?
> > >
> > > I'll try and work out what it is about Miguel's config that exposes
> > > this vs our defconfig, that might give us a clue.
> > Great job!
> > >
> > > cheers
> > Thanks
> > Zhouyi
Re: rcu_sched self-detected stall on CPU
On Fri, Apr 8, 2022 at 3:23 PM Michael Ellerman wrote:
>
> "Paul E. McKenney" writes:
> > On Wed, Apr 06, 2022 at 05:31:10PM +0800, Zhouyi Zhou wrote:
> >> Hi
> >>
> >> I can reproduce it in a ppc virtual cloud server provided by Oregon
> >> State University. Following is what I do:
> >> 1) curl -l
> >> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/snapshot/linux-5.18-rc1.tar.gz
> >> -o linux-5.18-rc1.tar.gz
> >> 2) tar zxf linux-5.18-rc1.tar.gz
> >> 3) cp config linux-5.18-rc1/.config
> >> 4) cd linux-5.18-rc1
> >> 5) make vmlinux -j 8
> >> 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot
> >> -smp 2 (QEMU 4.2.1)
> >> 7) after 12 rounds, the bug got reproduced:
> >> (http://154.223.142.244/logs/20220406/qemu.log.txt)
> >
> > Just to make sure, are you both seeing the same thing? Last I knew,
> > Zhouyi was chasing an RCU-tasks issue that appears only in kernels
> > built with CONFIG_PROVE_RCU=y, which Miguel does not have set. Or did
> > I miss something?
> >
> > Miguel is instead seeing an RCU CPU stall warning where RCU's grace-period
> > kthread slept for three milliseconds, but did not wake up for more than
> > 20 seconds. This kthread would normally have awakened on CPU 1, but
> > CPU 1 looks to me to be very unhealthy, as can be seen in your console
> > output below (but maybe my idea of what is healthy for powerpc systems
> > is outdated). Please see also the inline annotations.
> >
> > Thoughts from the PPC guys?
>
> I haven't seen it in my testing. But using Miguel's config I can
> reproduce it seemingly on every boot.
>
> For me it bisects to:
>
> 35de589cb879 ("powerpc/time: improve decrementer clockevent processing")
>
> Which seems plausible.
I also bisect to 35de589cb879 ("powerpc/time: improve decrementer
clockevent processing")
>
> Reverting that on mainline makes the bug go away.
I also revert that on the mainline, and am currently doing a pressure
test (by repeatedly invoking qemu and checking the console.log) on PPC
VM in Oregon State University.
>
> I don't see an obvious bug in the diff, but I could be wrong, or the old
> code was papering over an existing bug?
>
> I'll try and work out what it is about Miguel's config that exposes
> this vs our defconfig, that might give us a clue.
Great job!
>
> cheers
Thanks
Zhouyi
Re: rcu_sched self-detected stall on CPU
Dear Paul and Miguel On Fri, Apr 8, 2022 at 1:55 AM Paul E. McKenney wrote: > > On Thu, Apr 07, 2022 at 07:05:58PM +0200, Miguel Ojeda wrote: > > On Thu, Apr 7, 2022 at 5:15 PM Paul E. McKenney wrote: > > > > > > Ah. So you would instead look for boot to have completed within 10 > > > seconds? Either way, reliable automation might well more important than > > > reduction in time. > > > > No (although I guess that could be an option), I was only pointing out > > that when no stall is produced, the run should be much quicker than 30 > > seconds (at least it was in my setup), which would be the majority of the > > runs. > > Ah, thank you for the clarification! Thank both of you for the information. In my setup (PPC cloud VM), the majority of the runs complete at least for 50 seconds. From last evening to this morning (Beijing Time), following experiments have been done: 1) torture mainline: the test quickly finished by hitting "rcu_sched self-detected stall" after 12 runs 2) torture v5.17: the test last 10 hours plus 14 minutes, 702 runs have been done without trigger the bug Conclusion: There must be a commit that causes the bug as Paul has pointed out. I am going to do the bisect, and estimate to locate the bug within a week (at most). This is a good learning experience, thanks for the guidance ;-) Kind Regards Zhouyi > > Thanx, Paul
Re: rcu_sched self-detected stall on CPU
Hi Paul On Thu, Apr 7, 2022 at 3:50 AM Paul E. McKenney wrote: > > On Thu, Apr 07, 2022 at 02:25:59AM +0800, Zhouyi Zhou wrote: > > Hi Paul > > > > On Thu, Apr 7, 2022 at 1:00 AM Paul E. McKenney wrote: > > > > > > On Wed, Apr 06, 2022 at 05:31:10PM +0800, Zhouyi Zhou wrote: > > > > Hi > > > > > > > > I can reproduce it in a ppc virtual cloud server provided by Oregon > > > > State University. Following is what I do: > > > > 1) curl -l > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/snapshot/linux-5.18-rc1.tar.gz > > > > -o linux-5.18-rc1.tar.gz > > > > 2) tar zxf linux-5.18-rc1.tar.gz > > > > 3) cp config linux-5.18-rc1/.config > > > > 4) cd linux-5.18-rc1 > > > > 5) make vmlinux -j 8 > > > > 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot > > > > -smp 2 (QEMU 4.2.1) > > > > 7) after 12 rounds, the bug got reproduced: > > > > (http://154.223.142.244/logs/20220406/qemu.log.txt) > > > > > > Just to make sure, are you both seeing the same thing? Last I knew, > > > Zhouyi was chasing an RCU-tasks issue that appears only in kernels > > > built with CONFIG_PROVE_RCU=y, which Miguel does not have set. Or did > > > I miss something? > > We are both seeing the same thing, I work in parallel. > > 1) I am chasing the RCU-tasks issue which I will report my discoveries > > to you later. > > 2) I am reproducing the RCU CPU stall issue reported by Miguel > > yesterday. Lucky enough, I can reproduce it and thanks to Oregon State > > University who provides me with the environment! I am also very > > interested in helping chase the reason behind the issue. Lucky enough > > the issue can be reproduced in a non-hardware accelerated qemu > > environment so that I can give a hand. > > How quickly does this happen? The console log that Miguel sent had > within 30 seconds of boot. If it always happens this quickly, it Yes, this happens within 30 seconds after kernel boot. If we take all into account (qemu preparing, kernel loading), we can do one test within 54 seconds. > should be possible to do a bisection, especially when running qemu. Thank you for your guidance! I will do it. > The trick would be to boot a given commit until you see it fail on the > one hand or until it boots successfully 70 times. In the latter case, > report success to "git bisect", in the former case report failure. Yes, I will do it. > If the one-out-of-5 failure rate is accurate, you will have a 99.997% > chance of reporting the correct failure state on each step, resulting > in better than a 99.9% chance of converging on the correct commit. Agree, I have learned that probability from your book. > > Of course, you would hit the preceding commit hard to double-check. Agree > > Does this seem reasonable? Or am I being overly optimstic on the > failure times? This is very reasonable, I have written a test script (based on the script I used to test RCU-tasks issue), and will perform the bisection in the coming days. #!/bin/sh if [ "$#" -ne 1 ]; then echo "Usage: test.sh kernel" exit fi COUNTER=0 while [ $COUNTER -lt 1000 ] ; do mv /tmp/console.log /tmp/console.log.orig echo $COUNTER > /tmp/console.log date qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot -smp 2 -serial file:/tmp/console.log -m 2048 -append "console=ttyS0"& qemu_pid=$! echo "Start round $COUNTER" while true ; do if grep -q "rcu_sched self-detected stall" /tmp/console.log; then echo "find rcu_sched detected stalls" break fi if grep -q "Unable to mount root fs" /tmp/console.log; then echo "kernel test round $COUNTER finish" break fi sleep 1 done kill $qemu_pid if grep -q "rcu_sched self-detected stall" /tmp/console.log; then echo $COUNTER exit fi COUNTER=$(($COUNTER+1)) done Thanks Zhouyi > > Thanx, Paul > > > Thanks > > Zhouyi > > > > > > Miguel is instead seeing an RCU CPU stall warning where RCU's grace-period > > > kthread slept for three milliseconds, but did not wake up for more than > > > 20 seconds. This kthread would normally have awakened on CPU 1, but > > > CPU 1 looks to me to be very unhealthy, as can be seen in your console > > > output below (but maybe my idea of what is healthy for powerpc systems > > > is
Re: rcu_sched self-detected stall on CPU
Hi Paul On Thu, Apr 7, 2022 at 1:00 AM Paul E. McKenney wrote: > > On Wed, Apr 06, 2022 at 05:31:10PM +0800, Zhouyi Zhou wrote: > > Hi > > > > I can reproduce it in a ppc virtual cloud server provided by Oregon > > State University. Following is what I do: > > 1) curl -l > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/snapshot/linux-5.18-rc1.tar.gz > > -o linux-5.18-rc1.tar.gz > > 2) tar zxf linux-5.18-rc1.tar.gz > > 3) cp config linux-5.18-rc1/.config > > 4) cd linux-5.18-rc1 > > 5) make vmlinux -j 8 > > 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot > > -smp 2 (QEMU 4.2.1) > > 7) after 12 rounds, the bug got reproduced: > > (http://154.223.142.244/logs/20220406/qemu.log.txt) > > Just to make sure, are you both seeing the same thing? Last I knew, > Zhouyi was chasing an RCU-tasks issue that appears only in kernels > built with CONFIG_PROVE_RCU=y, which Miguel does not have set. Or did > I miss something? We are both seeing the same thing, I work in parallel. 1) I am chasing the RCU-tasks issue which I will report my discoveries to you later. 2) I am reproducing the RCU CPU stall issue reported by Miguel yesterday. Lucky enough, I can reproduce it and thanks to Oregon State University who provides me with the environment! I am also very interested in helping chase the reason behind the issue. Lucky enough the issue can be reproduced in a non-hardware accelerated qemu environment so that I can give a hand. Thanks Zhouyi > > Miguel is instead seeing an RCU CPU stall warning where RCU's grace-period > kthread slept for three milliseconds, but did not wake up for more than > 20 seconds. This kthread would normally have awakened on CPU 1, but > CPU 1 looks to me to be very unhealthy, as can be seen in your console > output below (but maybe my idea of what is healthy for powerpc systems > is outdated). Please see also the inline annotations. > > Thoughts from the PPC guys? > > Thanx, Paul > > > > [ 21.186912] rcu: INFO: rcu_sched self-detected stall on CPU > [ 21.187331] rcu: 1-...!: (4712629 ticks this GP) idle=2c1/0/0x3 > softirq=8/8 fqs=0 > [ 21.187529] (t=21000 jiffies g=-1183 q=3) > [ 21.187681] rcu: rcu_sched kthread timer wakeup didn't happen for 20997 > jiffies! g-1183 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 > > The grace-period kthread is still asleep (->state=0x402). > This indicates that the three-jiffy timer has somehow been > prevented from expiring for almost a full 21 seconds. Of course, > if timers don't work, RCU cannot work. > > [ 21.187770] rcu: Possible timer handling issue on cpu=1 timer-softirq=1 > [ 21.187927] rcu: rcu_sched kthread starved for 21001 jiffies! g-1183 f0x0 > RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1 > [ 21.188019] rcu: Unless rcu_sched kthread gets sufficient CPU time, > OOM is now expected behavior. > [ 21.188087] rcu: RCU grace-period kthread stack dump: > [ 21.188196] task:rcu_sched state:I stack:0 pid: 10 ppid: 2 > flags:0x0800 > [ 21.188453] Call Trace: > [ 21.188525] [c61e78a0] [c61e78e0] 0xc61e78e0 > (unreliable) > [ 21.188900] [c61e7a90] [c0017210] __switch_to+0x250/0x310 > [ 21.189210] [c61e7b00] [c03ed660] __schedule+0x210/0x660 > [ 21.189315] [c61e7b80] [c03edb14] schedule+0x64/0x110 > [ 21.189387] [c61e7bb0] [c03f6648] > schedule_timeout+0x1d8/0x390 > [ 21.189473] [c61e7c80] [c01c] > rcu_gp_fqs_loop+0x2dc/0x3d0 > [ 21.189555] [c61e7d30] [c01144ec] > rcu_gp_kthread+0x13c/0x160 > [ 21.189633] [c61e7dc0] [c00c1770] kthread+0x110/0x120 > [ 21.189714] [c61e7e10] [c000c9e4] > ret_from_kernel_thread+0x5c/0x64 > > The above stack trace is expected behavior when the RCU > grace-period kthread is waiting to do its next FQS scan. > > [ 21.189938] rcu: Stack dump where RCU GP kthread last ran: > > And here is the stalled CPU, which also happens to be the CPU > that RCU last ran on: > > [ 21.189992] Task dump for CPU 1: > [ 21.190059] task:swapper/1 state:R running task stack:0 > pid:0 ppid: 1 flags:0x0804 > [ 21.190169] Call Trace: > [ 21.190194] [c61ef2d0] [c00c9a40] > sched_show_task+0x180/0x1c0 (unreliable) > [ 21.190278] [c61ef340] [c0116ca0] > rcu_check_gp_kthread_starvation+0x16c/0x19c > [ 21.190370]
Re: rcu_sched self-detected stall on CPU
Hi I can reproduce it in a ppc virtual cloud server provided by Oregon State University. Following is what I do: 1) curl -l https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/snapshot/linux-5.18-rc1.tar.gz -o linux-5.18-rc1.tar.gz 2) tar zxf linux-5.18-rc1.tar.gz 3) cp config linux-5.18-rc1/.config 4) cd linux-5.18-rc1 5) make vmlinux -j 8 6) qemu-system-ppc64 -kernel vmlinux -nographic -vga none -no-reboot -smp 2 (QEMU 4.2.1) 7) after 12 rounds, the bug got reproduced: (http://154.223.142.244/logs/20220406/qemu.log.txt) Cheers ;-) Zhouyi On Wed, Apr 6, 2022 at 3:47 PM Miguel Ojeda wrote: > > Hi PPC/RCU, > > While merging v5.18-rc1 changes I noticed our CI PPC runs broke. I > reproduced the problem in v5.18-rc1 as well as next-20220405, under > both QEMU 4.2.1 and 6.1.0, with `-smp 2`; but I cannot reproduce it in > v5.17 from a few tries. > > Sadly, the problem is not deterministic although it is not too hard to > reproduce (1 out of 5?). Please see attached config and QEMU output. > > Cheers, > Miguel
Re: rcutorture’s init segfaults in ppc64le VM
Dear Paul On Thu, Mar 10, 2022 at 4:10 PM Paul Menzel wrote: > > Dear Zhouyi, > > > Thank you for still looking into this. You are very welcome ;-) > > > Am 10.03.22 um 03:37 schrieb Zhouyi Zhou: > > > I try to reproduce the bug in ppc64 VM in Oregon State University > > using the vmlinux extracted from > > https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz > > > > the ppc64 VM in which I run the qemu without hardware acceleration is: > > Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc version > > 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb 3 18:43:11 > > UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166) > > > > > > The qemu command I use to test: > > cd > > /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$ > > $qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M > > pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log > > -m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1 > > console=ttyS0 rcutorture.onoff_interval=200 > > rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12 > > rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 > > rcutree.kthread_prio=2 threadirqs tree.use_softirq=0 > > rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 > > rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1 > > rcutorture.verbose=1" > > > > The console.log is uploaded to: > > http://154.223.142.244/logs/20220310/console.paul.log > > The log tells us it is illegal instruction that causes the trouble: > > [4.246387][T1] init[1]: illegal instruction (4) at 1002c308 nip > > 1002c308 lr 10001684 code 1 in init[1000+d] > > [4.251400][T1] init[1]: code: f90d88c0 f92a0008 f9480008 7c2004ac > > 2c2d f949 386d88d0 38e8 > > [4.253416][T1] init[1]: code: 41820098 e92d8f98 75290010 4182008c > > <4401> 2c2d 6000 8902f438 > > > > > > Meanwhile, the vmlinux compiled by myself runs smoothly. > > How did you build it? Using GCC or clang? I forgot, if the problem was I built vmlinux(es) using GCC and clang both. The compiled vmlinux(es) runs smoothly. > only reproducible if the host Linux kernel was built with clang or the > VM kernel. Yes, I also remember this, the dependence of how the host Linux kernel is built makes things more complex. > > > Then I modify mkinitrd.sh to let it panic manually: > > http://154.223.142.244/logs/20220310/mkinitrd.sh > > I only see the change: > > - > + int *ptr = 0; > + *ptr = 0; > Yes, I make the segfault happen manually. > > The log tells us it is a segfault (instead of a illegal instruction): > > http://154.223.142.244/logs/20220310/console.zhouyi.log > > > > Then I use gdb to debug the init in host: > > ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb > > tools/testing/selftests/rcutorture/initrd/init > > (gdb) run > > Starting program: > > /home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init > > > > Program received signal SIGSEGV, Segmentation fault. > > 0x1b2c in ?? () > > (gdb) x/10i $pc > > => 0x1b2c:stw r9,0(r9) > > 0x1b30:trap > > 0x1b34:.long 0x0 > > 0x1b38:.long 0x0 > > 0x1b3c:.long 0x0 > > 0x1b40:lis r2,4110 > > 0x1b44:addir2,r2,31488 > > 0x1b48:mr r9,r1 > > 0x1b4c:rldicr r1,r1,0,59 > > 0x1b50:li r0,0 > > (gdb) p $r9 > > $1 = 0 > > (gdb) x/30x $pc - 0x30 > > 0x1afc:0x388400400x387f00400xf80100400x48026919 > > 0x1b0c:0x60000xe80100400x7c0803a60x4b24 > > 0x1b1c:0x0x01000x01800x3920 > > 0x1b2c:0x91290x7fe80x0x > > which matches the hex content of > > http://154.223.142.244/logs/20220310/console.zhouyi.log: > > [5.077431][T1] init[1]: segfault (11) at 0 nip 1b2c lr 10001024 > > code 1 in init[1000+d] > > [5.087167][T1] init[1]: code: 38840040 387f0040 f8010040 48026919 > > 6000 e8010040 7c0803a6 4b24 > > [5.093987][T1] init[1]: code: 0100 0180 3920 > > <9129> 7fe8 > > > > > > Conclusions: there might be something wrong when packing the init into > > vmlinux in your environment. > > > > I will continue to do research on this interesting problem with you. > > As written I think it’s a problem with LLVM/clang. Unfortunately, I > won’t be able to retest before next week. Roger that, no need to hurry ;-) Kind regards Zhouyi > Kind regards, > > Paul
Re: rcutorture’s init segfaults in ppc64le VM
Dear Paul I try to reproduce the bug in ppc64 VM in Oregon State University using the vmlinux extracted from https://owww.molgen.mpg.de/~pmenzel/rcutorture-2022.02.01-21.52.37-torture-locktorture-kasan-lock01.tar.xz the ppc64 VM in which I run the qemu without hardware acceleration is: Linux version 5.4.0-100-generic (buildd@bos02-ppc64el-021) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #113-Ubuntu SMP Thu Feb 3 18:43:11 UTC 2022 (Ubuntu 5.4.0-100.113-generic 5.4.166) The qemu command I use to test: cd /tmp/dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-locktorture-kasan/LOCK01$ $qemu-system-ppc64 -nographic -smp cores=2,threads=1 -net none -M pseries -nodefaults -device spapr-vscsi -serial file:/tmp/console.log -m 512 -kernel ./vmlinux -append "debug_boot_weak_hash panic=-1 console=ttyS0 rcutorture.onoff_interval=200 rcutorture.onoff_holdoff=30 rcutree.gp_preinit_delay=12 rcutree.gp_init_delay=3 rcutree.gp_cleanup_delay=3 rcutree.kthread_prio=2 threadirqs tree.use_softirq=0 rcutorture.n_barrier_cbs=4 rcutorture.stat_interval=15 rcutorture.shutdown_secs=1800 rcutorture.test_no_idle_hz=1 rcutorture.verbose=1" The console.log is uploaded to: http://154.223.142.244/logs/20220310/console.paul.log The log tells us it is illegal instruction that causes the trouble: [4.246387][T1] init[1]: illegal instruction (4) at 1002c308 nip 1002c308 lr 10001684 code 1 in init[1000+d] [4.251400][T1] init[1]: code: f90d88c0 f92a0008 f9480008 7c2004ac 2c2d f949 386d88d0 38e8 [4.253416][T1] init[1]: code: 41820098 e92d8f98 75290010 4182008c <4401> 2c2d 6000 8902f438 Meanwhile, the vmlinux compiled by myself runs smoothly. Then I modify mkinitrd.sh to let it panic manually: http://154.223.142.244/logs/20220310/mkinitrd.sh The log tells us it is a segfault (instead of a illegal instruction): http://154.223.142.244/logs/20220310/console.zhouyi.log Then I use gdb to debug the init in host: ubuntu@zhouzhouyi-1:~/newkernel/linux-next$ gdb tools/testing/selftests/rcutorture/initrd/init (gdb) run Starting program: /home/ubuntu/newkernel/linux-next/tools/testing/selftests/rcutorture/initrd/init Program received signal SIGSEGV, Segmentation fault. 0x1b2c in ?? () (gdb) x/10i $pc => 0x1b2c:stw r9,0(r9) 0x1b30:trap 0x1b34:.long 0x0 0x1b38:.long 0x0 0x1b3c:.long 0x0 0x1b40:lis r2,4110 0x1b44:addir2,r2,31488 0x1b48:mr r9,r1 0x1b4c:rldicr r1,r1,0,59 0x1b50:li r0,0 (gdb) p $r9 $1 = 0 (gdb) x/30x $pc - 0x30 0x1afc:0x388400400x387f00400xf80100400x48026919 0x1b0c:0x60000xe80100400x7c0803a60x4b24 0x1b1c:0x0x01000x01800x3920 0x1b2c:0x91290x7fe80x0x which matches the hex content of http://154.223.142.244/logs/20220310/console.zhouyi.log: [5.077431][T1] init[1]: segfault (11) at 0 nip 1b2c lr 10001024 code 1 in init[1000+d] [5.087167][T1] init[1]: code: 38840040 387f0040 f8010040 48026919 6000 e8010040 7c0803a6 4b24 [5.093987][T1] init[1]: code: 0100 0180 3920 <9129> 7fe8 Conclusions: there might be something wrong when packing the init into vmlinux in your environment. I will continue to do research on this interesting problem with you. Thanks Kind Regards Zhouyi On Tue, Feb 8, 2022 at 8:12 PM Paul Menzel wrote: > > Dear Michael, > > > Thank you for looking into this. > > Am 08.02.22 um 11:09 schrieb Michael Ellerman: > > Paul Menzel writes: > > […] > > >> On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux > >> 5.17-rc2+ with rcutorture tests > > > > I'm not sure if that's the host kernel version or the version you're > > using of rcutorture? Can you tell us the sha1 of your host kernel and of > > the tree you're running rcutorture from? > > The host system runs Linux 5.17-rc1+ started with kexec. Unfortunately, > I am unable to find the exact sha1. > > $ more /proc/version > Linux version 5.17.0-rc1+ > (pmen...@flughafenberlinbrandenburgwillybrandt.molgen.mpg.de) (Ubuntu > clang version 13.0.0-2, LLD 13.0.0) #1 SMP Fri Jan 28 > 17:13:04 CET 2022 > > The Linux tree, from where I run rcutorture from, is at commit > dfd42facf1e4 (Linux 5.17-rc3) with four patches on top: > > $ git log --oneline -6 > 207cec79e752 (HEAD -> master, origin/master, origin/HEAD) Problems > with rcutorture on ppc64le: allmodconfig(2) and other failures > 8c82f96fbe57 ata: libata-sata: improve sata_link_debounce() > a447541d925f ata: libata-sata: remove debounce delay by default > afd84e1eeafc ata: libata-sata: introduce struct sata_deb_timing > f4caf7e48b75 ata: libata-sata: Simplify sata_link_resume() interface > dfd42facf1e4
Re: BUG: sleeping function called from invalid context at include/linux/sched/mm.h:256
Thank Matthew for correcting me On Sun, Feb 13, 2022 at 12:09 PM Matthew Wilcox wrote: > > On Sun, Feb 13, 2022 at 11:19:09AM +0800, Zhouyi Zhou wrote: > > I think the key to the problem lies in your attached console.log > > (pasted below), at times 0.014987 and 0.015995, I see there are two > > locks (cpu_hotplug_lock and jump_label_mutex) holded while > > kmem_cache_alloc calls __might_resched (0.023356). > > Those are both sleeping locks (a percpu_rwsem and mutex, respectively). > There is no problem with sleeping while holding a mutex or rwsem. >From console.log, I see [0.012154][T1] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:256 [0.013128][T1] in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 1, name: swapper/0 >From ___might_sleep, I see 9506 if ((preempt_count_equals(preempt_offset) && !irqs_disabled() && 9507 !is_idle_task(current) && !current->non_block_count) || 9508 system_state == SYSTEM_BOOTING || system_state > SYSTEM_RUNNING || 9509 oops_in_progress) 9510 return; I guess it is irq_disable which cause the bug. Thanks Zhouyi
Re: BUG: sleeping function called from invalid context at include/linux/sched/mm.h:256
Dear Paul
I think the key to the problem lies in your attached console.log
(pasted below), at times 0.014987 and 0.015995, I see there are two
locks (cpu_hotplug_lock and jump_label_mutex) holded while
kmem_cache_alloc calls __might_resched (0.023356).
I guess PowerPC's map_patch_area should temporary release above two
locks before invoking map_kernel_page (which will call
kmem_cache_alloc):
static int map_patch_area(void *addr, unsigned long text_poke_addr)
{
unsigned long pfn;
int err;
if (is_vmalloc_or_module_addr(addr))
pfn = vmalloc_to_pfn(addr);
else
pfn = __pa_symbol(addr) >> PAGE_SHIFT;
err = map_kernel_page(text_poke_addr, (pfn << PAGE_SHIFT), PAGE_KERNEL);
pr_devel("Mapped addr %lx with pfn %lx:%d\n", text_poke_addr, pfn, err);
if (err)
return -1;
return 0;
}
I will try to think it over in the coming days.
[0.012154][T1] BUG: sleeping function called from invalid
context at include/linux/sched/mm.h:256
[0.013128][T1] in_atomic(): 0, irqs_disabled(): 1, non_block:
0, pid: 1, name: swapper/0
[0.014015][T1] preempt_count: 0, expected: 0
[0.014505][T1] 2 locks held by swapper/0/1:
[0.014987][T1] #0: c26108a0
(cpu_hotplug_lock){.+.+}-{0:0}, at: static_key_enable+0x24/0x50
[0.015995][T1] #1: c27416c8
(jump_label_mutex){+.+.}-{3:3}, at:
static_key_enable_cpuslocked+0x88/0x120
[0.017107][T1] irq event stamp: 46
[0.017507][T1] hardirqs last enabled at (45):
[] _raw_spin_unlock_irqrestore+0x94/0xd0
[0.018549][T1] hardirqs last disabled at (46):
[] do_patch_instruction+0x3b4/0x4a0
[0.019549][T1] softirqs last enabled at (0):
[] copy_process+0x8d0/0x1df0
[0.020474][T1] softirqs last disabled at (0): [<>] 0x0
[0.021200][T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
5.17.0-rc3-00349-gd3a9fd9fed88 #34
[0.022115][T1] Call Trace:
[0.022443][T1] [c84837d0] [c0961aac]
dump_stack_lvl+0xa0/0xec (unreliable)
[0.023356][T1] [c8483820] [c019b314]
__might_resched+0x2f4/0x310
[0.024174][T1] [c84838b0] [c04c0c70]
kmem_cache_alloc+0x220/0x4b0
[0.025000][T1] [c8483920] [c0448af4]
__pud_alloc+0x74/0x1d0
[0.025772][T1] [c8483970] [c008fe3c]
hash__map_kernel_page+0x2cc/0x390
[0.026643][T1] [c8483a20] [c00a9944]
do_patch_instruction+0x134/0x4a0
[0.027511][T1] [c8483a70] [c00559d4]
arch_jump_label_transform+0x64/0x78
[0.028401][T1] [c8483a90] [c03d6288]
__jump_label_update+0x148/0x180
[0.029254][T1] [c8483b30] [c03d6800]
static_key_enable_cpuslocked+0xd0/0x120
[0.030179][T1] [c8483ba0] [c03d6880]
static_key_enable+0x30/0x50
[0.030996][T1] [c8483bd0] [c200a8f4]
check_kvm_guest+0x60/0x88
[0.031799][T1] [c8483c00] [c2027744]
pSeries_smp_probe+0x54/0xb0
[0.032617][T1] [c8483c30] [c2011db8]
smp_prepare_cpus+0x3e0/0x430
[0.033444][T1] [c8483cd0] [c2004908]
kernel_init_freeable+0x20c/0x43c
[0.034307][T1] [c8483db0] [c0012c00]
kernel_init+0x30/0x1a0
[0.035078][T1] [c8483e10] [c000cd64]
ret_from_kernel_thread+0x5c/0x64
Thanks for your trust in me!
Cheers
Zhouyi
On Sun, Feb 13, 2022 at 7:05 AM Paul Menzel wrote:
>
> Dear Linux folks,
>
>
> Running rcutorture on the POWER8 system IBM S822LC with Ubuntu 20.10, it
> found the bug below. I more or less used rcu/dev (0ba8896d2fd7
> (lib/irq_poll: Declare IRQ_POLL softirq vector as ksoftirqd-parking
> safe)) [1]. The bug manifested for the four configurations below.
>
> 1. results-rcutorture-kasan/SRCU-T
> 2. results-rcutorture-kasan/TINY02
> 3. results-rcutorture/SRCU-T
> 4. results-rcutorture/TINY02
>
> For example, the attached
>
>
> /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.11-22.00.51-torture/results-rcutorture-kasan/SRCU-T/console.log
>
> contains:
>
> ```
> [0.012154][T1] BUG: sleeping function called from invalid
> context at include/linux/sched/mm.h:256
> [0.013128][T1] in_atomic(): 0, irqs_disabled(): 1, non_block: 0,
> pid: 1, name: swapper/0
> [0.014015][T1] preempt_count: 0, expected: 0
> [0.014505][T1] 2 locks held by swapper/0/1:
> [0.014987][T1] #0: c26108a0
> (cpu_hotplug_lock){.+.+}-{0:0}, at: static_key_enable+0x24/0x50
> [0.015995][T1] #1: c27416c8
> (jump_label_mutex){+.+.}-{3:3}, at: static_key_enable_cpuslocked+0x88/0x120
> [0.017107][T1] irq event stamp: 46
> [0.017507][T1] hardirqs last enabled at (45):
> [] _raw_spin_unlock_irqrestore+0x94/0xd0
> [0.018549][T1] hardirqs last disabled at (46):
> []
Re: BUG: Kernel NULL pointer dereference on write at 0x00000000 (rtmsg_ifinfo_build_skb)
Hi Paul
Below are my preliminary test results tested on PPC VM supplied by
Open source lab of Oregon State University, thank you for your
support!
[Preliminary test results on ppc64le virtual guest]
1. Conclusion
Some other kernel configuration besides RCU may lead to "BUG: Kernel
NULL pointer dereference" at boot
2. Test Environment
2.1 host hardware
8 core ppc64le virtual guest with 16G ram and 160G disk
cpu: POWER9 (architected), altivec supported
clock: 2200.00MHz
revision: 2.2 (pvr 004e 1202)
2.2 host software
Operating System: Ubuntu 20.04.3 LTS, Compiler: gcc version 9.3.0
3. Test Procedure
3.1 kernel source
next-20220203
3.2 build and boot the kernel with CONFIG_DRM_BOCHS=m and
CONFIG_RCU_TORTURE_TEST=y
test result: "BUG: Kernel NULL pointer dereference" at boot
config file: http://154.223.142.244/Feb2022/config-5.17.0-rc2-next.bochs.torture
boot msg: http://154.223.142.244/Feb2022/dmesg.torture.bochs
3.3 build and boot the kernel with CONFIG_DRM_BOCHS=m
test result: "BUG: Kernel NULL pointer dereference" at boot
config file: http://154.223.142.244/Feb2022/config-5.17.0-rc2-next.bochs
boot msg: http://154.223.142.244/Feb2022/dmesg.bochs
3.4 build and boot the kernel with CONFIG_RCU_TORTURE_TEST=y (without
CONFIG_DRM_BOCHS)
test result: boot without error
config file: http://154.223.142.244/Feb2022/config-5.17.0-rc2-next.torture
boot msg: http://154.223.142.244/Feb2022/dmesg.torture
3.5 build and boot the kernel with CONFIG_RCU_TORTURE_TEST=m (without
CONFIG_DRM_BOCHS)
test result: boot without error
config file: http://154.223.142.244/Feb2022/config-5.17.0-rc2-next
boot msg: http://154.223.142.244/Feb2022/dmesg
4. Acknowledgement
Thank Open source lab of Oregon State University and Paul Menzel and
all other community members who support my tiny research.
Thanks
Zhouyi
On Wed, Feb 2, 2022 at 10:39 AM Zhouyi Zhou wrote:
>
> Thank Paul for your encouragement!
>
> On Wed, Feb 2, 2022 at 1:50 AM Paul E. McKenney wrote:
> >
> > On Mon, Jan 31, 2022 at 09:08:40AM +0800, Zhouyi Zhou wrote:
> > > Thank Paul for joining us!
> > >
> > > On Mon, Jan 31, 2022 at 1:44 AM Paul E. McKenney
> > > wrote:
> > > >
> > > > On Sun, Jan 30, 2022 at 09:24:44PM +0800, Zhouyi Zhou wrote:
> > > > > Dear Paul
> > > > >
> > > > > On Sun, Jan 30, 2022 at 4:19 PM Paul Menzel
> > > > > wrote:
> > > > > >
> > > > > > Dear Zhouyi,
> > > > > >
> > > > > >
> > > > > > Am 30.01.22 um 01:21 schrieb Zhouyi Zhou:
> > > > > >
> > > > > > > Thank you for your instructions, I learned a lot from this
> > > > > > > process.
> > > > > >
> > > > > > Same on my end.
> > > > > >
> > > > > > > On Sun, Jan 30, 2022 at 12:52 AM Paul Menzel
> > > > > > > wrote:
> > > > > >
> > > > > > >> Am 29.01.22 um 03:23 schrieb Zhouyi Zhou:
> > > > > > >>
> > > > > > >>> I don't have an IBM machine, but I tried to analyze the problem
> > > > > > >>> using
> > > > > > >>> my x86_64 kvm virtual machine, I can't reproduce the bug using
> > > > > > >>> my
> > > > > > >>> x86_64 kvm virtual machine.
> > > > > > >>
> > > > > > >> No idea, if it’s architecture specific.
> > > > > > >>
> > > > > > >>> I saw the panic is caused by registration of sit device (A sit
> > > > > > >>> device
> > > > > > >>> is a type of virtual network device that takes our IPv6 traffic,
> > > > > > >>> encapsulates/decapsulates it in IPv4 packets, and
> > > > > > >>> sends/receives it
> > > > > > >>> over the IPv4 Internet to another host)
> > > > > > >>>
> > > > > > >>> sit device is registered in function sit_init_net:
> > > > > > >>> 1895static int __net_init sit_init_net(struct net *net)
> > > > > > >>> 1896{
> > > > > > >>> 1897struct sit_net *sitn = net_generic(net, sit_net_id);
> > > > > > >>> 1898struct ip_tunnel *t;
> > > > > > >>> 1899int err;
> > > > > > >>> 1900
> > > > > > >>> 1901sitn->tunnels[0] = sitn-
Re: rcutorture’s init segfaults in ppc64le VM
Hi,
The mailing list forward the emails to me in periodic style, very
sorry not seeing Willy's email until I visited
https://lore.kernel.org/rcu/20220207180901.gb14...@1wt.eu/T/#u, I am
also very interested in testing Willy's proposal.
Thanks a lot
Zhouyi
On Tue, Feb 8, 2022 at 1:46 PM Zhouyi Zhou wrote:
>
> Dear Paul
>
> I am also very interested in the topic.
> The Open source lab of Oregon State University has lent me a 8 core
> power ppc64el VM for 3 months, I guess I can try reproducing this bug
> in the Virtual Machine by executing qemu in non hardware accelerated
> mode (using -no-kvm argument).
> I am currently doing research on
> https://lore.kernel.org/rcu/20220201175023.GW4285@paulmck-ThinkPad-P17-Gen-1/T/#mc7e5f8ec99e3794bec1e38fbbb130e71172e4759,
> I think I can give a preliminary short report on that previous topic
> tomorrow. And I am very interested in doing a search on the new topic
> the day after tomorrow.
>
> Thank you both for providing me an opportunity to improve myself ;-)
>
> Thanks again
> Zhouyi
>
> On Tue, Feb 8, 2022 at 12:10 PM Paul E. McKenney wrote:
> >
> > On Mon, Feb 07, 2022 at 05:44:47PM +0100, Paul Menzel wrote:
> > > Dear Linux folks,
> > >
> > >
> > > On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
> > > 5.17-rc2+ with rcutorture tests
> > >
> > > $ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
> > >
> > > the built init
> > >
> > > $ file tools/testing/selftests/rcutorture/initrd/init
> > > tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
> > > executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
> > > linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
> > > GNU/Linux 3.10.0, stripped
> > >
> > > segfaults in QEMU. From one of the log files
> > >
> > >
> > > /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
> > >
> > > [1.119803][T1] Run /init as init process
> > > [1.122011][T1] init[1]: segfault (11) at f0656d90 nip 1a18
> > > lr 0 code 1 in init[1000+d]
> > > [1.124863][T1] init[1]: code: 2c2903e7 f9210030 4081ff84
> > > 4b58 0100 0580 3c40100f
> > > [1.128823][T1] init[1]: code: 38427c00 7c290b78 782106e4
> > > 3800 7c0803a6 f801 e9028010
> > >
> > > Executing the init, which just seems to be an endless loop, from userspace
> > > work:
> > >
> > > $ strace ./tools/testing/selftests/rcutorture/initrd/init
> > > execve("./tools/testing/selftests/rcutorture/initrd/init",
> > > ["./tools/testing/selftests/rcutor"...], 0x7db9e860 /* 31 vars */) = 0
> > > brk(NULL) = 0x1001d94
> > > brk(0x1001d940b98) = 0x1001d940b98
> > > set_tid_address(0x1001d9400d0) = 2890832
> > > set_robust_list(0x1001d9400e0, 24) = 0
> > > uname({sysname="Linux",
> > > nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
> > > prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
> > > rlim_max=RLIM64_INFINITY}) = 0
> > > readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"...,
> > > 4096)
> > > = 61
> > > getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
> > > brk(0x1001d970b98) = 0x1001d970b98
> > > brk(0x1001d98) = 0x1001d98
> > > mprotect(0x100e, 65536, PROT_READ) = 0
> > > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
> > > 0x7b22c8a8) = 0
> > > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
> > > 0x7b22c8a8) = 0
> > > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^C{tv_sec=0,
> > > tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
> > > strace: Process 2890832 detached
> >
> > Huh. In PowerPC, is there some difference between system calls
> > executed in initrd and those same system calls executed in userspace?
> >
> > And just to make sure, the above strace was from exactly the same
> > binary "init" file that is included in initrd, correct?
> >
> > Adding Willy Tarreau for his thoughts.
>
Re: rcutorture’s init segfaults in ppc64le VM
Dear Paul
I am also very interested in the topic.
The Open source lab of Oregon State University has lent me a 8 core
power ppc64el VM for 3 months, I guess I can try reproducing this bug
in the Virtual Machine by executing qemu in non hardware accelerated
mode (using -no-kvm argument).
I am currently doing research on
https://lore.kernel.org/rcu/20220201175023.GW4285@paulmck-ThinkPad-P17-Gen-1/T/#mc7e5f8ec99e3794bec1e38fbbb130e71172e4759,
I think I can give a preliminary short report on that previous topic
tomorrow. And I am very interested in doing a search on the new topic
the day after tomorrow.
Thank you both for providing me an opportunity to improve myself ;-)
Thanks again
Zhouyi
On Tue, Feb 8, 2022 at 12:10 PM Paul E. McKenney wrote:
>
> On Mon, Feb 07, 2022 at 05:44:47PM +0100, Paul Menzel wrote:
> > Dear Linux folks,
> >
> >
> > On the POWER8 server IBM S822LC running Ubuntu 21.10, building Linux
> > 5.17-rc2+ with rcutorture tests
> >
> > $ tools/testing/selftests/rcutorture/bin/torture.sh --duration 10
> >
> > the built init
> >
> > $ file tools/testing/selftests/rcutorture/initrd/init
> > tools/testing/selftests/rcutorture/initrd/init: ELF 64-bit LSB
> > executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), statically
> > linked, BuildID[sha1]=0ded0e45649184a296f30d611f7a03cc51ecb616, for
> > GNU/Linux 3.10.0, stripped
> >
> > segfaults in QEMU. From one of the log files
> >
> >
> > /dev/shm/linux/tools/testing/selftests/rcutorture/res/2022.02.01-21.52.37-torture/results-rcutorture/TREE03/console.log
> >
> > [1.119803][T1] Run /init as init process
> > [1.122011][T1] init[1]: segfault (11) at f0656d90 nip 1a18
> > lr 0 code 1 in init[1000+d]
> > [1.124863][T1] init[1]: code: 2c2903e7 f9210030 4081ff84
> > 4b58 0100 0580 3c40100f
> > [1.128823][T1] init[1]: code: 38427c00 7c290b78 782106e4
> > 3800 7c0803a6 f801 e9028010
> >
> > Executing the init, which just seems to be an endless loop, from userspace
> > work:
> >
> > $ strace ./tools/testing/selftests/rcutorture/initrd/init
> > execve("./tools/testing/selftests/rcutorture/initrd/init",
> > ["./tools/testing/selftests/rcutor"...], 0x7db9e860 /* 31 vars */) = 0
> > brk(NULL) = 0x1001d94
> > brk(0x1001d940b98) = 0x1001d940b98
> > set_tid_address(0x1001d9400d0) = 2890832
> > set_robust_list(0x1001d9400e0, 24) = 0
> > uname({sysname="Linux",
> > nodename="flughafenberlinbrandenburgwillybrandt.molgen.mpg.de", ...}) = 0
> > prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024,
> > rlim_max=RLIM64_INFINITY}) = 0
> > readlink("/proc/self/exe", "/dev/shm/linux/tools/testing/sel"..., 4096)
> > = 61
> > getrandom("\xf1\x30\x4c\x9e\x82\x8d\x26\xd7", 8, GRND_NONBLOCK) = 8
> > brk(0x1001d970b98) = 0x1001d970b98
> > brk(0x1001d98) = 0x1001d98
> > mprotect(0x100e, 65536, PROT_READ) = 0
> > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
> > 0x7b22c8a8) = 0
> > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0},
> > 0x7b22c8a8) = 0
> > clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, ^C{tv_sec=0,
> > tv_nsec=872674044}) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
> > strace: Process 2890832 detached
>
> Huh. In PowerPC, is there some difference between system calls
> executed in initrd and those same system calls executed in userspace?
>
> And just to make sure, the above strace was from exactly the same
> binary "init" file that is included in initrd, correct?
>
> Adding Willy Tarreau for his thoughts.
>
> Thanx, Paul
>
> > Any ideas, what `mkinitrd.sh` [2] should do differently?
> >
> > ```
> > cat > init.c << '___EOF___'
> > #ifndef NOLIBC
> > #include
> > #include
> > #endif
> >
> > volatile unsigned long delaycount;
> >
> > int main(int argc, int argv[])
> > {
> > int i;
> > struct timeval tv;
> > struct timeval tvb;
> >
> > for (;;) {
> > sleep(1);
> > /* Need some userspace time. */
> > if (gettimeofday(, NULL))
> > continue;
> > do {
> > for (i = 0; i < 1000 * 100; i++)
> > delaycount = i * i;
> > if (gettimeofday(, NULL))
> > break;
> > tv.tv_sec -= tvb.tv_sec;
> > if (tv.tv_sec > 1)
> > break;
> > tv.tv_usec += tv.tv_sec * 1000 * 1000;
> > tv.tv_usec -= tvb.tv_usec;
> > } while (tv.tv_usec < 1000);
> > }
> > return 0;
> > }
> > ___EOF___
> >
> > # build using nolibc on supported archs (smaller executable)
[PATCH 1/1] powerpc/jump_label: use HAVE_JUMP_LABEL?
CONFIG_JUMP_LABEL doesn't ensure HAVE_JUMP_LABEL, if it is not the case use maintainers's own mutex to guard the modification of global values. Signed-off-by: Zhouyi Zhou yizhouz...@ict.ac.cn --- arch/powerpc/platforms/powernv/opal-tracepoints.c |2 +- arch/powerpc/platforms/pseries/lpar.c |2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/platforms/powernv/opal-tracepoints.c b/arch/powerpc/platforms/powernv/opal-tracepoints.c index d8a000a..ae14c40 100644 --- a/arch/powerpc/platforms/powernv/opal-tracepoints.c +++ b/arch/powerpc/platforms/powernv/opal-tracepoints.c @@ -2,7 +2,7 @@ #include linux/jump_label.h #include asm/trace.h -#ifdef CONFIG_JUMP_LABEL +#ifdef HAVE_JUMP_LABEL struct static_key opal_tracepoint_key = STATIC_KEY_INIT; void opal_tracepoint_regfunc(void) diff --git a/arch/powerpc/platforms/pseries/lpar.c b/arch/powerpc/platforms/pseries/lpar.c index 34e6423..059cfe0 100644 --- a/arch/powerpc/platforms/pseries/lpar.c +++ b/arch/powerpc/platforms/pseries/lpar.c @@ -642,7 +642,7 @@ EXPORT_SYMBOL(arch_free_page); #endif #ifdef CONFIG_TRACEPOINTS -#ifdef CONFIG_JUMP_LABEL +#ifdef HAVE_JUMP_LABEL struct static_key hcall_tracepoint_key = STATIC_KEY_INIT; void hcall_tracepoint_regfunc(void) -- 1.7.10.4 ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH 1/1] powerpc/iommu: Handling null return of kzalloc_node
NULL return of kzalloc_node should be handled
Signed-off-by: Zhouyi Zhou yizhouz...@ict.ac.cn
---
arch/powerpc/platforms/pseries/iommu.c | 22 --
1 file changed, 20 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/pseries/iommu.c
b/arch/powerpc/platforms/pseries/iommu.c
index 33b552f..593cd3d 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -613,7 +613,11 @@ static void pci_dma_bus_setup_pSeries(struct pci_bus *bus)
tbl = kzalloc_node(sizeof(struct iommu_table), GFP_KERNEL,
pci-phb-node);
-
+ if (!tbl) {
+ pr_debug( out of memory, can't create iommu_table !\n);
+ return;
+ }
+
iommu_table_setparms(pci-phb, dn, tbl);
pci-iommu_table = iommu_init_table(tbl, pci-phb-node);
iommu_register_group(tbl, pci_domain_nr(bus), 0);
@@ -659,6 +663,10 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus
*bus)
if (!ppci-iommu_table) {
tbl = kzalloc_node(sizeof(struct iommu_table), GFP_KERNEL,
ppci-phb-node);
+ if (!tbl) {
+ pr_debug( out of memory, can't create iommu_table
!\n);
+ return;
+ }
iommu_table_setparms_lpar(ppci-phb, pdn, tbl, dma_window);
ppci-iommu_table = iommu_init_table(tbl, ppci-phb-node);
iommu_register_group(tbl, pci_domain_nr(bus), 0);
@@ -686,6 +694,11 @@ static void pci_dma_dev_setup_pSeries(struct pci_dev *dev)
pr_debug( -- first child, no bridge. Allocating iommu
table.\n);
tbl = kzalloc_node(sizeof(struct iommu_table), GFP_KERNEL,
phb-node);
+ if (!tbl) {
+ pr_debug( out of memory, can't create iommu_table
!\n);
+ return;
+ }
+
iommu_table_setparms(phb, dn, tbl);
PCI_DN(dn)-iommu_table = iommu_init_table(tbl, phb-node);
iommu_register_group(tbl, pci_domain_nr(phb-bus), 0);
@@ -1102,6 +1116,10 @@ static void pci_dma_dev_setup_pSeriesLP(struct pci_dev
*dev)
if (!pci-iommu_table) {
tbl = kzalloc_node(sizeof(struct iommu_table), GFP_KERNEL,
pci-phb-node);
+ if (!tbl) {
+ pr_debug( out of memory, can't create iommu_table
!\n);
+ return;
+ }
iommu_table_setparms_lpar(pci-phb, pdn, tbl, dma_window);
pci-iommu_table = iommu_init_table(tbl, pci-phb-node);
iommu_register_group(tbl, pci_domain_nr(pci-phb-bus), 0);
--
1.7.10.4
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
ping: re: [PATCH 1/1] kernel code that do not handle NULL return of kmem_cache_zalloc
ping
I do a grep for kmem_cache_zalloc and kmem_cache_alloc
in kernel tree, and find some code do not handle NULL
return of kmem_cache_zalloc correctly
Signed-off-by: Zhouyi Zhou yizhouz...@ict.ac.cn
---
arch/powerpc/kvm/book3s_32_mmu_host.c |5 +
drivers/iommu/omap-iommu.c|3 ++-
fs/jffs2/malloc.c |4
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kvm/book3s_32_mmu_host.c
b/arch/powerpc/kvm/book3s_32_mmu_host.c
index 3a0abd2..5fac89d 100644
--- a/arch/powerpc/kvm/book3s_32_mmu_host.c
+++ b/arch/powerpc/kvm/book3s_32_mmu_host.c
@@ -243,6 +243,11 @@ next_pteg:
/* Now tell our Shadow PTE code about the new page */
pte = kvmppc_mmu_hpte_cache_next(vcpu);
+ if (!pte) {
+ kvm_release_pfn_clean(hpaddr PAGE_SHIFT);
+ r = -EAGAIN;
+ goto out;
+ }
dprintk_mmu(KVM: %c%c Map 0x%llx: [%lx] 0x%llx (0x%llx) - %lx\n,
orig_pte-may_write ? 'w' : '-',
diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c
index bcd78a7..5155714 100644
--- a/drivers/iommu/omap-iommu.c
+++ b/drivers/iommu/omap-iommu.c
@@ -551,7 +551,8 @@ static u32 *iopte_alloc(struct omap_iommu *obj, u32
*iopgd, u32 da)
dev_vdbg(obj-dev, %s: a new pte:%p\n, __func__, iopte);
} else {
/* We raced, free the reduniovant table */
- iopte_free(iopte);
+ if (iopte)
+ iopte_free(iopte);
}
pte_ready:
diff --git a/fs/jffs2/malloc.c b/fs/jffs2/malloc.c
index 4f47aa2..58e2336 100644
--- a/fs/jffs2/malloc.c
+++ b/fs/jffs2/malloc.c
@@ -287,6 +287,8 @@ struct jffs2_xattr_datum *jffs2_alloc_xattr_datum(void)
{
struct jffs2_xattr_datum *xd;
xd = kmem_cache_zalloc(xattr_datum_cache, GFP_KERNEL);
+ if (!xd)
+ return NULL;
dbg_memalloc(%p\n, xd);
xd-class = RAWNODE_CLASS_XATTR_DATUM;
@@ -305,6 +307,8 @@ struct jffs2_xattr_ref *jffs2_alloc_xattr_ref(void)
{
struct jffs2_xattr_ref *ref;
ref = kmem_cache_zalloc(xattr_ref_cache, GFP_KERNEL);
+ if (!ref)
+ return NULL;
dbg_memalloc(%p\n, ref);
ref-class = RAWNODE_CLASS_XATTR_REF;
--
1.7.10.4
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH 1/1] kernel code that do not handle NULL return of kmem_cache_zalloc
I do a grep for kmem_cache_zalloc and kmem_cache_alloc
in kernel tree, and find some code do not handle NULL
return of kmem_cache_zalloc correctly
Signed-off-by: Zhouyi Zhou yizhouz...@ict.ac.cn
---
arch/powerpc/kvm/book3s_32_mmu_host.c |5 +
drivers/iommu/omap-iommu.c|3 ++-
fs/jffs2/malloc.c |4
3 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kvm/book3s_32_mmu_host.c
b/arch/powerpc/kvm/book3s_32_mmu_host.c
index 3a0abd2..5fac89d 100644
--- a/arch/powerpc/kvm/book3s_32_mmu_host.c
+++ b/arch/powerpc/kvm/book3s_32_mmu_host.c
@@ -243,6 +243,11 @@ next_pteg:
/* Now tell our Shadow PTE code about the new page */
pte = kvmppc_mmu_hpte_cache_next(vcpu);
+ if (!pte) {
+ kvm_release_pfn_clean(hpaddr PAGE_SHIFT);
+ r = -EAGAIN;
+ goto out;
+ }
dprintk_mmu(KVM: %c%c Map 0x%llx: [%lx] 0x%llx (0x%llx) - %lx\n,
orig_pte-may_write ? 'w' : '-',
diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c
index bcd78a7..5155714 100644
--- a/drivers/iommu/omap-iommu.c
+++ b/drivers/iommu/omap-iommu.c
@@ -551,7 +551,8 @@ static u32 *iopte_alloc(struct omap_iommu *obj, u32 *iopgd,
u32 da)
dev_vdbg(obj-dev, %s: a new pte:%p\n, __func__, iopte);
} else {
/* We raced, free the reduniovant table */
- iopte_free(iopte);
+ if (iopte)
+ iopte_free(iopte);
}
pte_ready:
diff --git a/fs/jffs2/malloc.c b/fs/jffs2/malloc.c
index 4f47aa2..58e2336 100644
--- a/fs/jffs2/malloc.c
+++ b/fs/jffs2/malloc.c
@@ -287,6 +287,8 @@ struct jffs2_xattr_datum *jffs2_alloc_xattr_datum(void)
{
struct jffs2_xattr_datum *xd;
xd = kmem_cache_zalloc(xattr_datum_cache, GFP_KERNEL);
+ if (!xd)
+ return NULL;
dbg_memalloc(%p\n, xd);
xd-class = RAWNODE_CLASS_XATTR_DATUM;
@@ -305,6 +307,8 @@ struct jffs2_xattr_ref *jffs2_alloc_xattr_ref(void)
{
struct jffs2_xattr_ref *ref;
ref = kmem_cache_zalloc(xattr_ref_cache, GFP_KERNEL);
+ if (!ref)
+ return NULL;
dbg_memalloc(%p\n, ref);
ref-class = RAWNODE_CLASS_XATTR_REF;
--
1.7.10.4
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
