On Thu, Sep 17, 2026 at 03:58:42PM +0530, Shrikanth Hegde wrote:
> Hi Michal,
> 
> On 9/17/26 3:44 PM, Michal Suchánek wrote:
> > Hello,
> > 
> > On Wed, Sep 16, 2026 at 11:41:38PM +0530, Shrikanth Hegde wrote:
> > > Hi Michal,
> > > 
> > > > 
> > > > It seems easy to re-create.
> > > > The key is running bcc/bpf program in parallel to kernel build.
> > > > 
> > > > For example, these two in parallal, leads to stack corruption panic.
> > > > Stacktrace does differ sometimes.
> > > > 
> > > > ./funccount  rcu* -d 100
> > > > make -j 32
> > > > 
> > > > Boom. Will try to debug this further.
> > > 
> > > A bit more debugging.
> > > 
> > > 1. A few functions seems to crash, specially ones that call schedule.
> > >     So picked schedule itself for tracing
> > > 2. Used plain ftrace. No crash.
> > > 3. Used kprobes. Boom. Crash appears. But this helped to avoid BPF JIT, 
> > > or BCC part of it.
> > > 4. Did some preempt count validation with annotations. Seems to be 
> > > holding fine.
> > > 5. Since unlock can call schedule while tracing schedule, there is 
> > > possible looping.
> > >     Need to see further why it happens to be crashing there. Maybe regs 
> > > or paca
> > >     handling needs a closer look.
> > > 
> > > For now, below patch helps to keep the system stable under kprobes, bcc 
> > > tools.
> > > I will look further into see why exactly it is crashing and below one is 
> > > going to
> > > be the proper fix or not.
> > > 
> > > Feel free to give it a try.
> > 
> > thanks for the patch.
> > 
> > While it is mangled by the e-mail client in different way compared to
> > the previous patch it does seem to mitigate the issue.
> 
> Thanks for trying.
> 
> > 
> > As it is unlikely the board where KVM lockup was originally observed is
> > running kpbobes it will probably not address the original problem.
> > 
> 
> Give it a try.
> 
> I am suspecting both lockup, stack corruptions are secondary symptoms here.
> While recreate, I have different type of traces.

Yes, it does work for the crash I see. However, we hijacked the report
of KVM locking up on FSL Cyrus+, and that is likely a different problem
still not addressed.

Thanks

Michal

> 
> > Thanks
> > 
> > Michal
> > 
> > > 
> > > ---
> > >   arch/powerpc/kernel/kprobes-ftrace.c | 2 ++
> > >   1 file changed, 2 insertions(+)
> > > 
> > > diff --git a/arch/powerpc/kernel/kprobes-ftrace.c 
> > > b/arch/powerpc/kernel/kprobes-ftrace.c
> > > index f8208c027148..bcb1e8a8995d 100644
> > > --- a/arch/powerpc/kernel/kprobes-ftrace.c
> > > +++ b/arch/powerpc/kernel/kprobes-ftrace.c
> > > @@ -63,7 +63,9 @@ void kprobe_ftrace_handler(unsigned long nip, unsigned 
> > > long parent_nip,
> > >                   __this_cpu_write(current_kprobe, NULL);
> > >           }
> > >   out:
> > > + preempt_disable_notrace();
> > >           ftrace_test_recursion_unlock(bit);
> > > + preempt_enable_no_resched_notrace();
> > >   }
> > >   NOKPROBE_SYMBOL(kprobe_ftrace_handler);
> > > -- 
> > > 2.52.0
> 

Reply via email to