RE: ftrace function_graph causes system crash
> -Original Message- > From: Steven Rostedt [mailto:rost...@goodmis.org] > Sent: Mittwoch, 21. September 2016 20:17 > To: Jisheng Zhang <jszh...@marvell.com> > Cc: Bean Huo (beanhuo) <bean...@micron.com>; Zoltan Szubbocsev > (zszubbocsev) <zszubboc...@micron.com>; catalin.mari...@arm.com; > will.dea...@arm.com; r...@lists.rocketboards.org; linux- > ker...@vger.kernel.org; mi...@redhat.com; linux-arm- > ker...@lists.infradead.org > Subject: Re: ftrace function_graph causes system crash > > On Wed, 21 Sep 2016 17:13:07 +0800 > Jisheng Zhang <jszh...@marvell.com> wrote: > > > I'm not sure whether the commit d6df3576e6b4 > ("clocksource/drivers/arm_global_timer > > : Prevent ftrace recursion") can fix this issue. > > > > this commit is merged since v4.3, I noticed your kernel version is v4.0 > > BTW, yes, that would be the fix. > > -- Steve > > > > > Thanks, > > Jisheng > > > > > Do you know now how to deeply debug and trace which line is wrong > through Ftrace? > > > Hi, Steven and Jisheng Thanks to both warm-hearted guys. I merged d6df3576e6b4 patch into my kernel 4.0. Then it is true, no cash appears again. I have one more question that current ftrace can trace DMA latency, include mapping and unmapping? Means I want to know when one BIO request be completed. Just like blktrace. But blktrace can not tell me the function calling sequence. --Bean
RE: ftrace function_graph causes system crash
> -Original Message- > From: Steven Rostedt [mailto:rost...@goodmis.org] > Sent: Mittwoch, 21. September 2016 20:17 > To: Jisheng Zhang > Cc: Bean Huo (beanhuo) ; Zoltan Szubbocsev > (zszubbocsev) ; catalin.mari...@arm.com; > will.dea...@arm.com; r...@lists.rocketboards.org; linux- > ker...@vger.kernel.org; mi...@redhat.com; linux-arm- > ker...@lists.infradead.org > Subject: Re: ftrace function_graph causes system crash > > On Wed, 21 Sep 2016 17:13:07 +0800 > Jisheng Zhang wrote: > > > I'm not sure whether the commit d6df3576e6b4 > ("clocksource/drivers/arm_global_timer > > : Prevent ftrace recursion") can fix this issue. > > > > this commit is merged since v4.3, I noticed your kernel version is v4.0 > > BTW, yes, that would be the fix. > > -- Steve > > > > > Thanks, > > Jisheng > > > > > Do you know now how to deeply debug and trace which line is wrong > through Ftrace? > > > Hi, Steven and Jisheng Thanks to both warm-hearted guys. I merged d6df3576e6b4 patch into my kernel 4.0. Then it is true, no cash appears again. I have one more question that current ftrace can trace DMA latency, include mapping and unmapping? Means I want to know when one BIO request be completed. Just like blktrace. But blktrace can not tell me the function calling sequence. --Bean
Re: ftrace function_graph causes system crash
On Wed, 21 Sep 2016 17:13:07 +0800 Jisheng Zhangwrote: > I'm not sure whether the commit d6df3576e6b4 > ("clocksource/drivers/arm_global_timer > : Prevent ftrace recursion") can fix this issue. > > this commit is merged since v4.3, I noticed your kernel version is v4.0 BTW, yes, that would be the fix. -- Steve > > Thanks, > Jisheng > > > Do you know now how to deeply debug and trace which line is wrong through > > Ftrace? > >
Re: ftrace function_graph causes system crash
On Wed, 21 Sep 2016 17:13:07 +0800 Jisheng Zhang wrote: > I'm not sure whether the commit d6df3576e6b4 > ("clocksource/drivers/arm_global_timer > : Prevent ftrace recursion") can fix this issue. > > this commit is merged since v4.3, I noticed your kernel version is v4.0 BTW, yes, that would be the fix. -- Steve > > Thanks, > Jisheng > > > Do you know now how to deeply debug and trace which line is wrong through > > Ftrace? > >
Re: ftrace function_graph causes system crash
On Wed, 21 Sep 2016 07:50:58 + "Bean Huo (beanhuo)"wrote: > > Hi, Steve > Thanks very much! This is a very useful trace tool, I now know the problem > function, > It is gt_counter_read, if not trace this function, ftrace function_graph work > well. > Do you know now how to deeply debug and trace which line is wrong through > Ftrace? Hmm, maybe I should add this into the scripts directory. Yeah, I should do that. -- Steve
Re: ftrace function_graph causes system crash
On Wed, 21 Sep 2016 07:50:58 + "Bean Huo (beanhuo)" wrote: > > Hi, Steve > Thanks very much! This is a very useful trace tool, I now know the problem > function, > It is gt_counter_read, if not trace this function, ftrace function_graph work > well. > Do you know now how to deeply debug and trace which line is wrong through > Ftrace? Hmm, maybe I should add this into the scripts directory. Yeah, I should do that. -- Steve
Re: ftrace function_graph causes system crash
Hi Bean, On Wed, 21 Sep 2016 07:50:58 + "Bean Huo (beanhuo)" wrote: > > From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] > > On Behalf Of Steven Rostedt > > Sent: Dienstag, 20. September 2016 16:07 > > To: Bean Huo (beanhuo) <bean...@micron.com> > > Cc: Zoltan Szubbocsev (zszubbocsev) <zszubboc...@micron.com>; > > catalin.mari...@arm.com; will.dea...@arm.com; r...@lists.rocketboards.org; > > linux-kernel@vger.kernel.org; mi...@redhat.com; linux-arm- > > ker...@lists.infradead.org > > Subject: Re: ftrace function_graph causes system crash > > > > On Tue, 20 Sep 2016 13:10:39 + > > "Bean Huo (beanhuo)" <bean...@micron.com> wrote: > > > > > Hi, all > > > I just use ftrace to do some latency study, found that function_graph > > > can not Work, as long as enable it, will cause kernel panic. I searched > > > this > > online. > > > Found that there are also some cause the same as mine. I am a newer of > > ftrace. > > > I want to know who know what root cause? Here is some partial log: > > > > > > > > > > Can you do a function bisect to find what function this is. > > > > This script is used to help find functions that are being traced by > > function tracer > > or function graph tracing that causes the machine to reboot, hang, or crash. > > Here's the steps to take. > > > > First, determine if function graph is working with a single function: > > > > # cd /sys/kernel/debug/tracing > > # echo schedule > set_ftrace_filter > > # echo function_graph > current_tracer > > > > If this works, then we know that something is being traced that shouldn't > > be. > > > > # echo nop > current_tracer > > > > # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file > > ~/test-file > > ~/non-test-file # cat ~/test-file > set_ftrace_filter > > > > *** Note *** this will take several minutes. Setting multiple functions is > > an > > O(n^2) operation, and we are dealing with thousands of functions. > > So go have coffee, talk with your coworkers, read facebook. And eventually, > > this operation will end. > > > > # echo function_graph > current_tracer > > > > If it crashes, we know that ~/test-file has a bad function. > > > >Reboot back to test kernel. > > > ># cd /sys/kernel/debug/tracing > ># mv ~/test-file ~/full-file > > > > If it didn't crash. > > > ># echo nop > current_tracer > ># mv ~/non-test-file ~/full-file > > > > Get rid of the other test file from previous run (or save them off > > somewhere. > > # rm -f ~/test-file ~/non-test-file > > > > And start again: > > > > # ftrace-bisect ~/full-file ~/test-file ~/non-test-file > > > > The good thing is, because this cuts the number of functions in ~/test-file > > by half, > > the cat of it into set_ftrace_filter takes half as long each iteration, so > > don't talk > > so much at the water cooler the second time. > > > > Eventually, if you did this correctly, you will get down to the problem > > function, > > and all we need to do is to notrace it. > > > > The way to figure out if the problem function is bad, just do: > > > > # echo > set_ftrace_notrace # echo > set_ftrace_filter # > > echo function_graph > current_tracer > > > > And if it doesn't crash, we are done. > > > > -- Steve > > > Hi, Steve > Thanks very much! This is a very useful trace tool, I now know the problem > function, > It is gt_counter_read, if not trace this function, ftrace function_graph work > well. I'm not sure whether the commit d6df3576e6b4 ("clocksource/drivers/arm_global_timer : Prevent ftrace recursion") can fix this issue. this commit is merged since v4.3, I noticed your kernel version is v4.0 Thanks, Jisheng > Do you know now how to deeply debug and trace which line is wrong through > Ftrace? > > --Bean > > ___ > linux-arm-kernel mailing list > linux-arm-ker...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: ftrace function_graph causes system crash
Hi Bean, On Wed, 21 Sep 2016 07:50:58 + "Bean Huo (beanhuo)" wrote: > > From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] > > On Behalf Of Steven Rostedt > > Sent: Dienstag, 20. September 2016 16:07 > > To: Bean Huo (beanhuo) > > Cc: Zoltan Szubbocsev (zszubbocsev) ; > > catalin.mari...@arm.com; will.dea...@arm.com; r...@lists.rocketboards.org; > > linux-kernel@vger.kernel.org; mi...@redhat.com; linux-arm- > > ker...@lists.infradead.org > > Subject: Re: ftrace function_graph causes system crash > > > > On Tue, 20 Sep 2016 13:10:39 + > > "Bean Huo (beanhuo)" wrote: > > > > > Hi, all > > > I just use ftrace to do some latency study, found that function_graph > > > can not Work, as long as enable it, will cause kernel panic. I searched > > > this > > online. > > > Found that there are also some cause the same as mine. I am a newer of > > ftrace. > > > I want to know who know what root cause? Here is some partial log: > > > > > > > > > > Can you do a function bisect to find what function this is. > > > > This script is used to help find functions that are being traced by > > function tracer > > or function graph tracing that causes the machine to reboot, hang, or crash. > > Here's the steps to take. > > > > First, determine if function graph is working with a single function: > > > > # cd /sys/kernel/debug/tracing > > # echo schedule > set_ftrace_filter > > # echo function_graph > current_tracer > > > > If this works, then we know that something is being traced that shouldn't > > be. > > > > # echo nop > current_tracer > > > > # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file > > ~/test-file > > ~/non-test-file # cat ~/test-file > set_ftrace_filter > > > > *** Note *** this will take several minutes. Setting multiple functions is > > an > > O(n^2) operation, and we are dealing with thousands of functions. > > So go have coffee, talk with your coworkers, read facebook. And eventually, > > this operation will end. > > > > # echo function_graph > current_tracer > > > > If it crashes, we know that ~/test-file has a bad function. > > > >Reboot back to test kernel. > > > ># cd /sys/kernel/debug/tracing > ># mv ~/test-file ~/full-file > > > > If it didn't crash. > > > ># echo nop > current_tracer > ># mv ~/non-test-file ~/full-file > > > > Get rid of the other test file from previous run (or save them off > > somewhere. > > # rm -f ~/test-file ~/non-test-file > > > > And start again: > > > > # ftrace-bisect ~/full-file ~/test-file ~/non-test-file > > > > The good thing is, because this cuts the number of functions in ~/test-file > > by half, > > the cat of it into set_ftrace_filter takes half as long each iteration, so > > don't talk > > so much at the water cooler the second time. > > > > Eventually, if you did this correctly, you will get down to the problem > > function, > > and all we need to do is to notrace it. > > > > The way to figure out if the problem function is bad, just do: > > > > # echo > set_ftrace_notrace # echo > set_ftrace_filter # > > echo function_graph > current_tracer > > > > And if it doesn't crash, we are done. > > > > -- Steve > > > Hi, Steve > Thanks very much! This is a very useful trace tool, I now know the problem > function, > It is gt_counter_read, if not trace this function, ftrace function_graph work > well. I'm not sure whether the commit d6df3576e6b4 ("clocksource/drivers/arm_global_timer : Prevent ftrace recursion") can fix this issue. this commit is merged since v4.3, I noticed your kernel version is v4.0 Thanks, Jisheng > Do you know now how to deeply debug and trace which line is wrong through > Ftrace? > > --Bean > > ___ > linux-arm-kernel mailing list > linux-arm-ker...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
RE: ftrace function_graph causes system crash
> From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] > On Behalf Of Steven Rostedt > Sent: Dienstag, 20. September 2016 16:07 > To: Bean Huo (beanhuo) <bean...@micron.com> > Cc: Zoltan Szubbocsev (zszubbocsev) <zszubboc...@micron.com>; > catalin.mari...@arm.com; will.dea...@arm.com; r...@lists.rocketboards.org; > linux-kernel@vger.kernel.org; mi...@redhat.com; linux-arm- > ker...@lists.infradead.org > Subject: Re: ftrace function_graph causes system crash > > On Tue, 20 Sep 2016 13:10:39 + > "Bean Huo (beanhuo)" <bean...@micron.com> wrote: > > > Hi, all > > I just use ftrace to do some latency study, found that function_graph > > can not Work, as long as enable it, will cause kernel panic. I searched this > online. > > Found that there are also some cause the same as mine. I am a newer of > ftrace. > > I want to know who know what root cause? Here is some partial log: > > > > > > Can you do a function bisect to find what function this is. > > This script is used to help find functions that are being traced by function > tracer > or function graph tracing that causes the machine to reboot, hang, or crash. > Here's the steps to take. > > First, determine if function graph is working with a single function: > > # cd /sys/kernel/debug/tracing > # echo schedule > set_ftrace_filter > # echo function_graph > current_tracer > > If this works, then we know that something is being traced that shouldn't be. > > # echo nop > current_tracer > > # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file > ~/test-file > ~/non-test-file # cat ~/test-file > set_ftrace_filter > > *** Note *** this will take several minutes. Setting multiple functions is an > O(n^2) operation, and we are dealing with thousands of functions. > So go have coffee, talk with your coworkers, read facebook. And eventually, > this operation will end. > > # echo function_graph > current_tracer > > If it crashes, we know that ~/test-file has a bad function. > >Reboot back to test kernel. > ># cd /sys/kernel/debug/tracing ># mv ~/test-file ~/full-file > > If it didn't crash. > ># echo nop > current_tracer ># mv ~/non-test-file ~/full-file > > Get rid of the other test file from previous run (or save them off somewhere. > # rm -f ~/test-file ~/non-test-file > > And start again: > > # ftrace-bisect ~/full-file ~/test-file ~/non-test-file > > The good thing is, because this cuts the number of functions in ~/test-file > by half, > the cat of it into set_ftrace_filter takes half as long each iteration, so > don't talk > so much at the water cooler the second time. > > Eventually, if you did this correctly, you will get down to the problem > function, > and all we need to do is to notrace it. > > The way to figure out if the problem function is bad, just do: > > # echo > set_ftrace_notrace # echo > set_ftrace_filter # > echo function_graph > current_tracer > > And if it doesn't crash, we are done. > > -- Steve Hi, Steve Thanks very much! This is a very useful trace tool, I now know the problem function, It is gt_counter_read, if not trace this function, ftrace function_graph work well. Do you know now how to deeply debug and trace which line is wrong through Ftrace? --Bean
RE: ftrace function_graph causes system crash
> From: linux-arm-kernel [mailto:linux-arm-kernel-boun...@lists.infradead.org] > On Behalf Of Steven Rostedt > Sent: Dienstag, 20. September 2016 16:07 > To: Bean Huo (beanhuo) > Cc: Zoltan Szubbocsev (zszubbocsev) ; > catalin.mari...@arm.com; will.dea...@arm.com; r...@lists.rocketboards.org; > linux-kernel@vger.kernel.org; mi...@redhat.com; linux-arm- > ker...@lists.infradead.org > Subject: Re: ftrace function_graph causes system crash > > On Tue, 20 Sep 2016 13:10:39 + > "Bean Huo (beanhuo)" wrote: > > > Hi, all > > I just use ftrace to do some latency study, found that function_graph > > can not Work, as long as enable it, will cause kernel panic. I searched this > online. > > Found that there are also some cause the same as mine. I am a newer of > ftrace. > > I want to know who know what root cause? Here is some partial log: > > > > > > Can you do a function bisect to find what function this is. > > This script is used to help find functions that are being traced by function > tracer > or function graph tracing that causes the machine to reboot, hang, or crash. > Here's the steps to take. > > First, determine if function graph is working with a single function: > > # cd /sys/kernel/debug/tracing > # echo schedule > set_ftrace_filter > # echo function_graph > current_tracer > > If this works, then we know that something is being traced that shouldn't be. > > # echo nop > current_tracer > > # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file > ~/test-file > ~/non-test-file # cat ~/test-file > set_ftrace_filter > > *** Note *** this will take several minutes. Setting multiple functions is an > O(n^2) operation, and we are dealing with thousands of functions. > So go have coffee, talk with your coworkers, read facebook. And eventually, > this operation will end. > > # echo function_graph > current_tracer > > If it crashes, we know that ~/test-file has a bad function. > >Reboot back to test kernel. > ># cd /sys/kernel/debug/tracing ># mv ~/test-file ~/full-file > > If it didn't crash. > ># echo nop > current_tracer ># mv ~/non-test-file ~/full-file > > Get rid of the other test file from previous run (or save them off somewhere. > # rm -f ~/test-file ~/non-test-file > > And start again: > > # ftrace-bisect ~/full-file ~/test-file ~/non-test-file > > The good thing is, because this cuts the number of functions in ~/test-file > by half, > the cat of it into set_ftrace_filter takes half as long each iteration, so > don't talk > so much at the water cooler the second time. > > Eventually, if you did this correctly, you will get down to the problem > function, > and all we need to do is to notrace it. > > The way to figure out if the problem function is bad, just do: > > # echo > set_ftrace_notrace # echo > set_ftrace_filter # > echo function_graph > current_tracer > > And if it doesn't crash, we are done. > > -- Steve Hi, Steve Thanks very much! This is a very useful trace tool, I now know the problem function, It is gt_counter_read, if not trace this function, ftrace function_graph work well. Do you know now how to deeply debug and trace which line is wrong through Ftrace? --Bean
Re: ftrace function_graph causes system crash
On Tue, 20 Sep 2016 13:10:39 + "Bean Huo (beanhuo)"wrote: > Hi, all > I just use ftrace to do some latency study, found that function_graph can not > Work, as long as enable it, will cause kernel panic. I searched this online. > Found that there are also some cause the same as mine. I am a newer of > ftrace. > I want to know who know what root cause? Here is some partial log: > > Can you do a function bisect to find what function this is. This script is used to help find functions that are being traced by function tracer or function graph tracing that causes the machine to reboot, hang, or crash. Here's the steps to take. First, determine if function graph is working with a single function: # cd /sys/kernel/debug/tracing # echo schedule > set_ftrace_filter # echo function_graph > current_tracer If this works, then we know that something is being traced that shouldn't be. # echo nop > current_tracer # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file ~/test-file ~/non-test-file # cat ~/test-file > set_ftrace_filter *** Note *** this will take several minutes. Setting multiple functions is an O(n^2) operation, and we are dealing with thousands of functions. So go have coffee, talk with your coworkers, read facebook. And eventually, this operation will end. # echo function_graph > current_tracer If it crashes, we know that ~/test-file has a bad function. Reboot back to test kernel. # cd /sys/kernel/debug/tracing # mv ~/test-file ~/full-file If it didn't crash. # echo nop > current_tracer # mv ~/non-test-file ~/full-file Get rid of the other test file from previous run (or save them off somewhere. # rm -f ~/test-file ~/non-test-file And start again: # ftrace-bisect ~/full-file ~/test-file ~/non-test-file The good thing is, because this cuts the number of functions in ~/test-file by half, the cat of it into set_ftrace_filter takes half as long each iteration, so don't talk so much at the water cooler the second time. Eventually, if you did this correctly, you will get down to the problem function, and all we need to do is to notrace it. The way to figure out if the problem function is bad, just do: # echo > set_ftrace_notrace # echo > set_ftrace_filter # echo function_graph > current_tracer And if it doesn't crash, we are done. -- Steve ftrace-bisect Description: Binary data
Re: ftrace function_graph causes system crash
On Tue, 20 Sep 2016 13:10:39 + "Bean Huo (beanhuo)" wrote: > Hi, all > I just use ftrace to do some latency study, found that function_graph can not > Work, as long as enable it, will cause kernel panic. I searched this online. > Found that there are also some cause the same as mine. I am a newer of > ftrace. > I want to know who know what root cause? Here is some partial log: > > Can you do a function bisect to find what function this is. This script is used to help find functions that are being traced by function tracer or function graph tracing that causes the machine to reboot, hang, or crash. Here's the steps to take. First, determine if function graph is working with a single function: # cd /sys/kernel/debug/tracing # echo schedule > set_ftrace_filter # echo function_graph > current_tracer If this works, then we know that something is being traced that shouldn't be. # echo nop > current_tracer # cat available_filter_functions > ~/full-file # ftrace-bisect ~/full-file ~/test-file ~/non-test-file # cat ~/test-file > set_ftrace_filter *** Note *** this will take several minutes. Setting multiple functions is an O(n^2) operation, and we are dealing with thousands of functions. So go have coffee, talk with your coworkers, read facebook. And eventually, this operation will end. # echo function_graph > current_tracer If it crashes, we know that ~/test-file has a bad function. Reboot back to test kernel. # cd /sys/kernel/debug/tracing # mv ~/test-file ~/full-file If it didn't crash. # echo nop > current_tracer # mv ~/non-test-file ~/full-file Get rid of the other test file from previous run (or save them off somewhere. # rm -f ~/test-file ~/non-test-file And start again: # ftrace-bisect ~/full-file ~/test-file ~/non-test-file The good thing is, because this cuts the number of functions in ~/test-file by half, the cat of it into set_ftrace_filter takes half as long each iteration, so don't talk so much at the water cooler the second time. Eventually, if you did this correctly, you will get down to the problem function, and all we need to do is to notrace it. The way to figure out if the problem function is bad, just do: # echo > set_ftrace_notrace # echo > set_ftrace_filter # echo function_graph > current_tracer And if it doesn't crash, we are done. -- Steve ftrace-bisect Description: Binary data
ftrace function_graph causes system crash
Hi, all I just use ftrace to do some latency study, found that function_graph can not Work, as long as enable it, will cause kernel panic. I searched this online. Found that there are also some cause the same as mine. I am a newer of ftrace. I want to know who know what root cause? Here is some partial log: echo function_graph > current_tracer [9.583813] Unable to handle kernel paging request at virtual address b0200083 [9.590997] pgd = c0004000 [9.593683] [b0200083] *pgd= [9.597253] Internal error: Oops: 5 [#1] PREEMPT SMP ARM [9.602542] Modules linked in: [9.605586] CPU: 1 PID: 15 Comm: kworker/1:0 Not tainted 4.0.0-xilinx-00043-gc701690-dirty #515 [9.614256] Hardware name: Xilinx Zynq Platform [9.618793] Workqueue: 0xe3a1 ( “å) [9.622765] task: df517500 ti: df518000 task.ti: df518000 [9.628162] PC is at rb_update_write_stamp+0x18/0xa0 [9.633100] LR is at rb_commit+0x34/0xdc [9.637005] pc : []lr : []psr: 6093 [9.637005] sp : df51a0f0 ip : df40b00c fp : df51a104 [9.648456] r10: r9 : r8 : 00bbff80 [9.653666] r7 : c0edcf18 r6 : df51a190 r5 : df404100 r4 : df484680 [9.660175] r3 : b020006b r2 : df40b000 r1 : df40b2bc r0 : df484680 [9.87] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel [9.674065] Control: 18c5387d Table: 1ee7004a DAC: 0015 [9.679792] Process kworker/1:0 (pid: 15, stack limit = 0xdf518210) [9.686041] Stack: (0xdf51a0f0 to 0xdf51a000) [9.690411] [] (rb_update_write_stamp) from [] (rb_commit+0x34/0xdc) [9.698478] [] (rb_commit) from [] (ring_buffer_unlock_commit+0x30/0x18c) [9.706987] [] (ring_buffer_unlock_commit) from [] (__buffer_unlock_commit+0x20/0x28) [9.716534] [] (__buffer_unlock_commit) from [] (__trace_graph_entry+0x94/0xa8) [9.725557] [] (__trace_graph_entry) from [] (trace_graph_entry+0x1b8/0x224) [9.734323] [] (trace_graph_entry) from [] (prepare_ftrace_return+0x70/0xac) [9.743089] [] (prepare_ftrace_return) from [] (ftrace_graph_caller+0x18/0x20) [9.752020] Code: e24cb004 e5903048 e3c1ceff e3cc200f (e5933018) [9.758104] ---[ end trace 5781781938261a54 ]--- [9.762689] Kernel panic - not syncing: Fatal exception [ 10.933397] SMP: failed to stop secondary CPUs [ 10.937805] ---[ end Kernel panic - not syncing: Fatal exception [ 10.943779] CPU1: stopping [ 10.946471] CPU: 1 PID: 15 Comm: kworker/1:0 Tainted: G D 4.0.0-xilinx-00043-gc701690-dirty #515 [ 10.956358] Hardware name: Xilinx Zynq Platform [ 10.960889] Workqueue: 0xe3a1 ( “å) [ 10.964896] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [ 10.972620] [] (show_stack) from [] (dump_stack+0x80/0xd0) [ 10.979822] [] (dump_stack) from [] (ipi_cpu_stop+0x4c/0x80) [ 10.987197] [] (ipi_cpu_stop) from [] (handle_IPI+0x74/0xbc) [ 10.994573] [] (handle_IPI) from [] (gic_handle_irq+0x68/0x70) [ 11.002123] [] (gic_handle_irq) from [] (__irq_svc+0x44/0x7c) [ 11.009569] Exception stack(0xdf519e70 to 0xdf519eb8) [ 11.014608] 9e60: 0034 df518000 c08d9524 [ 11.022770] 9e80: c00317c8 c00b6674 0008 df519edc [ 11.030929] 9ea0: df519eb8 df519eb8 c058f64c c058f650 6113 [ 11.037541] [] (__irq_svc) from [] (panic+0x188/0x20c) [ 11.044399] [] (panic) from [] (die+0x380/0x3e0) [ 11.050739] [] (die) from [] (__do_kernel_fault+0x74/0x94) [ 11.057941] [] (__do_kernel_fault) from [] (do_page_fault+0x2fc/0x31c) [ 11.066185] [] (do_page_fault) from [] (do_translation_fault+0x2c/0xc0)
ftrace function_graph causes system crash
Hi, all I just use ftrace to do some latency study, found that function_graph can not Work, as long as enable it, will cause kernel panic. I searched this online. Found that there are also some cause the same as mine. I am a newer of ftrace. I want to know who know what root cause? Here is some partial log: echo function_graph > current_tracer [9.583813] Unable to handle kernel paging request at virtual address b0200083 [9.590997] pgd = c0004000 [9.593683] [b0200083] *pgd= [9.597253] Internal error: Oops: 5 [#1] PREEMPT SMP ARM [9.602542] Modules linked in: [9.605586] CPU: 1 PID: 15 Comm: kworker/1:0 Not tainted 4.0.0-xilinx-00043-gc701690-dirty #515 [9.614256] Hardware name: Xilinx Zynq Platform [9.618793] Workqueue: 0xe3a1 ( “å) [9.622765] task: df517500 ti: df518000 task.ti: df518000 [9.628162] PC is at rb_update_write_stamp+0x18/0xa0 [9.633100] LR is at rb_commit+0x34/0xdc [9.637005] pc : []lr : []psr: 6093 [9.637005] sp : df51a0f0 ip : df40b00c fp : df51a104 [9.648456] r10: r9 : r8 : 00bbff80 [9.653666] r7 : c0edcf18 r6 : df51a190 r5 : df404100 r4 : df484680 [9.660175] r3 : b020006b r2 : df40b000 r1 : df40b2bc r0 : df484680 [9.87] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel [9.674065] Control: 18c5387d Table: 1ee7004a DAC: 0015 [9.679792] Process kworker/1:0 (pid: 15, stack limit = 0xdf518210) [9.686041] Stack: (0xdf51a0f0 to 0xdf51a000) [9.690411] [] (rb_update_write_stamp) from [] (rb_commit+0x34/0xdc) [9.698478] [] (rb_commit) from [] (ring_buffer_unlock_commit+0x30/0x18c) [9.706987] [] (ring_buffer_unlock_commit) from [] (__buffer_unlock_commit+0x20/0x28) [9.716534] [] (__buffer_unlock_commit) from [] (__trace_graph_entry+0x94/0xa8) [9.725557] [] (__trace_graph_entry) from [] (trace_graph_entry+0x1b8/0x224) [9.734323] [] (trace_graph_entry) from [] (prepare_ftrace_return+0x70/0xac) [9.743089] [] (prepare_ftrace_return) from [] (ftrace_graph_caller+0x18/0x20) [9.752020] Code: e24cb004 e5903048 e3c1ceff e3cc200f (e5933018) [9.758104] ---[ end trace 5781781938261a54 ]--- [9.762689] Kernel panic - not syncing: Fatal exception [ 10.933397] SMP: failed to stop secondary CPUs [ 10.937805] ---[ end Kernel panic - not syncing: Fatal exception [ 10.943779] CPU1: stopping [ 10.946471] CPU: 1 PID: 15 Comm: kworker/1:0 Tainted: G D 4.0.0-xilinx-00043-gc701690-dirty #515 [ 10.956358] Hardware name: Xilinx Zynq Platform [ 10.960889] Workqueue: 0xe3a1 ( “å) [ 10.964896] [] (unwind_backtrace) from [] (show_stack+0x20/0x24) [ 10.972620] [] (show_stack) from [] (dump_stack+0x80/0xd0) [ 10.979822] [] (dump_stack) from [] (ipi_cpu_stop+0x4c/0x80) [ 10.987197] [] (ipi_cpu_stop) from [] (handle_IPI+0x74/0xbc) [ 10.994573] [] (handle_IPI) from [] (gic_handle_irq+0x68/0x70) [ 11.002123] [] (gic_handle_irq) from [] (__irq_svc+0x44/0x7c) [ 11.009569] Exception stack(0xdf519e70 to 0xdf519eb8) [ 11.014608] 9e60: 0034 df518000 c08d9524 [ 11.022770] 9e80: c00317c8 c00b6674 0008 df519edc [ 11.030929] 9ea0: df519eb8 df519eb8 c058f64c c058f650 6113 [ 11.037541] [] (__irq_svc) from [] (panic+0x188/0x20c) [ 11.044399] [] (panic) from [] (die+0x380/0x3e0) [ 11.050739] [] (die) from [] (__do_kernel_fault+0x74/0x94) [ 11.057941] [] (__do_kernel_fault) from [] (do_page_fault+0x2fc/0x31c) [ 11.066185] [] (do_page_fault) from [] (do_translation_fault+0x2c/0xc0)