Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 10:58:05AM -0700, Paul E. McKenney wrote: > On Wed, Aug 16, 2017 at 12:41:40PM -0400, Steven Rostedt wrote: > > On Wed, 16 Aug 2017 09:32:28 -0700 > > "Paul E. McKenney" wrote: > > > > > Let me see if I understand you... About halfway to the

Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 10:58:05AM -0700, Paul E. McKenney wrote: > On Wed, Aug 16, 2017 at 12:41:40PM -0400, Steven Rostedt wrote: > > On Wed, 16 Aug 2017 09:32:28 -0700 > > "Paul E. McKenney" wrote: > > > > > Let me see if I understand you... About halfway to the stall limit, > > > RCU

Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 12:41:40PM -0400, Steven Rostedt wrote: > On Wed, 16 Aug 2017 09:32:28 -0700 > "Paul E. McKenney" wrote: > > > Let me see if I understand you... About halfway to the stall limit, > > RCU triggers an irq_work (on each CPU that has not yet

Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 12:41:40PM -0400, Steven Rostedt wrote: > On Wed, 16 Aug 2017 09:32:28 -0700 > "Paul E. McKenney" wrote: > > > Let me see if I understand you... About halfway to the stall limit, > > RCU triggers an irq_work (on each CPU that has not yet passed through > > a quiescent

Re: RCU stall when using function_graph

On Wed, 16 Aug 2017 09:32:28 -0700 "Paul E. McKenney" wrote: > Let me see if I understand you... About halfway to the stall limit, > RCU triggers an irq_work (on each CPU that has not yet passed through > a quiescent state, IPIing them in turn?), and if the irq_work

Re: RCU stall when using function_graph

On Wed, 16 Aug 2017 09:32:28 -0700 "Paul E. McKenney" wrote: > Let me see if I understand you... About halfway to the stall limit, > RCU triggers an irq_work (on each CPU that has not yet passed through > a quiescent state, IPIing them in turn?), and if the irq_work has > not completed by the

Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 10:04:21AM -0400, Steven Rostedt wrote: > On Wed, 16 Aug 2017 10:42:15 +0200 > Daniel Lezcano wrote: > > > Hi Steven, > > > > > > On 15/08/2017 15:29, Steven Rostedt wrote: > > > > > > [ I'm back from vacation! ] > > > > Did you get the

Re: RCU stall when using function_graph

On Wed, Aug 16, 2017 at 10:04:21AM -0400, Steven Rostedt wrote: > On Wed, 16 Aug 2017 10:42:15 +0200 > Daniel Lezcano wrote: > > > Hi Steven, > > > > > > On 15/08/2017 15:29, Steven Rostedt wrote: > > > > > > [ I'm back from vacation! ] > > > > Did you get the tapes? :) > > Yes, but

Re: RCU stall when using function_graph

On Wed, 16 Aug 2017 10:42:15 +0200 Daniel Lezcano wrote: > Hi Steven, > > > On 15/08/2017 15:29, Steven Rostedt wrote: > > > > [ I'm back from vacation! ] > > Did you get the tapes? :) Yes, but nothing in them would cause the reputation of the POTUS to become

Re: RCU stall when using function_graph

On Wed, 16 Aug 2017 10:42:15 +0200 Daniel Lezcano wrote: > Hi Steven, > > > On 15/08/2017 15:29, Steven Rostedt wrote: > > > > [ I'm back from vacation! ] > > Did you get the tapes? :) Yes, but nothing in them would cause the reputation of the POTUS to become any worse than it already is.

Re: RCU stall when using function_graph

Hi Steven, On 15/08/2017 15:29, Steven Rostedt wrote: > > [ I'm back from vacation! ] Did you get the tapes? :) > On Wed, 9 Aug 2017 17:51:33 +0200 > Daniel Lezcano wrote: > >> Well, may be the instruction pointer thing is not a good idea. >> >> I learnt from

Re: RCU stall when using function_graph

Hi Steven, On 15/08/2017 15:29, Steven Rostedt wrote: > > [ I'm back from vacation! ] Did you get the tapes? :) > On Wed, 9 Aug 2017 17:51:33 +0200 > Daniel Lezcano wrote: > >> Well, may be the instruction pointer thing is not a good idea. >> >> I learnt from this experience, an overloaded

Re: RCU stall when using function_graph

[ I'm back from vacation! ] On Wed, 9 Aug 2017 17:51:33 +0200 Daniel Lezcano wrote: > Well, may be the instruction pointer thing is not a good idea. > > I learnt from this experience, an overloaded kernel with a lot of > interrupts can hang the console and issue RCU

Re: RCU stall when using function_graph

[ I'm back from vacation! ] On Wed, 9 Aug 2017 17:51:33 +0200 Daniel Lezcano wrote: > Well, may be the instruction pointer thing is not a good idea. > > I learnt from this experience, an overloaded kernel with a lot of > interrupts can hang the console and issue RCU stall. > > However,

Re: RCU stall when using function_graph

On 10/08/2017 23:39, Paul E. McKenney wrote: > On Thu, Aug 10, 2017 at 11:45:09AM +0200, Daniel Lezcano wrote: [ ... ] >> Nothing coming in mind but may be worth to mention the slowness of the >> CPU is the aggravating factor. In particular I was able to reproduce the >> issue by setting to the

Re: RCU stall when using function_graph

On 10/08/2017 23:39, Paul E. McKenney wrote: > On Thu, Aug 10, 2017 at 11:45:09AM +0200, Daniel Lezcano wrote: [ ... ] >> Nothing coming in mind but may be worth to mention the slowness of the >> CPU is the aggravating factor. In particular I was able to reproduce the >> issue by setting to the

Re: RCU stall when using function_graph

On Thu, Aug 10, 2017 at 11:45:09AM +0200, Daniel Lezcano wrote: > On 09/08/2017 19:22, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: > >> On 09/08/2017 16:40, Paul E. McKenney wrote: > >>> On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote:

Re: RCU stall when using function_graph

On Thu, Aug 10, 2017 at 11:45:09AM +0200, Daniel Lezcano wrote: > On 09/08/2017 19:22, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: > >> On 09/08/2017 16:40, Paul E. McKenney wrote: > >>> On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote:

Re: RCU stall when using function_graph

On 09/08/2017 19:22, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: >> On 09/08/2017 16:40, Paul E. McKenney wrote: >>> On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: On 09/08/2017 14:58, Paul E. McKenney wrote: > On Wed, Aug 09,

Re: RCU stall when using function_graph

On 09/08/2017 19:22, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: >> On 09/08/2017 16:40, Paul E. McKenney wrote: >>> On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: On 09/08/2017 14:58, Paul E. McKenney wrote: > On Wed, Aug 09,

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: > On 09/08/2017 16:40, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: > >> On 09/08/2017 14:58, Paul E. McKenney wrote: > >>> On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote:

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 05:51:33PM +0200, Daniel Lezcano wrote: > On 09/08/2017 16:40, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: > >> On 09/08/2017 14:58, Paul E. McKenney wrote: > >>> On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote:

Re: RCU stall when using function_graph

On 09/08/2017 16:40, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: >> On 09/08/2017 14:58, Paul E. McKenney wrote: >>> On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: On Sunday 06 August 2017 10:32 PM, Paul E. McKenney

Re: RCU stall when using function_graph

On 09/08/2017 16:40, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: >> On 09/08/2017 14:58, Paul E. McKenney wrote: >>> On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: On Sunday 06 August 2017 10:32 PM, Paul E. McKenney

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: > On 09/08/2017 14:58, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: > >> > >> > >> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: > >>> On Sat, Aug 05, 2017 at 02:24:21PM

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 03:28:05PM +0200, Daniel Lezcano wrote: > On 09/08/2017 14:58, Paul E. McKenney wrote: > > On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: > >> > >> > >> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: > >>> On Sat, Aug 05, 2017 at 02:24:21PM

Re: RCU stall when using function_graph

On 09/08/2017 14:58, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: >> >> >> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: >>> On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: Dear All As for me, after configuring

Re: RCU stall when using function_graph

On 09/08/2017 14:58, Paul E. McKenney wrote: > On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: >> >> >> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: >>> On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: Dear All As for me, after configuring

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: > > > On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: > >On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: > >>Dear All > >> > >>As for me, after configuring function_graph as below, crash disappears. > >>"echo 0 >

Re: RCU stall when using function_graph

On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote: > > > On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: > >On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: > >>Dear All > >> > >>As for me, after configuring function_graph as below, crash disappears. > >>"echo 0 >

Re: RCU stall when using function_graph

On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: Dear All As for me, after configuring function_graph as below, crash disappears. "echo 0 > d/tracing/tracing_on" "sleep 1" "echo function_graph > d/tracing/current_tracer" "sleep

Re: RCU stall when using function_graph

On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote: On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: Dear All As for me, after configuring function_graph as below, crash disappears. "echo 0 > d/tracing/tracing_on" "sleep 1" "echo function_graph > d/tracing/current_tracer" "sleep

Re: RCU stall when using function_graph

On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: > Dear All > > As for me, after configuring function_graph as below, crash disappears. > "echo 0 > d/tracing/tracing_on" > "sleep 1" > > "echo function_graph > d/tracing/current_tracer" > "sleep 1" > > "echo smp_call_function_single >

Re: RCU stall when using function_graph

On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote: > Dear All > > As for me, after configuring function_graph as below, crash disappears. > "echo 0 > d/tracing/tracing_on" > "sleep 1" > > "echo function_graph > d/tracing/current_tracer" > "sleep 1" > > "echo smp_call_function_single >

Re: RCU stall when using function_graph

On Thu, Aug 03, 2017 at 05:44:21AM -0700, Paul E. McKenney wrote: [ ... ] > > > BTW, function_graph tracer is the most invasive of the tracers. It's 4x > > > slower than function tracer. I'm wondering if the tracer isn't the > > > cause, but just slows things down enough to cause a some other

Re: RCU stall when using function_graph

On Thu, Aug 03, 2017 at 05:44:21AM -0700, Paul E. McKenney wrote: [ ... ] > > > BTW, function_graph tracer is the most invasive of the tracers. It's 4x > > > slower than function tracer. I'm wondering if the tracer isn't the > > > cause, but just slows things down enough to cause a some other

Re: RCU stall when using function_graph

On Thu, Aug 03, 2017 at 01:41:11PM +0200, Daniel Lezcano wrote: > On 02/08/2017 15:07, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 14:42:39 +0200 > > Daniel Lezcano wrote: > > > >> On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > >>> On Wed, 2 Aug

Re: RCU stall when using function_graph

On Thu, Aug 03, 2017 at 01:41:11PM +0200, Daniel Lezcano wrote: > On 02/08/2017 15:07, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 14:42:39 +0200 > > Daniel Lezcano wrote: > > > >> On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > >>> On Wed, 2 Aug 2017 00:15:44 +0200 > >>>

Re: RCU stall when using function_graph

On 02/08/2017 15:07, Steven Rostedt wrote: > On Wed, 2 Aug 2017 14:42:39 +0200 > Daniel Lezcano wrote: > >> On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: >>> On Wed, 2 Aug 2017 00:15:44 +0200 >>> Daniel Lezcano wrote: >>>

Re: RCU stall when using function_graph

On 02/08/2017 15:07, Steven Rostedt wrote: > On Wed, 2 Aug 2017 14:42:39 +0200 > Daniel Lezcano wrote: > >> On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: >>> On Wed, 2 Aug 2017 00:15:44 +0200 >>> Daniel Lezcano wrote: >>> On 02/08/2017 00:04, Paul E. McKenney wrote:

Re: RCU stall when using function_graph

On Wed, Aug 02, 2017 at 09:07:44AM -0400, Steven Rostedt wrote: > On Wed, 2 Aug 2017 14:42:39 +0200 > Daniel Lezcano wrote: > > > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > > On Wed, 2 Aug 2017 00:15:44 +0200 > > > Daniel Lezcano

Re: RCU stall when using function_graph

On Wed, Aug 02, 2017 at 09:07:44AM -0400, Steven Rostedt wrote: > On Wed, 2 Aug 2017 14:42:39 +0200 > Daniel Lezcano wrote: > > > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > > On Wed, 2 Aug 2017 00:15:44 +0200 > > > Daniel Lezcano wrote: > > > > > > > On 02/08/2017

Re: RCU stall when using function_graph

On Wed, Aug 02, 2017 at 02:42:39PM +0200, Daniel Lezcano wrote: > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 00:15:44 +0200 > > Daniel Lezcano wrote: > > > > > On 02/08/2017 00:04, Paul E. McKenney wrote: > > > >> Hi Paul, >

Re: RCU stall when using function_graph

On Wed, Aug 02, 2017 at 02:42:39PM +0200, Daniel Lezcano wrote: > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 00:15:44 +0200 > > Daniel Lezcano wrote: > > > > > On 02/08/2017 00:04, Paul E. McKenney wrote: > > > >> Hi Paul, > > > >> > > > >> I have been

Re: RCU stall when using function_graph

On Wed, 2 Aug 2017 14:42:39 +0200 Daniel Lezcano wrote: > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 00:15:44 +0200 > > Daniel Lezcano wrote: > > > > > On 02/08/2017 00:04, Paul E. McKenney

Re: RCU stall when using function_graph

On Wed, 2 Aug 2017 14:42:39 +0200 Daniel Lezcano wrote: > On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > > On Wed, 2 Aug 2017 00:15:44 +0200 > > Daniel Lezcano wrote: > > > > > On 02/08/2017 00:04, Paul E. McKenney wrote: > > > >> Hi Paul, > > > >> > > > >> I have been

Re: RCU stall when using function_graph

On Tue, Aug 01, 2017 at 03:04:05PM -0700, Paul E. McKenney wrote: > > Hi Paul, > > > > I have been trying to set the function_graph tracer for ftrace and each > > time I > > get a CPU stall. > > > > How to reproduce: > > - > > > > echo function_graph >

Re: RCU stall when using function_graph

On Tue, Aug 01, 2017 at 03:04:05PM -0700, Paul E. McKenney wrote: > > Hi Paul, > > > > I have been trying to set the function_graph tracer for ftrace and each > > time I > > get a CPU stall. > > > > How to reproduce: > > - > > > > echo function_graph >

Re: RCU stall when using function_graph

On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > On Wed, 2 Aug 2017 00:15:44 +0200 > Daniel Lezcano wrote: > > > On 02/08/2017 00:04, Paul E. McKenney wrote: > > >> Hi Paul, > > >> > > >> I have been trying to set the function_graph tracer for ftrace

Re: RCU stall when using function_graph

On Tue, Aug 01, 2017 at 08:12:14PM -0400, Steven Rostedt wrote: > On Wed, 2 Aug 2017 00:15:44 +0200 > Daniel Lezcano wrote: > > > On 02/08/2017 00:04, Paul E. McKenney wrote: > > >> Hi Paul, > > >> > > >> I have been trying to set the function_graph tracer for ftrace and each > > >> time I > >

Re: RCU stall when using function_graph

On Wed, 2 Aug 2017 00:15:44 +0200 Daniel Lezcano wrote: > On 02/08/2017 00:04, Paul E. McKenney wrote: > >> Hi Paul, > >> > >> I have been trying to set the function_graph tracer for ftrace and each > >> time I > >> get a CPU stall. > >> > >> How to reproduce: > >>

Re: RCU stall when using function_graph

On Wed, 2 Aug 2017 00:15:44 +0200 Daniel Lezcano wrote: > On 02/08/2017 00:04, Paul E. McKenney wrote: > >> Hi Paul, > >> > >> I have been trying to set the function_graph tracer for ftrace and each > >> time I > >> get a CPU stall. > >> > >> How to reproduce: > >> - > >> > >>

Re: RCU stall when using function_graph

On 02/08/2017 00:04, Paul E. McKenney wrote: >> Hi Paul, >> >> I have been trying to set the function_graph tracer for ftrace and each time >> I >> get a CPU stall. >> >> How to reproduce: >> - >> >> echo function_graph > /sys/kernel/debug/tracing/current_tracer >>

Re: RCU stall when using function_graph

On 02/08/2017 00:04, Paul E. McKenney wrote: >> Hi Paul, >> >> I have been trying to set the function_graph tracer for ftrace and each time >> I >> get a CPU stall. >> >> How to reproduce: >> - >> >> echo function_graph > /sys/kernel/debug/tracing/current_tracer >>

Re: RCU stall when using function_graph

> Hi Paul, > > I have been trying to set the function_graph tracer for ftrace and each time I > get a CPU stall. > > How to reproduce: > - > >echo function_graph > /sys/kernel/debug/tracing/current_tracer > > This error appears with v4.13-rc3 and v4.12-rc6. > >

Re: RCU stall when using function_graph

> Hi Paul, > > I have been trying to set the function_graph tracer for ftrace and each time I > get a CPU stall. > > How to reproduce: > - > >echo function_graph > /sys/kernel/debug/tracing/current_tracer > > This error appears with v4.13-rc3 and v4.12-rc6. > >

RCU stall when using function_graph

Hi Paul, I have been trying to set the function_graph tracer for ftrace and each time I get a CPU stall. How to reproduce: - echo function_graph > /sys/kernel/debug/tracing/current_tracer This error appears with v4.13-rc3 and v4.12-rc6. Is it something already reported

RCU stall when using function_graph

Hi Paul, I have been trying to set the function_graph tracer for ftrace and each time I get a CPU stall. How to reproduce: - echo function_graph > /sys/kernel/debug/tracing/current_tracer This error appears with v4.13-rc3 and v4.12-rc6. Is it something already reported