Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 12/02/16 10:14, Donald Buczek wrote: On 11/30/16 12:43, Donald Buczek wrote: On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 12/02/16 10:14, Donald Buczek wrote: On 11/30/16 12:43, Donald Buczek wrote: On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 06:02:49PM +0100, Michal Hocko wrote: > On Fri 02-12-16 08:44:08, Paul E. McKenney wrote: > > On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > > > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > > > > > >> Well, I can confirm, that replacing

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 06:02:49PM +0100, Michal Hocko wrote: > On Fri 02-12-16 08:44:08, Paul E. McKenney wrote: > > On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > > > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > > > > > >> Well, I can confirm, that replacing

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri 02-12-16 08:44:08, Paul E. McKenney wrote: > On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > > > >> Well, I can confirm, that replacing cond_resched_rcu_qs in > > >> shrink_node_memcg by cond_resched also makes dmesg

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri 02-12-16 08:44:08, Paul E. McKenney wrote: > On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > > > >> Well, I can confirm, that replacing cond_resched_rcu_qs in > > >> shrink_node_memcg by cond_resched also makes dmesg

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > >> Well, I can confirm, that replacing cond_resched_rcu_qs in > >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU > >> CPU stall warnings. > >> > >> I've

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 07:39:24PM +0300, Boris Zhmurov wrote: > Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: > > >> Well, I can confirm, that replacing cond_resched_rcu_qs in > >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU > >> CPU stall warnings. > >> > >> I've

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: >> Well, I can confirm, that replacing cond_resched_rcu_qs in >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU >> CPU stall warnings. >> >> I've attached patch (just modification of Paul's patch), that >> fixes RCU stall

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Paul E. McKenney Thu Dec 01 2016 - 14:39:21 EST: >> Well, I can confirm, that replacing cond_resched_rcu_qs in >> shrink_node_memcg by cond_resched also makes dmesg clean from RCU >> CPU stall warnings. >> >> I've attached patch (just modification of Paul's patch), that >> fixes RCU stall

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 10:37:35AM +0100, Michal Hocko wrote: > On Thu 01-12-16 21:10:01, Boris Zhmurov wrote: > > Michal Hocko 30/11/16 21:25: > > > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > > >> > > >> I didn't try that. I've tried 4 patches from Paul's

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Fri, Dec 02, 2016 at 10:37:35AM +0100, Michal Hocko wrote: > On Thu 01-12-16 21:10:01, Boris Zhmurov wrote: > > Michal Hocko 30/11/16 21:25: > > > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > > >> > > >> I didn't try that. I've tried 4 patches from Paul's

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu 01-12-16 21:10:01, Boris Zhmurov wrote: > Michal Hocko 30/11/16 21:25: > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > >> > >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. > >> I can try another portion of patches, no problem :) > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu 01-12-16 21:10:01, Boris Zhmurov wrote: > Michal Hocko 30/11/16 21:25: > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > >> > >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. > >> I can try another portion of patches, no problem :) > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:43, Donald Buczek wrote: On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a chance to run. Just as an experiment,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:43, Donald Buczek wrote: On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a chance to run. Just as an experiment,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 09:10:01PM +0300, Boris Zhmurov wrote: > Michal Hocko 30/11/16 21:25: > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > >> > >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. > >> I can try another portion of patches, no

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 09:10:01PM +0300, Boris Zhmurov wrote: > Michal Hocko 30/11/16 21:25: > > >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? > >> > >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. > >> I can try another portion of patches, no

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 10:42:52AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 07:09:53PM +0100, Peter Zijlstra wrote: > > Thing is, I'm slightly uncomfortable with de-coupling rcu-sched from > > actual schedule() calls. > > OK, what is the source of your discomfort? Good question;

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 10:42:52AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 07:09:53PM +0100, Peter Zijlstra wrote: > > Thing is, I'm slightly uncomfortable with de-coupling rcu-sched from > > actual schedule() calls. > > OK, what is the source of your discomfort? Good question;

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 07:09:53PM +0100, Peter Zijlstra wrote: > On Thu, Dec 01, 2016 at 08:59:18AM -0800, Paul E. McKenney wrote: > > On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > > > Well, with the above change cond_resched() is already sufficient, no? > > > > Maybe. Right

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 07:09:53PM +0100, Peter Zijlstra wrote: > On Thu, Dec 01, 2016 at 08:59:18AM -0800, Paul E. McKenney wrote: > > On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > > > Well, with the above change cond_resched() is already sufficient, no? > > > > Maybe. Right

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Michal Hocko 30/11/16 21:25: >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? >> >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. >> I can try another portion of patches, no problem :) > > Replacing cond_resched_rcu_qs in shrink_node_memcg by

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Michal Hocko 30/11/16 21:25: >>> Do I get it right that s@cond_resched_rcu_qs@cond_resched@ didn't help? >> >> I didn't try that. I've tried 4 patches from Paul's linux-rcu tree. >> I can try another portion of patches, no problem :) > > Replacing cond_resched_rcu_qs in shrink_node_memcg by

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 08:59:18AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > > Well, with the above change cond_resched() is already sufficient, no? > > Maybe. Right now, cond_resched_rcu_qs() gets a quiescent state to > the RCU core in

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 08:59:18AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > > Well, with the above change cond_resched() is already sufficient, no? > > Maybe. Right now, cond_resched_rcu_qs() gets a quiescent state to > the RCU core in

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > On Thu, Dec 01, 2016 at 04:40:24AM -0800, Paul E. McKenney wrote: > > On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > > > > Sure, we all dislike IPIs, but I'm thinking this half-way point is > > > sensible, no

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 05:36:14PM +0100, Peter Zijlstra wrote: > On Thu, Dec 01, 2016 at 04:40:24AM -0800, Paul E. McKenney wrote: > > On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > > > > Sure, we all dislike IPIs, but I'm thinking this half-way point is > > > sensible, no

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 04:40:24AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > > Sure, we all dislike IPIs, but I'm thinking this half-way point is > > sensible, no point in issuing user visible annoyance if indeed we can > > prod things back

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 04:40:24AM -0800, Paul E. McKenney wrote: > On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > > Sure, we all dislike IPIs, but I'm thinking this half-way point is > > sensible, no point in issuing user visible annoyance if indeed we can > > prod things back

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 11:40:19AM -0800, Paul E. McKenney wrote: > > > > See commit: > > > > > > 4a81e8328d37 ("rcu: Reduce overhead of cond_resched() checks for RCU") > > > > > > Someone actually wrote down what the problem

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Thu, Dec 01, 2016 at 06:30:35AM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 11:40:19AM -0800, Paul E. McKenney wrote: > > > > See commit: > > > > > > 4a81e8328d37 ("rcu: Reduce overhead of cond_resched() checks for RCU") > > > > > > Someone actually wrote down what the problem

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 11:40:19AM -0800, Paul E. McKenney wrote: > > See commit: > > > > 4a81e8328d37 ("rcu: Reduce overhead of cond_resched() checks for RCU") > > > > Someone actually wrote down what the problem was. > > Don't worry, it won't happen again. ;-) > > OK, so the regressions

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 11:40:19AM -0800, Paul E. McKenney wrote: > > See commit: > > > > 4a81e8328d37 ("rcu: Reduce overhead of cond_resched() checks for RCU") > > > > Someone actually wrote down what the problem was. > > Don't worry, it won't happen again. ;-) > > OK, so the regressions

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:50:16PM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > > We can, and you are correct that

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:50:16PM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > > We can, and you are correct that

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > We can, and you are correct that cond_resched() does not unconditionally > > > supply RCU quiescent states,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > We can, and you are correct that cond_resched() does not unconditionally > > > supply RCU quiescent states,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 09:23:55, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > > We can, and you are correct that cond_resched() does

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 09:23:55, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > > We can, and you are correct that cond_resched() does

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > We can, and you are correct that cond_resched() does not unconditionally > > > supply RCU quiescent states,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:05:57PM +0100, Michal Hocko wrote: > On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > > We can, and you are correct that cond_resched() does not unconditionally > > > supply RCU quiescent states,

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > We can, and you are correct that cond_resched() does not unconditionally > > supply RCU quiescent states, and never has. Last time I tried to add > > cond_resched_rcu_qs()

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 17:38:20, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > We can, and you are correct that cond_resched() does not unconditionally > > supply RCU quiescent states, and never has. Last time I tried to add > > cond_resched_rcu_qs()

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 05:38:20PM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > We can, and you are correct that cond_resched() does not unconditionally > > supply RCU quiescent states, and never has. Last time I tried to add > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 05:38:20PM +0100, Peter Zijlstra wrote: > On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > > We can, and you are correct that cond_resched() does not unconditionally > > supply RCU quiescent states, and never has. Last time I tried to add > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > We can, and you are correct that cond_resched() does not unconditionally > supply RCU quiescent states, and never has. Last time I tried to add > cond_resched_rcu_qs() semantics to cond_resched(), I got told "no", > but perhaps

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 06:29:55AM -0800, Paul E. McKenney wrote: > We can, and you are correct that cond_resched() does not unconditionally > supply RCU quiescent states, and never has. Last time I tried to add > cond_resched_rcu_qs() semantics to cond_resched(), I got told "no", > but perhaps

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 01:31:37PM +0100, Paul Menzel wrote: > On 11/30/16 12:54, Paul E. McKenney wrote: > > On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: > >> On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > >>> [CCing Paul] > >>> > >>> On Wed 30-11-16

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 01:31:37PM +0100, Paul Menzel wrote: > On 11/30/16 12:54, Paul E. McKenney wrote: > > On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: > >> On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > >>> [CCing Paul] > >>> > >>> On Wed 30-11-16

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 02:19:10PM +0100, Michal Hocko wrote: > On Wed 30-11-16 03:53:20, Paul E. McKenney wrote: > > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > > [CCing Paul] > > > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > > [...] > > > > shrink_active_list

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 02:19:10PM +0100, Michal Hocko wrote: > On Wed 30-11-16 03:53:20, Paul E. McKenney wrote: > > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > > [CCing Paul] > > > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > > [...] > > > > shrink_active_list

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 03:53:20, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > [CCing Paul] > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > [...] > > > shrink_active_list gets and releases the spinlock and calls > > > cond_resched(). > > > This

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 30-11-16 03:53:20, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > [CCing Paul] > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > [...] > > > shrink_active_list gets and releases the spinlock and calls > > > cond_resched(). > > > This

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:54, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: >> On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: >>> [CCing Paul] >>> >>> On Wed 30-11-16 11:28:34, Donald Buczek wrote: >>> [...] shrink_active_list gets and

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:54, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: >> On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: >>> [CCing Paul] >>> >>> On Wed 30-11-16 11:28:34, Donald Buczek wrote: >>> [...] shrink_active_list gets and

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > [CCing Paul] > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > [...] > > > shrink_active_list gets and releases the spinlock and calls > > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 03:53:20AM -0800, Paul E. McKenney wrote: > On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > > [CCing Paul] > > > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > > [...] > > > shrink_active_list gets and releases the spinlock and calls > > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > [CCing Paul] > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > [...] > > shrink_active_list gets and releases the spinlock and calls cond_resched(). > > This should give other tasks a chance to run. Just as an experiment, I'm > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 30, 2016 at 12:09:44PM +0100, Michal Hocko wrote: > [CCing Paul] > > On Wed 30-11-16 11:28:34, Donald Buczek wrote: > [...] > > shrink_active_list gets and releases the spinlock and calls cond_resched(). > > This should give other tasks a chance to run. Just as an experiment, I'm > >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a chance to run. Just as an experiment, I'm trying --- a/mm/vmscan.c +++

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/30/16 12:09, Michal Hocko wrote: [CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] shrink_active_list gets and releases the spinlock and calls cond_resched(). This should give other tasks a chance to run. Just as an experiment, I'm trying --- a/mm/vmscan.c +++

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

[CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] > shrink_active_list gets and releases the spinlock and calls cond_resched(). > This should give other tasks a chance to run. Just as an experiment, I'm > trying > > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1921,7 +1921,7 @@

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

[CCing Paul] On Wed 30-11-16 11:28:34, Donald Buczek wrote: [...] > shrink_active_list gets and releases the spinlock and calls cond_resched(). > This should give other tasks a chance to run. Just as an experiment, I'm > trying > > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1921,7 +1921,7 @@

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/28/16 13:26, Paul Menzel wrote: [...] On 11/28/16 12:04, Michal Hocko wrote: [...] OK, so one of the stall is reported at [118077.988410] INFO: rcu_sched detected stalls on CPUs/tasks: [118077.988416] 1-...: (181 ticks this GP) idle=6d5/140/0 softirq=46417663/46417663

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/28/16 13:26, Paul Menzel wrote: [...] On 11/28/16 12:04, Michal Hocko wrote: [...] OK, so one of the stall is reported at [118077.988410] INFO: rcu_sched detected stalls on CPUs/tasks: [118077.988416] 1-...: (181 ticks this GP) idle=6d5/140/0 softirq=46417663/46417663

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

+linux...@kvack.org -linux-...@vger.kernel.org Dear Michal, Thank you for your reply, and for looking at the log files. On 11/28/16 12:04, Michal Hocko wrote: On Sun 27-11-16 10:19:06, Donald Buczek wrote: On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

+linux...@kvack.org -linux-...@vger.kernel.org Dear Michal, Thank you for your reply, and for looking at the log files. On 11/28/16 12:04, Michal Hocko wrote: On Sun 27-11-16 10:19:06, Donald Buczek wrote: On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Sun 27-11-16 10:19:06, Donald Buczek wrote: > On 24.11.2016 11:15, Michal Hocko wrote: > > On Mon 21-11-16 16:35:53, Donald Buczek wrote: > > [...] > > > Hello, > > > > > > thanks a lot for looking into this! > > > > > > Let me add some information from the reporting site: > > > > > > *

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Sun 27-11-16 10:19:06, Donald Buczek wrote: > On 24.11.2016 11:15, Michal Hocko wrote: > > On Mon 21-11-16 16:35:53, Donald Buczek wrote: > > [...] > > > Hello, > > > > > > thanks a lot for looking into this! > > > > > > Let me add some information from the reporting site: > > > > > > *

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Am Donnerstag, den 24.11.2016, 19:50 +0100 schrieb Donald Buczek: > On 24.11.2016 11:15, Michal Hocko wrote: > > > On Mon 21-11-16 16:35:53, Donald Buczek wrote: > > [...] > >> Let me add some information from the reporting site: > >> > >> * We've tried the patch from Paul E. McKenney (the one

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

Am Donnerstag, den 24.11.2016, 19:50 +0100 schrieb Donald Buczek: > On 24.11.2016 11:15, Michal Hocko wrote: > > > On Mon 21-11-16 16:35:53, Donald Buczek wrote: > > [...] > >> Let me add some information from the reporting site: > >> > >> * We've tried the patch from Paul E. McKenney (the one

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] Hello, thanks a lot for looking into this! Let me add some information from the reporting site: * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov 2016) and it doesn't shut

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] Hello, thanks a lot for looking into this! Let me add some information from the reporting site: * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov 2016) and it doesn't shut

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

> On Nov 24, 2016, at 5:15 AM, Michal Hocko wrote: > >> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 , >> 4.8.8 and now 4.9.0-rc5+Pauls patch > > I assume you haven't tried the Linus 4.8 kernel without any further > stable patches? Just to be sure

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

> On Nov 24, 2016, at 5:15 AM, Michal Hocko wrote: > >> * No rcu_* warnings on that machine with 4.7.2, but with 4.8.4 , 4.8.6 , >> 4.8.8 and now 4.9.0-rc5+Pauls patch > > I assume you haven't tried the Linus 4.8 kernel without any further > stable patches? Just to be sure we are not talking

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] Hello, thanks a lot for looking into this! Let me add some information from the reporting site: * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov 2016) and it doesn't shut

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 24.11.2016 11:15, Michal Hocko wrote: On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] Hello, thanks a lot for looking into this! Let me add some information from the reporting site: * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov 2016) and it doesn't shut

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] > Hello, > > thanks a lot for looking into this! > > Let me add some information from the reporting site: > > * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov > 2016) and it doesn't shut up the rcu stall warnings. >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon 21-11-16 16:35:53, Donald Buczek wrote: [...] > Hello, > > thanks a lot for looking into this! > > Let me add some information from the reporting site: > > * We've tried the patch from Paul E. McKenney (the one posted Wed, 16 Nov > 2016) and it doesn't shut up the rcu stall warnings. >

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/21/16 15:29, Paul E. McKenney wrote: On Mon, Nov 21, 2016 at 03:18:19PM +0100, Michal Hocko wrote: On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: [...] To the patch. I cannot say I would like it. cond_resched_rcu_qs

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On 11/21/16 15:29, Paul E. McKenney wrote: On Mon, Nov 21, 2016 at 03:18:19PM +0100, Michal Hocko wrote: On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: [...] To the patch. I cannot say I would like it. cond_resched_rcu_qs

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon, Nov 21, 2016 at 03:18:19PM +0100, Michal Hocko wrote: > On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: > > On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: > [...] > > > To the patch. I cannot say I would like it. cond_resched_rcu_qs sounds > > > way too lowlevel for this

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon, Nov 21, 2016 at 03:18:19PM +0100, Michal Hocko wrote: > On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: > > On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: > [...] > > > To the patch. I cannot say I would like it. cond_resched_rcu_qs sounds > > > way too lowlevel for this

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: > On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: [...] > > To the patch. I cannot say I would like it. cond_resched_rcu_qs sounds > > way too lowlevel for this usage. If anything cond_resched somewhere inside > > mem_cgroup_iter

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon 21-11-16 06:01:22, Paul E. McKenney wrote: > On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: [...] > > To the patch. I cannot say I would like it. cond_resched_rcu_qs sounds > > way too lowlevel for this usage. If anything cond_resched somewhere inside > > mem_cgroup_iter

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: > On Wed 16-11-16 09:30:36, Paul E. McKenney wrote: > > On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > > > Dear Linux folks, > > > > > > > > > On 11/08/16 19:39, Paul E. McKenney wrote: > > > >On Tue, Nov 08, 2016 at

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Mon, Nov 21, 2016 at 02:41:31PM +0100, Michal Hocko wrote: > On Wed 16-11-16 09:30:36, Paul E. McKenney wrote: > > On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > > > Dear Linux folks, > > > > > > > > > On 11/08/16 19:39, Paul E. McKenney wrote: > > > >On Tue, Nov 08, 2016 at

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 16-11-16 09:30:36, Paul E. McKenney wrote: > On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > > Dear Linux folks, > > > > > > On 11/08/16 19:39, Paul E. McKenney wrote: > > >On Tue, Nov 08, 2016 at 06:38:18PM +0100, Paul Menzel wrote: > > >>On 11/08/16 18:03, Paul E.

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed 16-11-16 09:30:36, Paul E. McKenney wrote: > On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > > Dear Linux folks, > > > > > > On 11/08/16 19:39, Paul E. McKenney wrote: > > >On Tue, Nov 08, 2016 at 06:38:18PM +0100, Paul Menzel wrote: > > >>On 11/08/16 18:03, Paul E.

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > Dear Linux folks, > > > On 11/08/16 19:39, Paul E. McKenney wrote: > >On Tue, Nov 08, 2016 at 06:38:18PM +0100, Paul Menzel wrote: > >>On 11/08/16 18:03, Paul E. McKenney wrote: > >>>On Tue, Nov 08, 2016 at 01:22:28PM +0100, Paul

Re: INFO: rcu_sched detected stalls on CPUs/tasks with `kswapd` and `mem_cgroup_shrink_node`

On Wed, Nov 16, 2016 at 06:01:19PM +0100, Paul Menzel wrote: > Dear Linux folks, > > > On 11/08/16 19:39, Paul E. McKenney wrote: > >On Tue, Nov 08, 2016 at 06:38:18PM +0100, Paul Menzel wrote: > >>On 11/08/16 18:03, Paul E. McKenney wrote: > >>>On Tue, Nov 08, 2016 at 01:22:28PM +0100, Paul