Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-26 Thread Tomasz Chmielewski
On 2016-09-26 04:07, Nikolay Borisov wrote: Not sure if that's expected behaviour or not. Why don't you sample the stacks of some of those kworker processes to see if they are all executing a parituclar piece of work. That might help you narrow down where they originate from. Cat multiple /pr

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-25 Thread Nikolay Borisov
On 25.09.2016 15:40, Tomasz Chmielewski wrote: > On 2016-09-25 18:29, Tomasz Chmielewski wrote: > >>> I'll try to bisect. >> >> OK, not a kernel regression, but some config change caused it. >> However, I'm not able to locate which change exactly. >> >> I'm attaching two configs which I've tried

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-25 Thread Mike Galbraith
On Sun, 2016-09-25 at 19:21 +0200, Mike Galbraith wrote: > On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote: > > > The problem is the allocator. > > > > -CONFIG_SLUB=y > > +CONFIG_SLAB=y > > > > > > With SLUB, I'm getting a handful of kworker processes, as expected. > > > > With SLA

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-25 Thread Mike Galbraith
On Sun, 2016-09-25 at 21:40 +0900, Tomasz Chmielewski wrote: > The problem is the allocator. > > -CONFIG_SLUB=y > +CONFIG_SLAB=y > > > With SLUB, I'm getting a handful of kworker processes, as expected. > > With SLAB, I'm getting thousands of kworker processes. > > > Not sure if that's expec

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-25 Thread Tomasz Chmielewski
On 2016-09-25 18:29, Tomasz Chmielewski wrote: I'll try to bisect. OK, not a kernel regression, but some config change caused it. However, I'm not able to locate which change exactly. I'm attaching two configs which I've tried with 4.7.3 - one results in thousands of kworkers, and the other d

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-23 Thread Tomasz Chmielewski
On 2016-09-23 23:10, Mike Galbraith wrote: I did some experiments to see when the problem first appeared. Thousands of kworker processes start to show up in 4.7.0-rc5. kernel version | kworker count after boot --- 4.6.3 > > 37 4.6.4 > >

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-23 Thread Mike Galbraith
On Fri, 2016-09-23 at 22:23 +0900, Tomasz Chmielewski wrote: > On 2016-09-19 16:08, Tomasz Chmielewski wrote: > > On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing > > thousands of kworker processes. > > # ps auxf|grep -c kworker > > 2104 > > Load average goes into hundreds on a pre

Re: thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-23 Thread Tomasz Chmielewski
On 2016-09-19 16:08, Tomasz Chmielewski wrote: On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes. # ps auxf|grep -c kworker 2104 Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one us

thousands of kworker processes with 4.7.x and 4.8-rc*

2016-09-19 Thread Tomasz Chmielewski
On several servers running 4.7.x and 4.8-rc6/7 kernels I'm seeing thousands of kworker processes. # ps auxf|grep -c kworker 2104 Load average goes into hundreds on a pretty much idle server (biggest CPU and RAM consumers are probably SSHD with one user logged in and rsyslog writing ~1 line p