RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Friday, April 20, 2018 03:22 > On Thu 19-04-18 21:37:25, Dexuan Cui wrote: > > > From: Jan Kara > > > Sent: Thursday, April 19, 2018 13:23 > > > Good news guys, Robert has just spotted a bug which looks like what I'd > > > expect can cause your lockups /

RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Friday, April 20, 2018 03:22 > On Thu 19-04-18 21:37:25, Dexuan Cui wrote: > > > From: Jan Kara > > > Sent: Thursday, April 19, 2018 13:23 > > > Good news guys, Robert has just spotted a bug which looks like what I'd > > > expect can cause your lockups / crashes. I've

Re: kernel panics with 4.14.X versions

On Thu 19-04-18 21:37:25, Dexuan Cui wrote: > > From: Jan Kara > > Sent: Thursday, April 19, 2018 13:23 > > Good news guys, Robert has just spotted a bug which looks like what I'd > > expect can cause your lockups / crashes. I've merged his patch to my tree > > and will push it to Linus for -rc3

Re: kernel panics with 4.14.X versions

On Thu 19-04-18 21:37:25, Dexuan Cui wrote: > > From: Jan Kara > > Sent: Thursday, April 19, 2018 13:23 > > Good news guys, Robert has just spotted a bug which looks like what I'd > > expect can cause your lockups / crashes. I've merged his patch to my tree > > and will push it to Linus for -rc3

RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Thursday, April 19, 2018 13:23 > Good news guys, Robert has just spotted a bug which looks like what I'd > expect can cause your lockups / crashes. I've merged his patch to my tree > and will push it to Linus for -rc3 so eventually it should land in > appropriate stable

RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Thursday, April 19, 2018 13:23 > Good news guys, Robert has just spotted a bug which looks like what I'd > expect can cause your lockups / crashes. I've merged his patch to my tree > and will push it to Linus for -rc3 so eventually it should land in > appropriate stable

Re: kernel panics with 4.14.X versions

On Thu, Apr 19, 2018 at 2:16 PM, Pavlos Parissis wrote: > On 19/04/2018 10:23 μμ, Jan Kara wrote: >> On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: >>> On 17/04/2018 02:12 μμ, Jan Kara wrote: On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > On 16/04/2018

Re: kernel panics with 4.14.X versions

On Thu, Apr 19, 2018 at 2:16 PM, Pavlos Parissis wrote: > On 19/04/2018 10:23 μμ, Jan Kara wrote: >> On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: >>> On 17/04/2018 02:12 μμ, Jan Kara wrote: On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > On 16/04/2018 04:40 μμ, Jan Kara wrote:

Re: kernel panics with 4.14.X versions

On 19/04/2018 10:23 μμ, Jan Kara wrote: > On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: >> On 17/04/2018 02:12 μμ, Jan Kara wrote: >>> On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: On 16/04/2018 04:40 μμ, Jan Kara wrote: >>> >>> >>> > How easily can you hit this? Very

Re: kernel panics with 4.14.X versions

On 19/04/2018 10:23 μμ, Jan Kara wrote: > On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: >> On 17/04/2018 02:12 μμ, Jan Kara wrote: >>> On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: On 16/04/2018 04:40 μμ, Jan Kara wrote: >>> >>> >>> > How easily can you hit this? Very

Re: kernel panics with 4.14.X versions

On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: > On 17/04/2018 02:12 μμ, Jan Kara wrote: > > On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > >> On 16/04/2018 04:40 μμ, Jan Kara wrote: > > > > > > > >>> How easily can you hit this? > >> > >> Very easily, I only need to wait 1-2 days for a

Re: kernel panics with 4.14.X versions

On Wed 18-04-18 10:32:21, Pavlos Parissis wrote: > On 17/04/2018 02:12 μμ, Jan Kara wrote: > > On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > >> On 16/04/2018 04:40 μμ, Jan Kara wrote: > > > > > > > >>> How easily can you hit this? > >> > >> Very easily, I only need to wait 1-2 days for a

Re: kernel panics with 4.14.X versions

On 17/04/2018 02:12 μμ, Jan Kara wrote: > On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: >> On 16/04/2018 04:40 μμ, Jan Kara wrote: > > > >>> How easily can you hit this? >> >> Very easily, I only need to wait 1-2 days for a crash to occur. > > I wouldn't call that very easily but opinions

Re: kernel panics with 4.14.X versions

On 17/04/2018 02:12 μμ, Jan Kara wrote: > On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: >> On 16/04/2018 04:40 μμ, Jan Kara wrote: > > > >>> How easily can you hit this? >> >> Very easily, I only need to wait 1-2 days for a crash to occur. > > I wouldn't call that very easily but opinions

RE: kernel panics with 4.14.X versions

> From: Greg KH > Sent: Tuesday, April 17, 2018 03:34 > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > > > From: Jan Kara > > > Sent: Monday, April 16, 2018 07:41 > > > ... > > > How easily can you hit this? Are you able to run debug kernels / inspect

RE: kernel panics with 4.14.X versions

> From: Greg KH > Sent: Tuesday, April 17, 2018 03:34 > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > > > From: Jan Kara > > > Sent: Monday, April 16, 2018 07:41 > > > ... > > > How easily can you hit this? Are you able to run debug kernels / inspect > > > crash dumps when the

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > On 16/04/2018 04:40 μμ, Jan Kara wrote: > > How easily can you hit this? > > Very easily, I only need to wait 1-2 days for a crash to occur. I wouldn't call that very easily but opinions may differ :). Anyway it's good (at least for

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 01:31:24, Pavlos Parissis wrote: > On 16/04/2018 04:40 μμ, Jan Kara wrote: > > How easily can you hit this? > > Very easily, I only need to wait 1-2 days for a crash to occur. I wouldn't call that very easily but opinions may differ :). Anyway it's good (at least for

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 13:18:46, Pavlos Parissis wrote: > On 17/04/2018 01:31 πμ, Pavlos Parissis wrote: > > On 16/04/2018 04:40 μμ, Jan Kara wrote: > >> On Mon 16-04-18 15:25:50, Guillaume Morin wrote: > >>> Fwiw, there have been already reports of similar soft lockups in > >>> fsnotify() on 4.14:

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 13:18:46, Pavlos Parissis wrote: > On 17/04/2018 01:31 πμ, Pavlos Parissis wrote: > > On 16/04/2018 04:40 μμ, Jan Kara wrote: > >> On Mon 16-04-18 15:25:50, Guillaume Morin wrote: > >>> Fwiw, there have been already reports of similar soft lockups in > >>> fsnotify() on 4.14:

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 14:48:35, Amir Goldstein wrote: > On Tue, Apr 17, 2018 at 1:33 PM, Greg KH wrote: > > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > >> > From: Jan Kara > >> > Sent: Monday, April 16, 2018 07:41 > >> > ... > >> > How easily can you hit

Re: kernel panics with 4.14.X versions

On Tue 17-04-18 14:48:35, Amir Goldstein wrote: > On Tue, Apr 17, 2018 at 1:33 PM, Greg KH wrote: > > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > >> > From: Jan Kara > >> > Sent: Monday, April 16, 2018 07:41 > >> > ... > >> > How easily can you hit this? Are you able to run

Re: kernel panics with 4.14.X versions

On Tue, Apr 17, 2018 at 1:33 PM, Greg KH wrote: > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: >> > From: Jan Kara >> > Sent: Monday, April 16, 2018 07:41 >> > ... >> > How easily can you hit this? Are you able to run debug kernels / inspect >> >

Re: kernel panics with 4.14.X versions

On Tue, Apr 17, 2018 at 1:33 PM, Greg KH wrote: > On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: >> > From: Jan Kara >> > Sent: Monday, April 16, 2018 07:41 >> > ... >> > How easily can you hit this? Are you able to run debug kernels / inspect >> > crash dumps when the issue occurs?

Re: kernel panics with 4.14.X versions

On 17/04/2018 01:31 πμ, Pavlos Parissis wrote: > On 16/04/2018 04:40 μμ, Jan Kara wrote: >> On Mon 16-04-18 15:25:50, Guillaume Morin wrote: >>> Fwiw, there have been already reports of similar soft lockups in >>> fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 >>> >>> We have also noticed

Re: kernel panics with 4.14.X versions

On 17/04/2018 01:31 πμ, Pavlos Parissis wrote: > On 16/04/2018 04:40 μμ, Jan Kara wrote: >> On Mon 16-04-18 15:25:50, Guillaume Morin wrote: >>> Fwiw, there have been already reports of similar soft lockups in >>> fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 >>> >>> We have also noticed

Re: kernel panics with 4.14.X versions

On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > > From: Jan Kara > > Sent: Monday, April 16, 2018 07:41 > > ... > > How easily can you hit this? Are you able to run debug kernels / inspect > > crash dumps when the issue occurs? Also testing with the latest mainline > >

Re: kernel panics with 4.14.X versions

On Mon, Apr 16, 2018 at 09:10:35PM +, Dexuan Cui wrote: > > From: Jan Kara > > Sent: Monday, April 16, 2018 07:41 > > ... > > How easily can you hit this? Are you able to run debug kernels / inspect > > crash dumps when the issue occurs? Also testing with the latest mainline > > kernel (4.16)

Re: kernel panics with 4.14.X versions

On 16/04/2018 04:40 μμ, Jan Kara wrote: > On Mon 16-04-18 15:25:50, Guillaume Morin wrote: >> Fwiw, there have been already reports of similar soft lockups in >> fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 >> >> We have also noticed similar softlockups with 4.14.22 here. > > Yeah. >

Re: kernel panics with 4.14.X versions

On 16/04/2018 04:40 μμ, Jan Kara wrote: > On Mon 16-04-18 15:25:50, Guillaume Morin wrote: >> Fwiw, there have been already reports of similar soft lockups in >> fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 >> >> We have also noticed similar softlockups with 4.14.22 here. > > Yeah. >

RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Monday, April 16, 2018 07:41 > ... > How easily can you hit this? Are you able to run debug kernels / inspect > crash dumps when the issue occurs? Also testing with the latest mainline > kernel (4.16) would be welcome whether this isn't just an issue with

RE: kernel panics with 4.14.X versions

> From: Jan Kara > Sent: Monday, April 16, 2018 07:41 > ... > How easily can you hit this? Are you able to run debug kernels / inspect > crash dumps when the issue occurs? Also testing with the latest mainline > kernel (4.16) would be welcome whether this isn't just an issue with the > backport

Re: kernel panics with 4.14.X versions

On 16 Apr 16:40, Jan Kara wrote: > Can you please run RIP through ./scripts/faddr2line to see where exactly > are we looping? I expect the loop iterating over marks to notify but better > be sure. > > How easily can you hit this? Are you able to run debug kernels / inspect > crash dumps when the

Re: kernel panics with 4.14.X versions

On 16 Apr 16:40, Jan Kara wrote: > Can you please run RIP through ./scripts/faddr2line to see where exactly > are we looping? I expect the loop iterating over marks to notify but better > be sure. > > How easily can you hit this? Are you able to run debug kernels / inspect > crash dumps when the

Re: kernel panics with 4.14.X versions

On Mon 16-04-18 15:25:50, Guillaume Morin wrote: > Fwiw, there have been already reports of similar soft lockups in > fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 > > We have also noticed similar softlockups with 4.14.22 here. Yeah. > On 16 Apr 13:54, Pavlos Parissis wrote: > > > >

Re: kernel panics with 4.14.X versions

On Mon 16-04-18 15:25:50, Guillaume Morin wrote: > Fwiw, there have been already reports of similar soft lockups in > fsnotify() on 4.14: https://lkml.org/lkml/2018/3/2/1038 > > We have also noticed similar softlockups with 4.14.22 here. Yeah. > On 16 Apr 13:54, Pavlos Parissis wrote: > > > >

kernel panics with 4.14.X versions

Hi all, We have observed kernel panics on several master kubernetes clusters, where we run kubernetes API services and not application workloads. Those clusters use kernel version 4.14.14 and 4.14.32, but we switched everything to kernel version 4.14.32 as a way to address the issue. We have

kernel panics with 4.14.X versions

Hi all, We have observed kernel panics on several master kubernetes clusters, where we run kubernetes API services and not application workloads. Those clusters use kernel version 4.14.14 and 4.14.32, but we switched everything to kernel version 4.14.32 as a way to address the issue. We have