Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 01:28:59PM +0100, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Andy Lutomirski wrote: > > Our memory map code is utter shite. This kind of bug should not be > > possible without a giant warning at boot that something is screwed up. > > You're right it's utter shite and

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 01:28:59PM +0100, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Andy Lutomirski wrote: > > Our memory map code is utter shite. This kind of bug should not be > > possible without a giant warning at boot that something is screwed up. > > You're right it's utter shite and

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, 4 Jan 2018, Andy Lutomirski wrote: > On Thu, Jan 4, 2018 at 4:28 AM, Thomas Gleixner wrote: > > --- a/arch/x86/include/asm/pgtable_64_types.h > > +++ b/arch/x86/include/asm/pgtable_64_types.h > > @@ -88,7 +88,7 @@ typedef struct { pteval_t pte; } pte_t; > > # define

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, 4 Jan 2018, Andy Lutomirski wrote: > On Thu, Jan 4, 2018 at 4:28 AM, Thomas Gleixner wrote: > > --- a/arch/x86/include/asm/pgtable_64_types.h > > +++ b/arch/x86/include/asm/pgtable_64_types.h > > @@ -88,7 +88,7 @@ typedef struct { pteval_t pte; } pte_t; > > # define VMALLOC_SIZE_TB

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 4, 2018 at 4:28 AM, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Andy Lutomirski wrote: >> On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert >> wrote: >> > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: >> >> Maybe try

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 4, 2018 at 4:28 AM, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Andy Lutomirski wrote: >> On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert >> wrote: >> > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: >> >> Maybe try rebuilding a bad kernel with free_ldt_pgtables()

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Andy Lutomirski wrote: > On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert > wrote: > > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: > >> Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified > >> to do nothing, and

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Andy Lutomirski wrote: > On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert > wrote: > > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: > >> Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified > >> to do nothing, and the read

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 08:20:31AM +0100, Ingo Molnar wrote: > > * Greg Kroah-Hartman wrote: > > > On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > > > - (or it's something I missed to consider) > > > > It was a operator error, the issue is also on

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 08:20:31AM +0100, Ingo Molnar wrote: > > * Greg Kroah-Hartman wrote: > > > On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > > > - (or it's something I missed to consider) > > > > It was a operator error, the issue is also on 4.15-rc6, see another > >

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Ingo Molnar wrote: > These will cherry-pick cleanly, so it would be nice to test them on top of of > the > -stable kernel that fails: > > for N in 450cbdd0125c 4d2dc2cc766c 1e0f25dbf246 be62a3204406 0c3292ca8025 > 9d0b62328d34; do git cherry-pick $N; done > > if this

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Ingo Molnar wrote: > These will cherry-pick cleanly, so it would be nice to test them on top of of > the > -stable kernel that fails: > > for N in 450cbdd0125c 4d2dc2cc766c 1e0f25dbf246 be62a3204406 0c3292ca8025 > 9d0b62328d34; do git cherry-pick $N; done > > if this brute-force

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Greg Kroah-Hartman wrote: > On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > > - (or it's something I missed to consider) > > It was a operator error, the issue is also on 4.15-rc6, see another > email in this thread :) ah, ok :-) Nevertheless it

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Greg Kroah-Hartman wrote: > On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > > - (or it's something I missed to consider) > > It was a operator error, the issue is also on 4.15-rc6, see another > email in this thread :) ah, ok :-) Nevertheless it made sense to go through all

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > - (or it's something I missed to consider) It was a operator error, the issue is also on 4.15-rc6, see another email in this thread :)

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Thu, Jan 04, 2018 at 08:14:21AM +0100, Ingo Molnar wrote: > - (or it's something I missed to consider) It was a operator error, the issue is also on 4.15-rc6, see another email in this thread :)

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > > > On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > > > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > > > is also there (or not)? > > > > I haven't

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > > > On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > > > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > > > is also there (or not)? > > > > I haven't been able to

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: >> Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified >> to do nothing, and the read /sys/kernel/debug/page_tables/current (or

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 3, 2018 at 8:35 PM, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: >> Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified >> to do nothing, and the read /sys/kernel/debug/page_tables/current (or >> current_kernel, or whatever

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 05:37:42PM -0800, Benjamin Gilbert wrote: > I was caught by the fact that 4.14.11 has PAGE_TABLE_ISOLATION default y > but 4.15-rc6 doesn't. Retesting. It turns out that 4.15-rc6 has the same problem as 4.14.11. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 05:37:42PM -0800, Benjamin Gilbert wrote: > I was caught by the fact that 4.14.11 has PAGE_TABLE_ISOLATION default y > but 4.15-rc6 doesn't. Retesting. It turns out that 4.15-rc6 has the same problem as 4.14.11. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: > Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified > to do nothing, and the read /sys/kernel/debug/page_tables/current (or > current_kernel, or whatever it's called). The problem may be obvious. current_kernel

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:37:53PM -0800, Andy Lutomirski wrote: > Maybe try rebuilding a bad kernel with free_ldt_pgtables() modified > to do nothing, and the read /sys/kernel/debug/page_tables/current (or > current_kernel, or whatever it's called). The problem may be obvious. current_kernel

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:33:03PM -0800, Benjamin Gilbert wrote: > I haven't been able to reproduce this on 4.15-rc6. This is bad data. I was caught by the fact that 4.14.11 has PAGE_TABLE_ISOLATION default y but 4.15-rc6 doesn't. Retesting. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:33:03PM -0800, Benjamin Gilbert wrote: > I haven't been able to reproduce this on 4.15-rc6. This is bad data. I was caught by the fact that 4.14.11 has PAGE_TABLE_ISOLATION default y but 4.15-rc6 doesn't. Retesting. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:27:04PM -0800, Andy Lutomirski wrote: > How much memory does the affected system have? It sounds like something > is mapped in the LDT region and is getting corrupted because the LDT code > expects to own that region. We've seen this on systems from 1 to 7 GB.

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:27:04PM -0800, Andy Lutomirski wrote: > How much memory does the affected system have? It sounds like something > is mapped in the LDT region and is getting corrupted because the LDT code > expects to own that region. We've seen this on systems from 1 to 7 GB.

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > > is also there (or not)? > > I haven't been able to reproduce this on 4.15-rc6. Hmm. So we need to

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 4:33 PM, Benjamin Gilbert > wrote: > >> On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: >> Ick, not good, any chance you can test 4.15-rc6 to verify that the issue >> is also there (or not)? > > I haven't been able to

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > > is also there (or not)? > > I haven't been able to reproduce this on 4.15-rc6. Hmm. So we need to

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 4:33 PM, Benjamin Gilbert > wrote: > >> On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: >> Ick, not good, any chance you can test 4.15-rc6 to verify that the issue >> is also there (or not)? > > I haven't been able to reproduce this on 4.15-rc6. Ah.

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > is also there (or not)? I haven't been able to reproduce this on 4.15-rc6. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 10:20:16AM +0100, Greg Kroah-Hartman wrote: > Ick, not good, any chance you can test 4.15-rc6 to verify that the issue > is also there (or not)? I haven't been able to reproduce this on 4.15-rc6. --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > > > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: >> >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: Can you please send me your .config and

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > > > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: >> >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: Can you please send me your .config and a full dmesg ? >>>

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Andy Lutomirski wrote: > > On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: > >> > >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: >

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Andy Lutomirski wrote: > > On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: > >> > >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > Can you please

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > > > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: >> >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: Can you please send me your .config and

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

> On Jan 3, 2018, at 2:58 PM, Thomas Gleixner wrote: > > > >> On Wed, 3 Jan 2018, Thomas Gleixner wrote: >> >>> On Wed, 3 Jan 2018, Benjamin Gilbert wrote: On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: Can you please send me your .config and a full dmesg ? >>>

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > > > Can you please send me your .config and a full dmesg ? > > > > I've attached a serial log from a local QEMU. I can rerun with a

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Thomas Gleixner wrote: > On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > > > Can you please send me your .config and a full dmesg ? > > > > I've attached a serial log from a local QEMU. I can rerun with a

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > > Can you please send me your .config and a full dmesg ? > > I've attached a serial log from a local QEMU. I can rerun with a higher > loglevel if need be. Thanks! Cc'ing Andy who

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > > Can you please send me your .config and a full dmesg ? > > I've attached a serial log from a local QEMU. I can rerun with a higher > loglevel if need be. Thanks! Cc'ing Andy who

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > Can you please send me your .config and a full dmesg ? I've attached a serial log from a local QEMU. I can rerun with a higher loglevel if need be. --Benjamin Gilbert config-4.14.11.gz Description: application/gzip

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 11:34:46PM +0100, Thomas Gleixner wrote: > Can you please send me your .config and a full dmesg ? I've attached a serial log from a local QEMU. I can rerun with a higher loglevel if need be. --Benjamin Gilbert config-4.14.11.gz Description: application/gzip

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 04:48:33PM +0100, Ingo Molnar wrote: > > first please test the latest WIP.x86/pti branch which has a couple of fixes. > > I'm still seeing the problem with that branch (3ffdeb1a02be, plus a couple > of local patches which

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, 3 Jan 2018, Benjamin Gilbert wrote: > On Wed, Jan 03, 2018 at 04:48:33PM +0100, Ingo Molnar wrote: > > first please test the latest WIP.x86/pti branch which has a couple of fixes. > > I'm still seeing the problem with that branch (3ffdeb1a02be, plus a couple > of local patches which

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:48:33PM +0100, Ingo Molnar wrote: > first please test the latest WIP.x86/pti branch which has a couple of fixes. I'm still seeing the problem with that branch (3ffdeb1a02be, plus a couple of local patches which shouldn't affect the resulting binary). --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 04:48:33PM +0100, Ingo Molnar wrote: > first please test the latest WIP.x86/pti branch which has a couple of fixes. I'm still seeing the problem with that branch (3ffdeb1a02be, plus a couple of local patches which shouldn't affect the resulting binary). --Benjamin Gilbert

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Greg Kroah-Hartman wrote: > On Wed, Jan 03, 2018 at 12:46:00AM -0800, Benjamin Gilbert wrote: > > [resending with less web] > > (adding lkml and x86 developers) > > > Hi all, > > > > In our regression tests on kernel 4.14.11, we're occasionally seeing a run > >

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

* Greg Kroah-Hartman wrote: > On Wed, Jan 03, 2018 at 12:46:00AM -0800, Benjamin Gilbert wrote: > > [resending with less web] > > (adding lkml and x86 developers) > > > Hi all, > > > > In our regression tests on kernel 4.14.11, we're occasionally seeing a run > > of "bad pmd" messages during

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 12:46:00AM -0800, Benjamin Gilbert wrote: > [resending with less web] (adding lkml and x86 developers) > Hi all, > > In our regression tests on kernel 4.14.11, we're occasionally seeing a run > of "bad pmd" messages during boot, followed by a "BUG: unable to handle >

Re: "bad pmd" errors + oops with KPTI on 4.14.11 after loading X.509 certs

On Wed, Jan 03, 2018 at 12:46:00AM -0800, Benjamin Gilbert wrote: > [resending with less web] (adding lkml and x86 developers) > Hi all, > > In our regression tests on kernel 4.14.11, we're occasionally seeing a run > of "bad pmd" messages during boot, followed by a "BUG: unable to handle >