Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > It makes my IVB very ill, starts spewing RCU stall warnings, but is > otherwise very unresponsive. > > Awesome... I'll prod at it when my brain works again. > Not sure if it's related, but I hit this on the core2 machine fuzzing overnight with

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > It makes my IVB very ill, starts spewing RCU stall warnings, but is > otherwise very unresponsive. > > Awesome... I'll prod at it when my brain works again. > Not sure if it's related, but I hit this on the core2 machine fuzzing overnight with

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 03:40:44PM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > > haswell. > > Confirmed, I can crash

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 03:40:44PM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > > haswell. > > Confirmed, I can crash

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > > > Yuck. This time it was stack recursion on the entry stack. In the > > > previous error, recursion was detected on the IRQ stack.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > > > Yuck. This time it was stack recursion on the entry stack. In the > > > previous error, recursion was detected on the IRQ stack.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 03:15:16PM -0500, Vince Weaver wrote: > > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > haswell. That certainly reduces the amount of code to stare at. Will do so tomorrow.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 03:15:16PM -0500, Vince Weaver wrote: > > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > haswell. That certainly reduces the amount of code to stare at. Will do so tomorrow.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > haswell. Confirmed, I can crash the system without the fuzzer, just by doing perf record

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > Not sure if this info helps, but if I make perf_fuzzer *not* create AUX > mmap() buffers, I'm unable to reproduce the hangs both on core2 and > haswell. Confirmed, I can crash the system without the fuzzer, just by doing perf record

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

Not sure if this info helps, but if I make perf_fuzzer *not* create AUX mmap() buffers, I'm unable to reproduce the hangs both on core2 and haswell. Vince

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

Not sure if this info helps, but if I make perf_fuzzer *not* create AUX mmap() buffers, I'm unable to reproduce the hangs both on core2 and haswell. Vince

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > > Yuck. This time it was stack recursion on the entry stack. In the > > previous error, recursion was detected on the IRQ stack. Otherwise they > > look quite similar. > > > > Was

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > > Yuck. This time it was stack recursion on the entry stack. In the > > previous error, recursion was detected on the IRQ stack. Otherwise they > > look quite similar. > > > > Was

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > Yuck. This time it was stack recursion on the entry stack. In the > previous error, recursion was detected on the IRQ stack. Otherwise they > look quite similar. > > Was that also with nopti? Both with pti enabled, nopti makes

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 01:21:12PM -0600, Josh Poimboeuf wrote: > Yuck. This time it was stack recursion on the entry stack. In the > previous error, recursion was detected on the IRQ stack. Otherwise they > look quite similar. > > Was that also with nopti? Both with pti enabled, nopti makes

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 02:00:27PM -0500, Vince Weaver wrote: > On Wed, 10 Jan 2018, Josh Poimboeuf wrote: > > > For the crash, you might try enabling CONFIG_DEBUG_ENTRY and seeing if > > that gives you any output. > > I did enable that, didn't seem to help on the haswell machien at least. > >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 02:00:27PM -0500, Vince Weaver wrote: > On Wed, 10 Jan 2018, Josh Poimboeuf wrote: > > > For the crash, you might try enabling CONFIG_DEBUG_ENTRY and seeing if > > that gives you any output. > > I did enable that, didn't seem to help on the haswell machien at least. > >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 06:03:47PM +0100, Peter Zijlstra wrote: > > On Thu, 11 Jan 2018, Vince Weaver wrote: > > [ 823.919729] BUG: unable to handle kernel paging request at > > 88011a7a1000 > > [ 823.926928] IP: 0x7fbda0042b3c > > > > I'm dumping vmlinux and can't find address

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 06:03:47PM +0100, Peter Zijlstra wrote: > > On Thu, 11 Jan 2018, Vince Weaver wrote: > > [ 823.919729] BUG: unable to handle kernel paging request at > > 88011a7a1000 > > [ 823.926928] IP: 0x7fbda0042b3c > > > > I'm dumping vmlinux and can't find address

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 01:20:10PM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > on the same core2 machine I got this which didn't crash the machine (but > > the perf_fuzzer process is stuck) > > also got this one: > > Cannot open /sys/kernel/tracing/kprobe_events

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 01:20:10PM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > on the same core2 machine I got this which didn't crash the machine (but > > the perf_fuzzer process is stuck) > > also got this one: > > Cannot open /sys/kernel/tracing/kprobe_events

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Wed, 10 Jan 2018, Josh Poimboeuf wrote: > For the crash, you might try enabling CONFIG_DEBUG_ENTRY and seeing if > that gives you any output. I did enable that, didn't seem to help on the haswell machien at least. > > > > WARNING: can't dereference iret registers at 0783fea8 for ip

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Wed, 10 Jan 2018, Josh Poimboeuf wrote: > For the crash, you might try enabling CONFIG_DEBUG_ENTRY and seeing if > that gives you any output. I did enable that, didn't seem to help on the haswell machien at least. > > > > WARNING: can't dereference iret registers at 0783fea8 for ip

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > on the same core2 machine I got this which didn't crash the machine (but > the perf_fuzzer process is stuck) also got this one: Cannot open /sys/kernel/tracing/kprobe_events [ 408.159243] watchdog: BUG: soft lockup - CPU#1 stuck for 23s!

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > on the same core2 machine I got this which didn't crash the machine (but > the perf_fuzzer process is stuck) also got this one: Cannot open /sys/kernel/tracing/kprobe_events [ 408.159243] watchdog: BUG: soft lockup - CPU#1 stuck for 23s!

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

on the same core2 machine I got this which didn't crash the machine (but the perf_fuzzer process is stuck) [ 4592.608066] INFO: task systemd-logind:488 blocked for more than 120 seconds. [ 4592.615159] Not tainted 4.15.0-rc7+ #211 [ 4592.619648] "echo 0 >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

on the same core2 machine I got this which didn't crash the machine (but the perf_fuzzer process is stuck) [ 4592.608066] INFO: task systemd-logind:488 blocked for more than 120 seconds. [ 4592.615159] Not tainted 4.15.0-rc7+ #211 [ 4592.619648] "echo 0 >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 11:58:24AM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > > > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > > > doesn't help, I'm going to switch to linus' tree with my

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 11:58:24AM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Vince Weaver wrote: > > > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > > > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > > > doesn't help, I'm going to switch to linus' tree with my

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > > doesn't help, I'm going to switch to linus' tree with my patches on. > > OK, I'm fuzzing on a core2 machine and it locks up too. >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > > doesn't help, I'm going to switch to linus' tree with my patches on. > > OK, I'm fuzzing on a core2 machine and it locks up too. >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > doesn't help, I'm going to switch to linus' tree with my patches on. OK, I'm fuzzing on a core2 machine and it locks up too. It did give the following first (but it kept going for

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > OK, I'm going to try fuzzing as a user with paranoid=0, and if that > doesn't help, I'm going to switch to linus' tree with my patches on. OK, I'm fuzzing on a core2 machine and it locks up too. It did give the following first (but it kept going for

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 10:26:14AM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > I'm seeing things like: > > > > Cannot open /sys/kernel/tracing/kprobe_events > > > > this is likely caused by me not having anything mounted there. Rostedt > > provided the magic

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 10:26:14AM -0500, Vince Weaver wrote: > On Thu, 11 Jan 2018, Peter Zijlstra wrote: > > > I'm seeing things like: > > > > Cannot open /sys/kernel/tracing/kprobe_events > > > > this is likely caused by me not having anything mounted there. Rostedt > > provided the magic

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 10:13:53AM +0100, Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > So remind me again, how are you running that fuzzer? I'm running > > > ./fast_repro99.sh as root. > > > > I'm

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, Jan 11, 2018 at 10:13:53AM +0100, Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > So remind me again, how are you running that fuzzer? I'm running > > > ./fast_repro99.sh as root. > > > > I'm

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > I'm seeing things like: > > Cannot open /sys/kernel/tracing/kprobe_events > > this is likely caused by me not having anything mounted there. Rostedt > provided the magic incantation to make that work, I'll go try now. The perf_fuzzer krpobe code is

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Thu, 11 Jan 2018, Peter Zijlstra wrote: > I'm seeing things like: > > Cannot open /sys/kernel/tracing/kprobe_events > > this is likely caused by me not having anything mounted there. Rostedt > provided the magic incantation to make that work, I'll go try now. The perf_fuzzer krpobe code is

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So remind me again, how are you running that fuzzer? I'm running > > ./fast_repro99.sh as root. > > I'm running ./fast_repro98.sh on a regular haswell machine with paranoid > set to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So remind me again, how are you running that fuzzer? I'm running > > ./fast_repro99.sh as root. > > I'm running ./fast_repro98.sh on a regular haswell machine with paranoid > set to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 11:07:16AM -0600, Josh Poimboeuf wrote: > On Tue, Jan 09, 2018 at 05:05:51PM +0100, Peter Zijlstra wrote: > > On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > > > So CONFIG_PAGE_TABLE_ISOLATION=y and

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 11:07:16AM -0600, Josh Poimboeuf wrote: > On Tue, Jan 09, 2018 at 05:05:51PM +0100, Peter Zijlstra wrote: > > On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > > > So CONFIG_PAGE_TABLE_ISOLATION=y and

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Vince Weaver wrote: > Also I managed to hit (presumably) the same bug on a skylake machine. > That one doesn't have a serial cable hooked up to it, I'll try to see if I > can find one. > > I am running debian-unstable with gcc 7.2 if it makes a difference. I built the

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Vince Weaver wrote: > Also I managed to hit (presumably) the same bug on a skylake machine. > That one doesn't have a serial cable hooked up to it, I'll try to see if I > can find one. > > I am running debian-unstable with gcc 7.2 if it makes a difference. I built the

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 01:09:57PM -0500, Steven Rostedt wrote: > On Tue, 9 Jan 2018 19:02:07 +0100 > Peter Zijlstra wrote: > > > This would globally serialize all perf_ioctl()'s, also that event_mutex > > is for trace_events and really does not belong in perf. > > > > So

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 01:09:57PM -0500, Steven Rostedt wrote: > On Tue, 9 Jan 2018 19:02:07 +0100 > Peter Zijlstra wrote: > > > This would globally serialize all perf_ioctl()'s, also that event_mutex > > is for trace_events and really does not belong in perf. > > > > So no, I really rather

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018 19:02:07 +0100 Peter Zijlstra wrote: > This would globally serialize all perf_ioctl()'s, also that event_mutex > is for trace_events and really does not belong in perf. > > So no, I really rather would not do this. > > The alternative I was thinking of

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018 19:02:07 +0100 Peter Zijlstra wrote: > This would globally serialize all perf_ioctl()'s, also that event_mutex > is for trace_events and really does not belong in perf. > > So no, I really rather would not do this. > > The alternative I was thinking of was lifting the cpuhp

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 12:53:46PM -0500, Steven Rostedt wrote: > Looking at ftrace_profile_set_filter(), I see it starts with: > > mutex_lock(_mutex); > > How much of a big deal would it be if we move taking event_mutex() into > perf_ioctl(), and then make ftrace_profile_set_filter() not

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 12:53:46PM -0500, Steven Rostedt wrote: > Looking at ftrace_profile_set_filter(), I see it starts with: > > mutex_lock(_mutex); > > How much of a big deal would it be if we move taking event_mutex() into > perf_ioctl(), and then make ftrace_profile_set_filter() not

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018 17:14:00 +0100 Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 04:12:53PM +0100, Peter Zijlstra wrote: > > > In any case, I found yet another lockdep splat, trying to figure out wth > > to do about that. > > An of course, testing this one yields yet

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018 17:14:00 +0100 Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 04:12:53PM +0100, Peter Zijlstra wrote: > > > In any case, I found yet another lockdep splat, trying to figure out wth > > to do about that. > > An of course, testing this one yields yet another lockdep splat..

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

I tried it again with your patch and force_early_printk, no luck. I can start dropping printks around the NMI code but I feel like I don't really know what I'm doing. Also I managed to hit (presumably) the same bug on a skylake machine. That one doesn't have a serial cable hooked up to it,

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

I tried it again with your patch and force_early_printk, no luck. I can start dropping printks around the NMI code but I feel like I don't really know what I'm doing. Also I managed to hit (presumably) the same bug on a skylake machine. That one doesn't have a serial cable hooked up to it,

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 05:05:51PM +0100, Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > > > 'work', right? > > > > yes.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 05:05:51PM +0100, Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > > > 'work', right? > > > > yes.

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 05:16:06PM +0100, Ingo Molnar wrote: > > + force_early_printk > > + Forcefully uses early_console (as per earlyprintk=) > > + usage for regular printk, bypassing everything, > > + including the syslog (dmesg will be

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 05:16:06PM +0100, Ingo Molnar wrote: > > + force_early_printk > > + Forcefully uses early_console (as per earlyprintk=) > > + usage for regular printk, bypassing everything, > > + including the syslog (dmesg will be

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

* Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 10:24:55AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > > I'll try your patch and see if it makes a difference. > > > > > > I suspect not, it shouldn't be PTI specific. > > > > yes,

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

* Peter Zijlstra wrote: > On Tue, Jan 09, 2018 at 10:24:55AM -0500, Vince Weaver wrote: > > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > > > I'll try your patch and see if it makes a difference. > > > > > > I suspect not, it shouldn't be PTI specific. > > > > yes, applying your patch

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 04:12:53PM +0100, Peter Zijlstra wrote: > In any case, I found yet another lockdep splat, trying to figure out wth > to do about that. An of course, testing this one yields yet another lockdep splat.. onwards to #3 :/ --- Subject: perf: Fix another perf,trace,cpuhp lock

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 04:12:53PM +0100, Peter Zijlstra wrote: > In any case, I found yet another lockdep splat, trying to figure out wth > to do about that. An of course, testing this one yields yet another lockdep splat.. onwards to #3 :/ --- Subject: perf: Fix another perf,trace,cpuhp lock

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > > 'work', right? > > yes. Previously I was changing CONFIG_PAGE_TABLE_ISOLATION and > recompiling, but just now

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 10:56:52AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > > 'work', right? > > yes. Previously I was changing CONFIG_PAGE_TABLE_ISOLATION and > recompiling, but just now

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > 'work', right? yes. Previously I was changing CONFIG_PAGE_TABLE_ISOLATION and recompiling, but just now I booted with it set to yes and pti=off and the fuzzer has been running

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > So CONFIG_PAGE_TABLE_ISOLATION=y and booting with "pti=off" makes it > 'work', right? yes. Previously I was changing CONFIG_PAGE_TABLE_ISOLATION and recompiling, but just now I booted with it set to yes and pti=off and the fuzzer has been running

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 10:24:55AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > I'll try your patch and see if it makes a difference. > > > > I suspect not, it shouldn't be PTI specific. > > yes, applying your patch didn't help, still locks up on the Haswell >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 10:24:55AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > > I'll try your patch and see if it makes a difference. > > > > I suspect not, it shouldn't be PTI specific. > > yes, applying your patch didn't help, still locks up on the Haswell >

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > I'll try your patch and see if it makes a difference. > > I suspect not, it shouldn't be PTI specific. yes, applying your patch didn't help, still locks up on the Haswell machine. Is there any debugging I could turn on that would help? I tried

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > I'll try your patch and see if it makes a difference. > > I suspect not, it shouldn't be PTI specific. yes, applying your patch didn't help, still locks up on the Haswell machine. Is there any debugging I could turn on that would help? I tried

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So remind me again, how are you running that fuzzer? I'm running > > ./fast_repro99.sh as root. > > I'm running ./fast_repro98.sh on a regular haswell machine with paranoid > set to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 08:44:23AM -0500, Vince Weaver wrote: > On Tue, 9 Jan 2018, Peter Zijlstra wrote: > > > So remind me again, how are you running that fuzzer? I'm running > > ./fast_repro99.sh as root. > > I'm running ./fast_repro98.sh on a regular haswell machine with paranoid > set to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > So remind me again, how are you running that fuzzer? I'm running > ./fast_repro99.sh as root. I'm running ./fast_repro98.sh on a regular haswell machine with paranoid set to "0". I'll try your patch and see if it makes a difference. I can also try

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, 9 Jan 2018, Peter Zijlstra wrote: > So remind me again, how are you running that fuzzer? I'm running > ./fast_repro99.sh as root. I'm running ./fast_repro98.sh on a regular haswell machine with paranoid set to "0". I'll try your patch and see if it makes a difference. I can also try

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 02:26:02PM +0100, Peter Zijlstra wrote: > OK, so I'm running on an IVB-EP with PTI enabled. I insta triggered a > lockdep splat --- Subject: perf: Fix lock inversion between perf,trace,cpuhp From: Peter Zijlstra Date: Tue Jan 9 13:10:30 CET 2018

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 02:26:02PM +0100, Peter Zijlstra wrote: > OK, so I'm running on an IVB-EP with PTI enabled. I insta triggered a > lockdep splat --- Subject: perf: Fix lock inversion between perf,trace,cpuhp From: Peter Zijlstra Date: Tue Jan 9 13:10:30 CET 2018 Lockdep gifted us:

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 11:25:07AM +0100, Peter Zijlstra wrote: > On Mon, Jan 08, 2018 at 03:29:42PM -0500, Vince Weaver wrote: > > On Mon, 8 Jan 2018, Ingo Molnar wrote: > > > > > > Note that the page table isolation (PTI) feature has a number of effects > > > on perf > > > and on NMI

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Tue, Jan 09, 2018 at 11:25:07AM +0100, Peter Zijlstra wrote: > On Mon, Jan 08, 2018 at 03:29:42PM -0500, Vince Weaver wrote: > > On Mon, 8 Jan 2018, Ingo Molnar wrote: > > > > > > Note that the page table isolation (PTI) feature has a number of effects > > > on perf > > > and on NMI

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Mon, Jan 08, 2018 at 03:29:42PM -0500, Vince Weaver wrote: > On Mon, 8 Jan 2018, Ingo Molnar wrote: > > > > Note that the page table isolation (PTI) feature has a number of effects on > > perf > > and on NMI handlers, so one of the things to try would be to disable PTI. > > Yes, it seems to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Mon, Jan 08, 2018 at 03:29:42PM -0500, Vince Weaver wrote: > On Mon, 8 Jan 2018, Ingo Molnar wrote: > > > > Note that the page table isolation (PTI) feature has a number of effects on > > perf > > and on NMI handlers, so one of the things to try would be to disable PTI. > > Yes, it seems to

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Mon, 8 Jan 2018, Ingo Molnar wrote: > > Note that the page table isolation (PTI) feature has a number of effects on > perf > and on NMI handlers, so one of the things to try would be to disable PTI. Yes, it seems to be a KPTI issue. With KPTI disabled I can fuzz for a few hours, no

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

On Mon, 8 Jan 2018, Ingo Molnar wrote: > > Note that the page table isolation (PTI) feature has a number of effects on > perf > and on NMI handlers, so one of the things to try would be to disable PTI. Yes, it seems to be a KPTI issue. With KPTI disabled I can fuzz for a few hours, no

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

* Vince Weaver wrote: > Hello > > Was trying out current git (4.15-rc7) and the perf_fuzzer very quickly > will lock up my Haswell test machine so solidly that I don't get any debug > info, even with a serial console. That's pretty concerning ... > I'll try

Re: perf: perf_fuzzer quickly locks up on 4.15-rc7

* Vince Weaver wrote: > Hello > > Was trying out current git (4.15-rc7) and the perf_fuzzer very quickly > will lock up my Haswell test machine so solidly that I don't get any debug > info, even with a serial console. That's pretty concerning ... > I'll try enabling various debug options

perf: perf_fuzzer quickly locks up on 4.15-rc7

Hello Was trying out current git (4.15-rc7) and the perf_fuzzer very quickly will lock up my Haswell test machine so solidly that I don't get any debug info, even with a serial console. I'll try enabling various debug options to see if I can get a more useful bug report. Vince

perf: perf_fuzzer quickly locks up on 4.15-rc7

Hello Was trying out current git (4.15-rc7) and the perf_fuzzer very quickly will lock up my Haswell test machine so solidly that I don't get any debug info, even with a serial console. I'll try enabling various debug options to see if I can get a more useful bug report. Vince