Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-04-16 Thread He Zhe
>From the last replies in the thread, it seems some work is going on. May I ask when we can possibly roughly have a fix or workaround? Thanks, Zhe On 3/21/19 10:15 AM, Steven Rostedt wrote: > From: Steven Rostedt (VMware) > > He Zhe reported a crash by enabling trace events and selecting >

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Juergen Gross
On 21/03/2019 19:18, Linus Torvalds wrote: > On Thu, Mar 21, 2019 at 11:05 AM Andy Lutomirski wrote: >> >> In the long run, I think the right solution is to rewrite even more of >> this mess in C. We really ought to be able to put the IRQ flag >> tracing and the context tracking into C code. >

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 1:50 PM Peter Zijlstra wrote: > > On Thu, Mar 21, 2019 at 11:27:00AM -0700, Andy Lutomirski wrote: > > Well, here's pass zero at this. Untested, because it obviously > > doesn't work. Here are just a few things that are almost certainly > > wrong with it > > Aah, you're

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 11:27:00AM -0700, Andy Lutomirski wrote: > Well, here's pass zero at this. Untested, because it obviously > doesn't work. Here are just a few things that are almost certainly > wrong with it Aah, you're proposing to simply not do TRACE_IRQS_OFF and

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 20:00:19 + Andrew Cooper wrote: > I'm afraid that I've not got the original context of this thread, but > I'm going to guess that something is resulting in a #PF before %cr2 > before it gets saved on the #PF path, and using a PVOP causes things to > explode? Actually it

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 21:03:16 +0100 Peter Zijlstra wrote: > Doesn't make sense; you say data, but then talk code and i$. Ah, you meant d$ > > Not the point, spinlock_t is 4 bytes, but growns into a monster with > lockdep on. There are plenty locations where the spinlock and the data > it

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 03:50:06PM -0400, Steven Rostedt wrote: > On Thu, 21 Mar 2019 20:31:52 +0100 > Peter Zijlstra wrote: > > > > > > > No I didn't. Some users only care about performance, but find memory > > > cheap. > > > > Because cache-misses are free? > > If I ever did implement

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andrew Cooper
On 21/03/2019 18:39, Andy Lutomirski wrote: > On Thu, Mar 21, 2019 at 11:37 AM Peter Zijlstra wrote: >> On Thu, Mar 21, 2019 at 11:25:44AM -0700, Linus Torvalds wrote: >>> On Thu, Mar 21, 2019 at 11:21 AM Andy Lutomirski wrote: I dunno. Lots of people at least use to have serious

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 20:31:52 +0100 Peter Zijlstra wrote: > > > > No I didn't. Some users only care about performance, but find memory > > cheap. > > Because cache-misses are free? If I ever did implement this, I would try to get all the data out of line as much as possible, where only a nop

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 02:55:51PM -0400, Steven Rostedt wrote: > On Thu, 21 Mar 2019 19:28:30 +0100 > Peter Zijlstra wrote: > > > On Thu, Mar 21, 2019 at 02:10:20PM -0400, Steven Rostedt wrote: > > > On Thu, 21 Mar 2019 11:05:06 -0700 > > > Andy Lutomirski wrote: > > > > > > > In the long

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 19:28:30 +0100 Peter Zijlstra wrote: > On Thu, Mar 21, 2019 at 02:10:20PM -0400, Steven Rostedt wrote: > > On Thu, 21 Mar 2019 11:05:06 -0700 > > Andy Lutomirski wrote: > > > > > In the long run, I think the right solution is to rewrite even more of > > > this mess in C.

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 11:38:25AM -0700, Andy Lutomirski wrote: > I suspect we'll want PV spinlocks and other goodies like PV TLB > shootdown for a long time. The stuff I'd like to kill eventually is > the PV "I'm not actually at CPL 0 so I'm faking it" part. Right, and it's exactly those bits

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 11:37 AM Peter Zijlstra wrote: > > On Thu, Mar 21, 2019 at 11:25:44AM -0700, Linus Torvalds wrote: > > On Thu, Mar 21, 2019 at 11:21 AM Andy Lutomirski wrote: > > > > > > I dunno. Lots of people at least use to have serious commercial interest > > > in it. > > > > Yes,

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 11:33 AM Linus Torvalds wrote: > > On Thu, Mar 21, 2019 at 11:21 AM Andy Lutomirski wrote: > > > > I dunno. Lots of people at least use to have serious commercial interest > > in it. > > Yes, it used to be a big deal. But full virtualization has gotten a > lot more

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 11:25:44AM -0700, Linus Torvalds wrote: > On Thu, Mar 21, 2019 at 11:21 AM Andy Lutomirski wrote: > > > > I dunno. Lots of people at least use to have serious commercial interest > > in it. > > Yes, it used to be a big deal. But full virtualization has gotten a > lot

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 11:28:50AM -0700, Andy Lutomirski wrote: > On Thu, Mar 21, 2019 at 11:27 AM Peter Zijlstra wrote: > > > > On Thu, Mar 21, 2019 at 11:05:06AM -0700, Andy Lutomirski wrote: > > > > > Ugh. > > > > > > I certainly agree in principle that sticking the CR2 read into the asm > >

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 11:27 AM Peter Zijlstra wrote: > > On Thu, Mar 21, 2019 at 11:05:06AM -0700, Andy Lutomirski wrote: > > > Ugh. > > > > I certainly agree in principle that sticking the CR2 read into the asm > > is the right solution. But this patch makes the spaghetti even more > >

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 02:10:20PM -0400, Steven Rostedt wrote: > On Thu, 21 Mar 2019 11:05:06 -0700 > Andy Lutomirski wrote: > > > In the long run, I think the right solution is to rewrite even more of > > this mess in C. We really ought to be able to put the IRQ flag > > tracing and the

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 11:05:06AM -0700, Andy Lutomirski wrote: > Ugh. > > I certainly agree in principle that sticking the CR2 read into the asm > is the right solution. But this patch makes the spaghetti even more > tangled. Maybe we can rearrange the code a bit so that the entry > sequence

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 11:10 AM Steven Rostedt wrote: > > On Thu, 21 Mar 2019 11:05:06 -0700 > Andy Lutomirski wrote: > > > In the long run, I think the right solution is to rewrite even more of > > this mess in C. We really ought to be able to put the IRQ flag > > tracing and the context

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Linus Torvalds
On Thu, Mar 21, 2019 at 11:21 AM Andy Lutomirski wrote: > > I dunno. Lots of people at least use to have serious commercial interest in > it. Yes, it used to be a big deal. But full virtualization has gotten a lot more common and better. > Hey Xen folks, how close are we to being able to say

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread hpa
On March 21, 2019 11:18:53 AM PDT, Linus Torvalds wrote: >On Thu, Mar 21, 2019 at 11:05 AM Andy Lutomirski >wrote: >> >> In the long run, I think the right solution is to rewrite even more >of >> this mess in C. We really ought to be able to put the IRQ flag >> tracing and the context tracking

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 11:19 AM Linus Torvalds wrote: > > On Thu, Mar 21, 2019 at 11:05 AM Andy Lutomirski wrote: > > > > In the long run, I think the right solution is to rewrite even more of > > this mess in C. We really ought to be able to put the IRQ flag > > tracing and the context

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Linus Torvalds
On Thu, Mar 21, 2019 at 11:05 AM Andy Lutomirski wrote: > > In the long run, I think the right solution is to rewrite even more of > this mess in C. We really ought to be able to put the IRQ flag > tracing and the context tracking into C code. Tangentially about this long run thing - can we

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 11:05:06 -0700 Andy Lutomirski wrote: > In the long run, I think the right solution is to rewrite even more of > this mess in C. We really ought to be able to put the IRQ flag > tracing and the context tracking into C code. And once we do that, we can work on getting the

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Andy Lutomirski
On Thu, Mar 21, 2019 at 10:22 AM Peter Zijlstra wrote: > > On Thu, Mar 21, 2019 at 09:32:42AM -0400, Steven Rostedt wrote: > > On Thu, 21 Mar 2019 11:45:17 +0100 > > Peter Zijlstra wrote: > > > > > - .if \paranoid > > > + .if \read_cr2 > > > + mov %cr2, %rdx /* XXX

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Linus Torvalds
On Thu, Mar 21, 2019 at 6:55 AM Steven Rostedt wrote: > > Looks to be an issue with the save_stack_trace_user() not checking if > the address is canonical before reading it. I guess access_ok() doesn't > check that. access_ok() definitely does check for non-canonical. But it only does so when

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 09:32:42AM -0400, Steven Rostedt wrote: > On Thu, 21 Mar 2019 11:45:17 +0100 > Peter Zijlstra wrote: > > > - .if \paranoid > > + .if \read_cr2 > > + mov %cr2, %rdx /* XXX paravirt crap */ > > + .endif > > + > > I'm guessing this breaks

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 09:32:42 -0400 Steven Rostedt wrote: > I tested your code and it also fixes the issue, Although I just hit this: [ cut here ] General protection fault in user access. Non-canonical address? WARNING: CPU: 2 PID: 1620 at arch/x86/mm/extable.c:125

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 11:45:17 +0100 Peter Zijlstra wrote: > - .if \paranoid > + .if \read_cr2 > + mov %cr2, %rdx /* XXX paravirt crap */ > + .endif > + I'm guessing this breaks paravirt, as that's one reason I didn't add the read_rc in assembly. I tested

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Steven Rostedt
On Thu, 21 Mar 2019 09:33:17 +0100 Peter Zijlstra wrote: > On Wed, Mar 20, 2019 at 10:15:34PM -0400, Steven Rostedt wrote: > > > And it would crash similarly each time I tried it, but always at a > > different place. After spending the day on this, I finally figured it > > out. The bug is

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 10:02:41AM +0100, Peter Zijlstra wrote: > On Thu, Mar 21, 2019 at 09:33:17AM +0100, Peter Zijlstra wrote: > > I'm thinking this problem wasn't new. > > > idtentry page_fault do_page_fault has_error_code=1 > > call error_entry > > TRACE_IRQS_OFF

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Thu, Mar 21, 2019 at 09:33:17AM +0100, Peter Zijlstra wrote: I'm thinking this problem wasn't new. > idtentry page_fault do_page_fault has_error_code=1 > call error_entry > TRACE_IRQS_OFF > call trace_hardirqs_off* > > # modifies CR2

Re: [RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-21 Thread Peter Zijlstra
On Wed, Mar 20, 2019 at 10:15:34PM -0400, Steven Rostedt wrote: > And it would crash similarly each time I tried it, but always at a > different place. After spending the day on this, I finally figured it > out. The bug is happening in entry_64.S right after error_entry. > There's two

[RFC][PATCH] tracing/x86: Save CR2 before tracing irqsoff on error_entry

2019-03-20 Thread Steven Rostedt
From: Steven Rostedt (VMware) He Zhe reported a crash by enabling trace events and selecting "userstacktrace" which will read the stack of userspace for every trace event recorded. Zhe narrowed it down to: c3bc8fd637a9 ("tracing: Centralize preemptirq tracepoints and unify their usage") With