Re: Getting empty callchain from perf_callchain_kernel()

2019-06-13 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 09:50:08AM -0500, Josh Poimboeuf wrote: > > Other than that, the same note as before, the 32bit JIT still seems > > buggered, but I'm not sure you (or anybody else) cares enough about that > > to fix it though. It seems to use ebp as its own frame pointer, which > >

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 10:54:23AM +0200, Peter Zijlstra wrote: > On Tue, Jun 11, 2019 at 10:05:01PM -0500, Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > > > For ORC, I'm thinking we may be able to just require that all generated > > > > code (BPF

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Josh Poimboeuf
On Wed, Jun 12, 2019 at 09:10:23AM -0400, Steven Rostedt wrote: > On Tue, 11 Jun 2019 22:05:01 -0500 > Josh Poimboeuf wrote: > > > Right now, ftrace has a special hook in the ORC unwinder > > (orc_ftrace_find). It would be great if we could get rid of that in > > favor of the "always use frame

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Steven Rostedt
On Tue, 11 Jun 2019 22:05:01 -0500 Josh Poimboeuf wrote: > Right now, ftrace has a special hook in the ORC unwinder > (orc_ftrace_find). It would be great if we could get rid of that in > favor of the "always use frame pointers" approach. I'll hold off on > doing the kpatch/kprobe trampoline

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-12 Thread Peter Zijlstra
On Tue, Jun 11, 2019 at 10:05:01PM -0500, Josh Poimboeuf wrote: > On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > > For ORC, I'm thinking we may be able to just require that all generated > > > code (BPF and others) always use frame pointers. Then when ORC doesn't > > >

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-11 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > > For ORC, I'm thinking we may be able to just require that all generated > > code (BPF and others) always use frame pointers. Then when ORC doesn't > > recognize a code address, it could try using the frame pointer as a > >

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-11 Thread Josh Poimboeuf
On Thu, Jun 06, 2019 at 04:04:48PM +, Song Liu wrote: > Hi Josh, > > Have you got luck fixing the ORC side? Here's the ORC fix. It's needed in addition to the bpf frame pointer fix (the previous patch). I'll clean the patches up and post them soon. diff --git

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-06 Thread Josh Poimboeuf
On Thu, Jun 06, 2019 at 04:04:48PM +, Song Liu wrote: > >> Here's a tentative BPF fix for the JIT frame pointer issue. It was a > >> bit harder than I expected. Encoding r12 as a base register requires a > >> SIB byte, so I had to add support for encoding that. I also simplified > >> the

Re: Getting empty callchain from perf_callchain_kernel()

2019-06-06 Thread Song Liu
> On May 27, 2019, at 4:57 AM, Kairui Song wrote: > > On Sat, May 25, 2019 at 7:23 AM Josh Poimboeuf wrote: >> >> On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: >>> On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: On Fri, May 24, 2019 at 12:41:59AM +0800,

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-27 Thread Kairui Song
On Sat, May 25, 2019 at 7:23 AM Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: > > On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > > > > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > > > On Thu, May 23, 2019 at 11:24 PM Josh

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:20:52AM +0800, Kairui Song wrote: > On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf > > > wrote: > > > > > > > > On Thu, May 23, 2019 at

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 10:53:19AM +0200, Peter Zijlstra wrote: > On Thu, May 23, 2019 at 10:24:13AM -0500, Josh Poimboeuf wrote: > > > Here's the latest version which should fix it in all cases (based on > > tip/master): > > > > > >

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-24 Thread Peter Zijlstra
On Thu, May 23, 2019 at 10:24:13AM -0500, Josh Poimboeuf wrote: > Here's the latest version which should fix it in all cases (based on > tip/master): > > > https://git.kernel.org/pub/scm/linux/kernel/git/jpoimboe/linux.git/commit/?h=bpf-orc-fix That patch suffers an inconsitency, the comment

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Fri, May 24, 2019 at 1:27 AM Josh Poimboeuf wrote: > > On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf > > wrote: > > > > > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > > > Hi Josh, this still won't

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Fri, May 24, 2019 at 12:41:59AM +0800, Kairui Song wrote: > On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf wrote: > > > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > > Hi Josh, this still won't fix the problem. > > > > > > > > > > Problem is not (or not only) with

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 11:24 PM Josh Poimboeuf wrote: > > On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > > Hi Josh, this still won't fix the problem. > > > > > > > > Problem is not (or not only) with ___bpf_prog_run, what actually went > > > > wrong is with the JITed bpf

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Thu, May 23, 2019 at 10:50:24PM +0800, Kairui Song wrote: > > > Hi Josh, this still won't fix the problem. > > > > > > Problem is not (or not only) with ___bpf_prog_run, what actually went > > > wrong is with the JITed bpf code. > > > > There seem to be a bunch of issues. My patch at least

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 9:32 PM Josh Poimboeuf wrote: > > On Thu, May 23, 2019 at 02:48:11PM +0800, Kairui Song wrote: > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > > > > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > > > On Wed, May 22, 2019 at 02:49:07PM

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Josh Poimboeuf
On Thu, May 23, 2019 at 02:48:11PM +0800, Kairui Song wrote: > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > > > The one that is broken is

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 4:28 PM Song Liu wrote: > > > On May 22, 2019, at 11:48 PM, Kairui Song wrote: > > > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > >> > >> On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > >>> On Wed, May 22, 2019 at 02:49:07PM +, Alexei

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Song Liu
> On May 22, 2019, at 11:48 PM, Kairui Song wrote: > > On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: >> >> On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: >>> On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: The one that is broken is

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-23 Thread Kairui Song
On Thu, May 23, 2019 at 7:46 AM Josh Poimboeuf wrote: > > On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > > The one that is broken is prog_tests/stacktrace_map.c > > > There we attach bpf to standard

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Wed, May 22, 2019 at 12:45:17PM -0500, Josh Poimboeuf wrote: > On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > > The one that is broken is prog_tests/stacktrace_map.c > > There we attach bpf to standard tracepoint where > > kernel suppose to collect pt_regs before calling

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Alexei Starovoitov
On 5/22/19 11:07 AM, Josh Poimboeuf wrote: > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >> On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote:

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > > Hi, > > > > > > > > We found a

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Josh Poimboeuf
On Wed, May 22, 2019 at 02:49:07PM +, Alexei Starovoitov wrote: > The one that is broken is prog_tests/stacktrace_map.c > There we attach bpf to standard tracepoint where > kernel suppose to collect pt_regs before calling into bpf. > And that's what bpf_get_stackid_tp() is doing. > It passes

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Alexei Starovoitov
On 5/22/19 7:02 AM, Peter Zijlstra wrote: > >> If the unwinder could trace back through the bpf func call then there >> will be no such problem. > > Why couldn't it trace back through the bpf stuff? And how can we fix > that? Most of the time there is no 'tracing through bpf stuff'. bpf infra

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Peter Zijlstra
On Mon, May 20, 2019 at 02:06:54AM +0800, Kairui Song wrote: > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > > > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > > > some other bfp functions) is

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-22 Thread Peter Zijlstra
On Mon, May 20, 2019 at 05:22:12PM +, Song Liu wrote: > I think this is still the best fix/workaround here? And only one level > of stack trace should be OK for tracepoint? No. That's still completely broken.

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
> On May 19, 2019, at 11:07 AM, Kairui Song wrote: > > On Sat, May 18, 2019 at 5:48 AM Song Liu wrote: >> >> >> >>> On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: >>> >>> On 5/17/19 11:40 AM, Song Liu wrote: +Alexei, Daniel, and bpf > On May 17, 2019, at 2:10 AM,

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
Sorry for previous empty email.. Clicked send by accident. > On May 19, 2019, at 11:06 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-20 Thread Song Liu
> On May 19, 2019, at 11:06 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >>> some other bfp functions) is now

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-19 Thread Kairui Song
On Sat, May 18, 2019 at 5:48 AM Song Liu wrote: > > > > > On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: > > > > On 5/17/19 11:40 AM, Song Liu wrote: > >> +Alexei, Daniel, and bpf > >> > >>> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: > >>> > >>> On Fri, May 17, 2019 at

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-19 Thread Kairui Song
On Fri, May 17, 2019 at 5:10 PM Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > > some other bfp functions) is now broken, or, strating an unwind > > directly inside a bpf program

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 2:06 PM, Alexei Starovoitov wrote: > > On 5/17/19 11:40 AM, Song Liu wrote: >> +Alexei, Daniel, and bpf >> >>> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: Hi, I think the actual problem

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Alexei Starovoitov
On 5/17/19 11:40 AM, Song Liu wrote: > +Alexei, Daniel, and bpf > >> On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: >> >> On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >>> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >>> some other bfp functions) is now

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
+Alexei, Daniel, and bpf > On May 17, 2019, at 2:10 AM, Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: >> Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe >> some other bfp functions) is now broken, or, strating an unwind >> directly

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 1:10 AM, Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: >> On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: >>> Hi, >>> >>> We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on >>> bpf/master

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Song Liu
> On May 17, 2019, at 1:32 AM, Kairui Song wrote: > > On Fri, May 17, 2019 at 4:15 PM Kairui Song wrote: >> >> On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: >>> >>> On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: On Thu, May 16, 2019 at 11:51:55PM +, Song

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Fri, May 17, 2019 at 04:15:39PM +0800, Kairui Song wrote: > Hi, I think the actual problem is that bpf_get_stackid_tp (and maybe > some other bfp functions) is now broken, or, strating an unwind > directly inside a bpf program will end up strangely. It have following > kernel message: Urgh,

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Kairui Song
On Fri, May 17, 2019 at 4:15 PM Kairui Song wrote: > > On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > > Hi, > > > > > > > > We found a failure with

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Kairui Song
On Fri, May 17, 2019 at 4:11 PM Peter Zijlstra wrote: > > On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > > Hi, > > > > > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map > > > (on

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Fri, May 17, 2019 at 09:46:00AM +0200, Peter Zijlstra wrote: > On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > > Hi, > > > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on > > bpf/master > > branch). > > > > After digging into the code, we found

Re: Getting empty callchain from perf_callchain_kernel()

2019-05-17 Thread Peter Zijlstra
On Thu, May 16, 2019 at 11:51:55PM +, Song Liu wrote: > Hi, > > We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on > bpf/master > branch). > > After digging into the code, we found that perf_callchain_kernel() is giving > empty > callchain for tracepoint

Getting empty callchain from perf_callchain_kernel()

2019-05-16 Thread Song Liu
Hi, We found a failure with selftests/bpf/tests_prog in test_stacktrace_map (on bpf/master branch). After digging into the code, we found that perf_callchain_kernel() is giving empty callchain for tracepoint sched/sched_switch. And it seems related to commit