Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
Hi Steve, On Sat, Jul 18, 2020 at 4:44 AM Steven Rostedt wrote: > > On Fri, 17 Jul 2020 16:34:55 -0300 > Arnaldo Carvalho de Melo wrote: > > Thinking a bit more, I have to ask. Does perf use the kernel when > getting all the children of an existing task, or is that done only in > userspace? > > That is, is there a perf syscall that says "start tracing this task and > all its existing children"? > > Or is it done by perf user space looking at the /proc filesystem (like > ps does). Yep, perf does look up the /proc to get a list of threads in a process. Thanks Namhyung > > I'm asking because if perf has a syscall to do that, then I probably > should add a way to do that with ftrace as well. But that's really > trivial, because all it would take is grabbing the task_list lock and > iterating over all the children. Getting new children was the > non-trivial part, which was what I focused on (with the fork options). > > If perf does it with proc files, then we don't need to change anything > as that could still be used with ftrace. > > > Changbin, you can take from here :-) > > > > And to reiterate, for me the value of 'perf ftrace' is to allow people > > used to perf to be able to switch to ftrace quickly, just changing: > > > >perf record/top/stat/trace/report/script/etc --pid 1234 > > > > by: > > > >perf ftrace --pid 1234 > > > > And have the tracefs ftrace knobs set up to have what is expected in > > terms of targets to trace as the other perf tools. > > > > And not just --pid and --tid, but --cgroup, --cpu, etc. > > > > i.e., 'perf ftrace' being _a_ front-end aplication to ftrace. > > > > :-) > > > I have no problem with this, and I'm quite excited about it. I would > like it to use libtracefs, as it looks to be exactly what we are > working on. And this is now a high priority to get out, and I don't > expect another year (or two) in doing so ;-) > > -- Steve
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
On Fri, Jul 17, 2020 at 01:01:24PM -0400, Steven Rostedt wrote:
> On Fri, 17 Jul 2020 21:26:50 +0800
> Changbin Du wrote:
>
> > On Thu, Jul 16, 2020 at 12:36:30PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu:
> > > > This allows us to trace single thread instead of the whole process.
> > > >
> > > > Signed-off-by: Changbin Du
> > > > ---
> > > > tools/perf/Documentation/perf-ftrace.txt | 4
> > > > tools/perf/builtin-ftrace.c | 2 ++
> > > > 2 files changed, 6 insertions(+)
> > > >
> > > > diff --git a/tools/perf/Documentation/perf-ftrace.txt
> > > > b/tools/perf/Documentation/perf-ftrace.txt
> > > > index d79560dea19f..e204bf6d50d8 100644
> > > > --- a/tools/perf/Documentation/perf-ftrace.txt
> > > > +++ b/tools/perf/Documentation/perf-ftrace.txt
> > > > @@ -38,6 +38,10 @@ OPTIONS
> > > > --pid=::
> > > > Trace on existing process id (comma separated list).
> > > >
> > > > +-t::
> > > > +--tid=::
> > > > + Trace on existing thread id (comma separated list).
> > > > +
> > >
> > >
> > > Humm, I just tried:
> > >
> > > [root@five ~]# yes > /dev/null &
> > > [1] 18265
> > > [root@five ~]# perf ftrace --tid 18265
> > > ^C[root@five ~]#
> > >
> > > After waiting for a while, nothing, what am I doing wrong?
> > >
> > I got it wrong. Currently ftrace only can filter by pid. If the pid is not
> > the main thread it won't work.
>
> Wait what?
>
> The "pid" for ftrace is really a "task id" which is a thread id.
>
My bad. I traced a sleeping thread yesterday so no event generated.
Now it works:
$ pstree -p 2378
qemu-system-x86(2378)─┬─{qemu-system-x86}(2379)
├─{qemu-system-x86}(2382)
├─{qemu-system-x86}(2385)
├─{qemu-system-x86}(2387)
├─{qemu-system-x86}(2388)
├─{qemu-system-x86}(2389)
├─{qemu-system-x86}(2390)
├─{qemu-system-x86}(2391)
├─{qemu-system-x86}(2392)
$ sudo ./perf ftrace --tid 2388
[sudo] password for changbin:
# tracer: function
#
# entries-in-buffer/entries-written: 0/0 #P:8
#
# TASK-PID CPU# TIMESTAMP FUNCTION
# | | | | |
-0 [001] 6561.553989: switch_mm_irqs_off <-__schedule
-0 [001] 6561.553996: load_new_mm_cr3 <-switch_mm_irqs_off
qemu-system-x86-2388 [001] 6561.553997: finish_task_switch <-__schedule
qemu-system-x86-2388 [001] 6561.553998: smp_irq_work_interrupt
<-irq_work_interrupt
qemu-system-x86-2388 [001] 6561.553999: irq_enter <-smp_irq_work_interrupt
qemu-system-x86-2388 [001] 6561.553999: rcu_irq_enter <-irq_enter
qemu-system-x86-2388 [001] 6561.554000: __wake_up <-rb_wake_up_waiters
qemu-system-x86-2388 [001] 6561.554000: __wake_up_common_lock <-__wake_up
qemu-system-x86-2388 [001] 6561.554000: _raw_spin_lock_irqsave
<-__wake_up_common_lock
qemu-system-x86-2388 [001] 6561.554000: __wake_up_common
<-__wake_up_common_lock
...
> [root@bxtest ~]# yes > /dev/null &
> [1] 6799
> [root@bxtest ~]# trace-cmd record -e all -P 6799
> Hit Ctrl^C to stop recording
> ^CCPU 0: 3573031 events lost
> CPU0 data recorded at offset=0x838000
> 627675136 bytes in size
> CPU1 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU2 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU3 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU4 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU5 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU6 data recorded at offset=0x25ed1000
> 0 bytes in size
> CPU7 data recorded at offset=0x25ed1000
> 0 bytes in size
> [root@bxtest ~]# trace-cmd report | head
> CPU 1 is empty
> CPU 2 is empty
> CPU 3 is empty
> CPU 4 is empty
> CPU 5 is empty
> CPU 6 is empty
> CPU 7 is empty
> cpus=8
> yes-6799 [000] 67825.580902: sys_exit: NR 1 = 8192
> yes-6799 [000] 67825.580903: sys_exit_write: 0x2000
>
>
> What's different about tid vs pid?
>
> -- Steve
>
>
>
> >
> > So this patch makes no sense. will drop this.
> >
--
Cheers,
Changbin Du
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
On Fri, 17 Jul 2020 16:34:55 -0300 Arnaldo Carvalho de Melo wrote: Thinking a bit more, I have to ask. Does perf use the kernel when getting all the children of an existing task, or is that done only in userspace? That is, is there a perf syscall that says "start tracing this task and all its existing children"? Or is it done by perf user space looking at the /proc filesystem (like ps does). I'm asking because if perf has a syscall to do that, then I probably should add a way to do that with ftrace as well. But that's really trivial, because all it would take is grabbing the task_list lock and iterating over all the children. Getting new children was the non-trivial part, which was what I focused on (with the fork options). If perf does it with proc files, then we don't need to change anything as that could still be used with ftrace. > Changbin, you can take from here :-) > > And to reiterate, for me the value of 'perf ftrace' is to allow people > used to perf to be able to switch to ftrace quickly, just changing: > >perf record/top/stat/trace/report/script/etc --pid 1234 > > by: > >perf ftrace --pid 1234 > > And have the tracefs ftrace knobs set up to have what is expected in > terms of targets to trace as the other perf tools. > > And not just --pid and --tid, but --cgroup, --cpu, etc. > > i.e., 'perf ftrace' being _a_ front-end aplication to ftrace. > > :-) I have no problem with this, and I'm quite excited about it. I would like it to use libtracefs, as it looks to be exactly what we are working on. And this is now a high priority to get out, and I don't expect another year (or two) in doing so ;-) -- Steve
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
Em Fri, Jul 17, 2020 at 01:53:51PM -0400, Steven Rostedt escreveu: > On Fri, 17 Jul 2020 14:40:53 -0300 Arnaldo Carvalho de Melo > wrote: > > Say you use: > > ^C[root@ssdandy ~]# cyclictest --smp -um -p95 > > # /dev/cpu_dma_latency set to 0us > > policy: fifo: loadavg: 0.05 0.03 0.06 2/409 29072 > > T: 0 (29065) P:95 I:1000 C:518 Min: 2 Act:2 Avg:2 Max: > > 6 > > T: 1 (29066) P:95 I:1500 C:343 Min: 2 Act:2 Avg:2 Max: > > 5 > > T: 2 (29067) P:95 I:2000 C:256 Min: 2 Act:2 Avg:2 Max: > > 7 > > T: 3 (29068) P:95 I:2500 C:203 Min: 2 Act:2 Avg:2 Max: > > 5 > > T: 4 (29069) P:95 I:3000 C:168 Min: 2 Act:2 Avg:3 Max: > > 6 > > T: 5 (29070) P:95 I:3500 C:143 Min: 2 Act:2 Avg:2 Max: > > 6 > > T: 6 (29071) P:95 I:4000 C:124 Min: 2 Act:2 Avg:2 Max: > > 7 > > T: 7 (29072) P:95 I:4500 C:110 Min: 2 Act:2 Avg:2 Max: > > Then we do: > > # pidof cyclictest > > 29064 > > # > > If we use: > > [root@ssdandy ~]# perf record --pid 29064 > > ^C[ perf record: Woken up 1 times to write data ] > > [ perf record: Captured and wrote 0.052 MB perf.data (866 samples) ] > > [root@ssdandy ~]# perf report --task > > # pid tid ppid comm > > 00 -1 |swapper > > 2906429064 -1 |cyclictest > > 2906429065 -1 |cyclictest > > 2906429066 -1 |cyclictest > > 2906429067 -1 |cyclictest > > 2906429068 -1 |cyclictest > > 2906429069 -1 |cyclictest > > 2906429070 -1 |cyclictest > > 2906429071 -1 |cyclictest > > 2906429072 -1 |cyclictest > > [root@ssdandy ~]# > "ftrace" is inside the kernel. But you could specify all those PIDs in > the set_ftrace_pid and set_event_pid files, and they will be traced. If > you want to trace the children of those PIDs, you would need to set the > options function function-fork and event-fork respectively. And then > any time a process with a pid in the set_ftrace_pid or set_event_pid > file forks, its child will also be added to that file and it too will > be traced. If the fork options are set, then when a task exits, its pid > will be removed from the file. > echo 1 > options/function-fork > echo 1 > options/event-fork cool, so its possible to have the same sort of behaviour one expects using --pid or --tid and --inherit with perf record/trace/top/etc Its just some confusion about what perf understands by --pid and --tid and the jargon used by ftrace, we can make 'perf ftrace' use the knobs you mentioned and achieve the expected results as with other perf subcommands, good. > > If we are interested only on the thread running on CPU3 we can do: > > > > [root@ssdandy ~]# perf report --task > > # pid tid ppid comm > > 00 -1 |swapper > > 2906429064 -1 |cyclictest > > 2906429068 -1 |cyclictest > > [root@ssdandy ~]# > > The first 29064 is just to have info on who created 29068, i.e.: > > Its just the synthesized info for 29068 creator: > > [root@ssdandy ~]# perf report -D | grep -w 29064/29064 > > 0 0x4690 [0x30]: PERF_RECORD_COMM: cyclictest:29064/29064 > > 0 0x46c0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x40(0xa000) @ 0 fd:00 > > 136246288 0]: r-xp /usr/bin/cyclictest > > 0 0x4730 [0x80]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f505000(0x15000) @ > > 0 fd:00 201479398 0]: r-xp /usr/lib64/libgcc_s-4.8.5-20150702.so.1 > > 0 0x47b0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f71b000(0x1c3000) @ > > 0 fd:00 201334455 0]: r-xp /usr/lib64/libc-2.17.so > > 0 0x4820 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990fae9000(0xa000) @ 0 > > fd:00 204604380 0]: r-xp /usr/lib64/libnuma.so.1.0.0 > > 0 0x4898 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990fcf5000(0x17000) @ > > 0 fd:00 201335636 0]: r-xp /usr/lib64/libpthread-2.17.so > > 0 0x4910 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990ff11000(0x7000) @ 0 > > fd:00 201335640 0]: r-xp /usr/lib64/librt-2.17.so > > 0 0x4988 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x7f9910119000(0x22000) @ > > 0 fd:00 203595299 0]: r-xp /usr/lib64/ld-2.17.so > > 0 0x49f8 [0x60]: PERF_RECORD_MMAP2 29064/29064: [0x7fff0b52d000(0x2000) @ 0 > > 00:00 0 0]: r-xp [vdso] > > 0 0x4a58 [0x68]: PERF_RECORD_MMAP2 29064/29064: [0xff60(0x1000) > > @ 0 00:00 0 0]: r-xp [vsyscall] > > [root@ssdandy ~]# > > No PERF_RECORD_SAMPLEs. > > Those are only for: > > [root@ssdandy ~]# perf report -D | grep PERF_RECORD_SAMPLE | head -5 > > 147224656598815 0x4ac0 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > > 0xa8e5b568 period: 1 addr: 0 > > 147224656606270 0x4ae8 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > > 0xa8e5b568 period: 1 addr: 0 > > 147224656611284 0x4b10 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > >
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
On Fri, 17 Jul 2020 14:40:53 -0300 Arnaldo Carvalho de Melo wrote: > Say you use: > > ^C[root@ssdandy ~]# cyclictest --smp -um -p95 > # /dev/cpu_dma_latency set to 0us > policy: fifo: loadavg: 0.05 0.03 0.06 2/409 29072 > > T: 0 (29065) P:95 I:1000 C:518 Min: 2 Act:2 Avg:2 Max: > 6 > T: 1 (29066) P:95 I:1500 C:343 Min: 2 Act:2 Avg:2 Max: > 5 > T: 2 (29067) P:95 I:2000 C:256 Min: 2 Act:2 Avg:2 Max: > 7 > T: 3 (29068) P:95 I:2500 C:203 Min: 2 Act:2 Avg:2 Max: > 5 > T: 4 (29069) P:95 I:3000 C:168 Min: 2 Act:2 Avg:3 Max: > 6 > T: 5 (29070) P:95 I:3500 C:143 Min: 2 Act:2 Avg:2 Max: > 6 > T: 6 (29071) P:95 I:4000 C:124 Min: 2 Act:2 Avg:2 Max: > 7 > T: 7 (29072) P:95 I:4500 C:110 Min: 2 Act:2 Avg:2 Max: > > Then we do: > > # pidof cyclictest > 29064 > # > > If we use: > > [root@ssdandy ~]# perf record --pid 29064 > ^C[ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.052 MB perf.data (866 samples) ] > > [root@ssdandy ~]# perf report --task > # pid tid ppid comm > 00 -1 |swapper > 2906429064 -1 |cyclictest > 2906429065 -1 |cyclictest > 2906429066 -1 |cyclictest > 2906429067 -1 |cyclictest > 2906429068 -1 |cyclictest > 2906429069 -1 |cyclictest > 2906429070 -1 |cyclictest > 2906429071 -1 |cyclictest > 2906429072 -1 |cyclictest > [root@ssdandy ~]# "ftrace" is inside the kernel. But you could specify all those PIDs in the set_ftrace_pid and set_event_pid files, and they will be traced. If you want to trace the children of those PIDs, you would need to set the options function function-fork and event-fork respectively. And then any time a process with a pid in the set_ftrace_pid or set_event_pid file forks, its child will also be added to that file and it too will be traced. If the fork options are set, then when a task exits, its pid will be removed from the file. echo 1 > options/function-fork echo 1 > options/event-fork > > If we are interested only on the thread running on CPU3 we can do: > > [root@ssdandy ~]# perf report --task > # pid tid ppid comm > 00 -1 |swapper > 2906429064 -1 |cyclictest > 2906429068 -1 |cyclictest > [root@ssdandy ~]# > > The first 29064 is just to have info on who created 29068, i.e.: > > Its just the synthesized info for 29068 creator: > > [root@ssdandy ~]# perf report -D | grep -w 29064/29064 > 0 0x4690 [0x30]: PERF_RECORD_COMM: cyclictest:29064/29064 > 0 0x46c0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x40(0xa000) @ 0 fd:00 > 136246288 0]: r-xp /usr/bin/cyclictest > 0 0x4730 [0x80]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f505000(0x15000) @ 0 > fd:00 201479398 0]: r-xp /usr/lib64/libgcc_s-4.8.5-20150702.so.1 > 0 0x47b0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f71b000(0x1c3000) @ 0 > fd:00 201334455 0]: r-xp /usr/lib64/libc-2.17.so > 0 0x4820 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990fae9000(0xa000) @ 0 > fd:00 204604380 0]: r-xp /usr/lib64/libnuma.so.1.0.0 > 0 0x4898 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990fcf5000(0x17000) @ 0 > fd:00 201335636 0]: r-xp /usr/lib64/libpthread-2.17.so > 0 0x4910 [0x78]: PERF_RECORD_MMAP2 29064/29064: [0x7f990ff11000(0x7000) @ 0 > fd:00 201335640 0]: r-xp /usr/lib64/librt-2.17.so > 0 0x4988 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x7f9910119000(0x22000) @ 0 > fd:00 203595299 0]: r-xp /usr/lib64/ld-2.17.so > 0 0x49f8 [0x60]: PERF_RECORD_MMAP2 29064/29064: [0x7fff0b52d000(0x2000) @ 0 > 00:00 0 0]: r-xp [vdso] > 0 0x4a58 [0x68]: PERF_RECORD_MMAP2 29064/29064: [0xff60(0x1000) @ > 0 00:00 0 0]: r-xp [vsyscall] > [root@ssdandy ~]# > > No PERF_RECORD_SAMPLEs. > > Those are only for: > > [root@ssdandy ~]# perf report -D | grep PERF_RECORD_SAMPLE | head -5 > 147224656598815 0x4ac0 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > 0xa8e5b568 period: 1 addr: 0 > 147224656606270 0x4ae8 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > 0xa8e5b568 period: 1 addr: 0 > 147224656611284 0x4b10 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > 0xa8e5b568 period: 5 addr: 0 > 147224656616225 0x4b38 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > 0xa8e5b568 period: 35 addr: 0 > 147224656621152 0x4b60 [0x28]: PERF_RECORD_SAMPLE(IP, 0x4001): 29064/29068: > 0xa8e5b568 period: 252 addr: 0 > [root@ssdandy ~]# > > [root@ssdandy ~]# perf report -D | grep PERF_RECORD_SAMPLE | cut -d: -f3 | > sort -u > 29064/29068 The above can somewhat be done in trace-cmd, but not fully. But that's all userspace commands, nothing with the kernel. > [root@ssdandy ~]# > > Is there a way in ftrace to ask for a pid
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
Em Fri, Jul 17, 2020 at 01:01:24PM -0400, Steven Rostedt escreveu: > On Fri, 17 Jul 2020 21:26:50 +0800 > Changbin Du wrote: > > > On Thu, Jul 16, 2020 at 12:36:30PM -0300, Arnaldo Carvalho de Melo wrote: > > > Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu: > > > > This allows us to trace single thread instead of the whole process. > > > > > > > > Signed-off-by: Changbin Du > > > > --- > > > > tools/perf/Documentation/perf-ftrace.txt | 4 > > > > tools/perf/builtin-ftrace.c | 2 ++ > > > > 2 files changed, 6 insertions(+) > > > > > > > > diff --git a/tools/perf/Documentation/perf-ftrace.txt > > > > b/tools/perf/Documentation/perf-ftrace.txt > > > > index d79560dea19f..e204bf6d50d8 100644 > > > > --- a/tools/perf/Documentation/perf-ftrace.txt > > > > +++ b/tools/perf/Documentation/perf-ftrace.txt > > > > @@ -38,6 +38,10 @@ OPTIONS > > > > --pid=:: > > > > Trace on existing process id (comma separated list). > > > > > > > > +-t:: > > > > +--tid=:: > > > > + Trace on existing thread id (comma separated list). > > > > + > > > Humm, I just tried: > > > [root@five ~]# yes > /dev/null & > > > [1] 18265 > > > [root@five ~]# perf ftrace --tid 18265 > > > ^C[root@five ~]# > > > After waiting for a while, nothing, what am I doing wrong? > > I got it wrong. Currently ftrace only can filter by pid. If the pid is not > > the main thread it won't work. > Wait what? > The "pid" for ftrace is really a "task id" which is a thread id. > [root@bxtest ~]# yes > /dev/null & > [1] 6799 > [root@bxtest ~]# trace-cmd record -e all -P 6799 > Hit Ctrl^C to stop recording > ^CCPU 0: 3573031 events lost > CPU0 data recorded at offset=0x838000 > 627675136 bytes in size > CPU1 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU2 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU3 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU4 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU5 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU6 data recorded at offset=0x25ed1000 > 0 bytes in size > CPU7 data recorded at offset=0x25ed1000 > 0 bytes in size > [root@bxtest ~]# trace-cmd report | head > CPU 1 is empty > CPU 2 is empty > CPU 3 is empty > CPU 4 is empty > CPU 5 is empty > CPU 6 is empty > CPU 7 is empty > cpus=8 > yes-6799 [000] 67825.580902: sys_exit: NR 1 = 8192 > yes-6799 [000] 67825.580903: sys_exit_write: 0x2000 > > > What's different about tid vs pid? Say you use: ^C[root@ssdandy ~]# cyclictest --smp -um -p95 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.05 0.03 0.06 2/409 29072 T: 0 (29065) P:95 I:1000 C:518 Min: 2 Act:2 Avg:2 Max: 6 T: 1 (29066) P:95 I:1500 C:343 Min: 2 Act:2 Avg:2 Max: 5 T: 2 (29067) P:95 I:2000 C:256 Min: 2 Act:2 Avg:2 Max: 7 T: 3 (29068) P:95 I:2500 C:203 Min: 2 Act:2 Avg:2 Max: 5 T: 4 (29069) P:95 I:3000 C:168 Min: 2 Act:2 Avg:3 Max: 6 T: 5 (29070) P:95 I:3500 C:143 Min: 2 Act:2 Avg:2 Max: 6 T: 6 (29071) P:95 I:4000 C:124 Min: 2 Act:2 Avg:2 Max: 7 T: 7 (29072) P:95 I:4500 C:110 Min: 2 Act:2 Avg:2 Max: Then we do: # pidof cyclictest 29064 # If we use: [root@ssdandy ~]# perf record --pid 29064 ^C[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.052 MB perf.data (866 samples) ] [root@ssdandy ~]# perf report --task # pid tid ppid comm 00 -1 |swapper 2906429064 -1 |cyclictest 2906429065 -1 |cyclictest 2906429066 -1 |cyclictest 2906429067 -1 |cyclictest 2906429068 -1 |cyclictest 2906429069 -1 |cyclictest 2906429070 -1 |cyclictest 2906429071 -1 |cyclictest 2906429072 -1 |cyclictest [root@ssdandy ~]# If we are interested only on the thread running on CPU3 we can do: [root@ssdandy ~]# perf report --task # pid tid ppid comm 00 -1 |swapper 2906429064 -1 |cyclictest 2906429068 -1 |cyclictest [root@ssdandy ~]# The first 29064 is just to have info on who created 29068, i.e.: Its just the synthesized info for 29068 creator: [root@ssdandy ~]# perf report -D | grep -w 29064/29064 0 0x4690 [0x30]: PERF_RECORD_COMM: cyclictest:29064/29064 0 0x46c0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x40(0xa000) @ 0 fd:00 136246288 0]: r-xp /usr/bin/cyclictest 0 0x4730 [0x80]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f505000(0x15000) @ 0 fd:00 201479398 0]: r-xp /usr/lib64/libgcc_s-4.8.5-20150702.so.1 0 0x47b0 [0x70]: PERF_RECORD_MMAP2 29064/29064: [0x7f990f71b000(0x1c3000) @ 0 fd:00 201334455 0]: r-xp /usr/lib64/libc-2.17.so 0 0x4820 [0x78]: PERF_RECORD_MMAP2 29064/29064:
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
On Fri, 17 Jul 2020 21:26:50 +0800 Changbin Du wrote: > On Thu, Jul 16, 2020 at 12:36:30PM -0300, Arnaldo Carvalho de Melo wrote: > > Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu: > > > This allows us to trace single thread instead of the whole process. > > > > > > Signed-off-by: Changbin Du > > > --- > > > tools/perf/Documentation/perf-ftrace.txt | 4 > > > tools/perf/builtin-ftrace.c | 2 ++ > > > 2 files changed, 6 insertions(+) > > > > > > diff --git a/tools/perf/Documentation/perf-ftrace.txt > > > b/tools/perf/Documentation/perf-ftrace.txt > > > index d79560dea19f..e204bf6d50d8 100644 > > > --- a/tools/perf/Documentation/perf-ftrace.txt > > > +++ b/tools/perf/Documentation/perf-ftrace.txt > > > @@ -38,6 +38,10 @@ OPTIONS > > > --pid=:: > > > Trace on existing process id (comma separated list). > > > > > > +-t:: > > > +--tid=:: > > > + Trace on existing thread id (comma separated list). > > > + > > > > > > Humm, I just tried: > > > > [root@five ~]# yes > /dev/null & > > [1] 18265 > > [root@five ~]# perf ftrace --tid 18265 > > ^C[root@five ~]# > > > > After waiting for a while, nothing, what am I doing wrong? > > > I got it wrong. Currently ftrace only can filter by pid. If the pid is not > the main thread it won't work. Wait what? The "pid" for ftrace is really a "task id" which is a thread id. [root@bxtest ~]# yes > /dev/null & [1] 6799 [root@bxtest ~]# trace-cmd record -e all -P 6799 Hit Ctrl^C to stop recording ^CCPU 0: 3573031 events lost CPU0 data recorded at offset=0x838000 627675136 bytes in size CPU1 data recorded at offset=0x25ed1000 0 bytes in size CPU2 data recorded at offset=0x25ed1000 0 bytes in size CPU3 data recorded at offset=0x25ed1000 0 bytes in size CPU4 data recorded at offset=0x25ed1000 0 bytes in size CPU5 data recorded at offset=0x25ed1000 0 bytes in size CPU6 data recorded at offset=0x25ed1000 0 bytes in size CPU7 data recorded at offset=0x25ed1000 0 bytes in size [root@bxtest ~]# trace-cmd report | head CPU 1 is empty CPU 2 is empty CPU 3 is empty CPU 4 is empty CPU 5 is empty CPU 6 is empty CPU 7 is empty cpus=8 yes-6799 [000] 67825.580902: sys_exit: NR 1 = 8192 yes-6799 [000] 67825.580903: sys_exit_write: 0x2000 What's different about tid vs pid? -- Steve > > So this patch makes no sense. will drop this. >
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
Em Fri, Jul 17, 2020 at 09:26:50PM +0800, Changbin Du escreveu: > On Thu, Jul 16, 2020 at 12:36:30PM -0300, Arnaldo Carvalho de Melo wrote: > > Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu: > > > +++ b/tools/perf/Documentation/perf-ftrace.txt > > > @@ -38,6 +38,10 @@ OPTIONS > > > --pid=:: > > > Trace on existing process id (comma separated list). > > > +-t:: > > > +--tid=:: > > > + Trace on existing thread id (comma separated list). > > Humm, I just tried: > > [root@five ~]# yes > /dev/null & > > [1] 18265 > > [root@five ~]# perf ftrace --tid 18265 > > ^C[root@five ~]# > > After waiting for a while, nothing, what am I doing wrong? > I got it wrong. Currently ftrace only can filter by pid. If the pid is not > the main thread it won't work. > So this patch makes no sense. will drop this. I think you could alternatively keep it but inform the user that this target, available to other perf commands, isn't available for ftrace as it doesn't support it, this way when the user goes from: perf trace|top|record|script|report --tid 1234 to: perf ftrace --tid 1234 He gets a message like: ERROR: 'ftrace' doesn't support the --tid target, try with --pid And that would be more useful, provides an explanation as why that target can't be used and suggests an alternative. - Arnaldo
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
On Thu, Jul 16, 2020 at 12:36:30PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu:
> > This allows us to trace single thread instead of the whole process.
> >
> > Signed-off-by: Changbin Du
> > ---
> > tools/perf/Documentation/perf-ftrace.txt | 4
> > tools/perf/builtin-ftrace.c | 2 ++
> > 2 files changed, 6 insertions(+)
> >
> > diff --git a/tools/perf/Documentation/perf-ftrace.txt
> > b/tools/perf/Documentation/perf-ftrace.txt
> > index d79560dea19f..e204bf6d50d8 100644
> > --- a/tools/perf/Documentation/perf-ftrace.txt
> > +++ b/tools/perf/Documentation/perf-ftrace.txt
> > @@ -38,6 +38,10 @@ OPTIONS
> > --pid=::
> > Trace on existing process id (comma separated list).
> >
> > +-t::
> > +--tid=::
> > + Trace on existing thread id (comma separated list).
> > +
>
>
> Humm, I just tried:
>
> [root@five ~]# yes > /dev/null &
> [1] 18265
> [root@five ~]# perf ftrace --tid 18265
> ^C[root@five ~]#
>
> After waiting for a while, nothing, what am I doing wrong?
>
I got it wrong. Currently ftrace only can filter by pid. If the pid is not
the main thread it won't work.
So this patch makes no sense. will drop this.
> - Arnaldo
>
>
> > -a::
> > --all-cpus::
> > Force system-wide collection. Scripts run without a
> > diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
> > index 244cc8e6bd60..1188b82c6541 100644
> > --- a/tools/perf/builtin-ftrace.c
> > +++ b/tools/perf/builtin-ftrace.c
> > @@ -515,6 +515,8 @@ int cmd_ftrace(int argc, const char **argv)
> > "Show available functions to filter"),
> > OPT_STRING('p', "pid", , "pid",
> >"trace on existing process id"),
> > + OPT_STRING('t', "tid", , "tid",
> > + "trace on existing thread id (exclusive to --pid)"),
> > OPT_INCR('v', "verbose", ,
> > "be more verbose"),
> > OPT_BOOLEAN('a', "all-cpus", _wide,
> > --
> > 2.25.1
> >
>
> --
>
> - Arnaldo
--
Cheers,
Changbin Du
Re: [PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
Em Sat, Jul 11, 2020 at 08:40:21PM +0800, Changbin Du escreveu:
> This allows us to trace single thread instead of the whole process.
>
> Signed-off-by: Changbin Du
> ---
> tools/perf/Documentation/perf-ftrace.txt | 4
> tools/perf/builtin-ftrace.c | 2 ++
> 2 files changed, 6 insertions(+)
>
> diff --git a/tools/perf/Documentation/perf-ftrace.txt
> b/tools/perf/Documentation/perf-ftrace.txt
> index d79560dea19f..e204bf6d50d8 100644
> --- a/tools/perf/Documentation/perf-ftrace.txt
> +++ b/tools/perf/Documentation/perf-ftrace.txt
> @@ -38,6 +38,10 @@ OPTIONS
> --pid=::
> Trace on existing process id (comma separated list).
>
> +-t::
> +--tid=::
> + Trace on existing thread id (comma separated list).
> +
Humm, I just tried:
[root@five ~]# yes > /dev/null &
[1] 18265
[root@five ~]# perf ftrace --tid 18265
^C[root@five ~]#
After waiting for a while, nothing, what am I doing wrong?
- Arnaldo
> -a::
> --all-cpus::
> Force system-wide collection. Scripts run without a
> diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
> index 244cc8e6bd60..1188b82c6541 100644
> --- a/tools/perf/builtin-ftrace.c
> +++ b/tools/perf/builtin-ftrace.c
> @@ -515,6 +515,8 @@ int cmd_ftrace(int argc, const char **argv)
> "Show available functions to filter"),
> OPT_STRING('p', "pid", , "pid",
> "trace on existing process id"),
> + OPT_STRING('t', "tid", , "tid",
> +"trace on existing thread id (exclusive to --pid)"),
> OPT_INCR('v', "verbose", ,
>"be more verbose"),
> OPT_BOOLEAN('a', "all-cpus", _wide,
> --
> 2.25.1
>
--
- Arnaldo
[PATCH v5 03/17] perf ftrace: add option -t/--tid to filter by thread id
This allows us to trace single thread instead of the whole process.
Signed-off-by: Changbin Du
---
tools/perf/Documentation/perf-ftrace.txt | 4
tools/perf/builtin-ftrace.c | 2 ++
2 files changed, 6 insertions(+)
diff --git a/tools/perf/Documentation/perf-ftrace.txt
b/tools/perf/Documentation/perf-ftrace.txt
index d79560dea19f..e204bf6d50d8 100644
--- a/tools/perf/Documentation/perf-ftrace.txt
+++ b/tools/perf/Documentation/perf-ftrace.txt
@@ -38,6 +38,10 @@ OPTIONS
--pid=::
Trace on existing process id (comma separated list).
+-t::
+--tid=::
+ Trace on existing thread id (comma separated list).
+
-a::
--all-cpus::
Force system-wide collection. Scripts run without a
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index 244cc8e6bd60..1188b82c6541 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -515,6 +515,8 @@ int cmd_ftrace(int argc, const char **argv)
"Show available functions to filter"),
OPT_STRING('p', "pid", , "pid",
"trace on existing process id"),
+ OPT_STRING('t', "tid", , "tid",
+ "trace on existing thread id (exclusive to --pid)"),
OPT_INCR('v', "verbose", ,
"be more verbose"),
OPT_BOOLEAN('a', "all-cpus", _wide,
--
2.25.1
