Re: perf tools: Call graph from Intel BTS
On 11/07/2014 6:36 p.m., Peter Zijlstra wrote: On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote: Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later. Currently that driver does not allow kernel tracing. Last time I tried it with that restriction removed, it would lose data on larger workloads. Generally, any loss of data ruins the call graph. For me it does not make sense to try to support it, since I am primarily trying to pave the way for the Intel PT driver. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
On 11/07/2014 6:18 p.m., Arnaldo Carvalho de Melo wrote: Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu: There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner. We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like: commit 244c87b15b124914827f3ce28d8e70c8d147c9d0 Author: Adrian Hunter Date: Wed Jun 11 09:33:17 2014 +0300 perf tools: Fix the value used for unknown pids The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups. Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence. Signed-off-by: Adrian Hunter But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch: [acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0 tools/perf/util/session.c: thread = machine__findnew_thread(>machines.host, 0, 0); That's the idle thread. tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0); tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3); Those are valid pids for that test. [acme@ssdandy linux]$ So I think that one way to reduce the size of that branch is to do just that: start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled. You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat. Sounds good, thanks! It is currently based on tip/perf/core from a few days ago, so the current patches should be mostly ok. I will make a selection and check them again. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote: > Hi > > Alexander Shishkin is working on the Intel PT driver for perf > and has included a driver for Intel BTS. I have taken that and There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu: > There are many perf tools patches and it would be helpful to start > considering how to get them into mainline. Many need to wait for > the driver, but others could be taken sooner. We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like: commit 244c87b15b124914827f3ce28d8e70c8d147c9d0 Author: Adrian Hunter Date: Wed Jun 11 09:33:17 2014 +0300 perf tools: Fix the value used for unknown pids The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups. Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence. Signed-off-by: Adrian Hunter But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch: [acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0 tools/perf/util/session.c: thread = machine__findnew_thread(>machines.host, 0, 0); tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0); tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3); [acme@ssdandy linux]$ So I think that one way to reduce the size of that branch is to do just that: start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled. You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat. - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
perf tools: Call graph from Intel BTS
Hi Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and extended perf tools to enable the creation of context-sensitive call graphs. A screen shot from an example call graph from tracing "ls" is shown here: http://git.infradead.org/~ahunter/call-graph-bts-ls.png The patches are here: http://git.infradead.org/users/ahunter/linux-perf.git There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner. The perf tools python API has been extended to facilitate exporting to databases, a script is provided to export to PostgreSQL, and another script reads the database and displays the call graph. The Intel BTS data provides branches (address from / to) but to create the call graph the type of instruction (call, return etc) is needed. That is found by reading the object code, but for the kernel that requires an accurate image. A copy of /proc/kcore is needed but it must be made under the same conditions as the trace. That has proven sufficiently awkward to warrant creating a script to do it. Here is how the "ls" call graph was created: $ sudo bash -c 'echo 0 > /proc/sys/kernel/kptr_restrict' $ sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid' $ sudo bash -c 'chmod go+rx /sys/kernel/debug' $ ~/libexec/perf-core/perf-with-kcore record bts_ls --per-thread -e intel_bts// -- ls Recording Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf record -o bts_ls/perf.data --per-thread -e intel_bts// -- ls CREDITSdrivers ipcMAINTAINERS modules.order REPORTING-BUGS signing_key.priv tools vmlinux.o arch cryptofirmware Kconfig Makefile Module.symvers samples signing_key.x509 usr x509.genkey block fsincludekernelmm net scripts sound virt bts_ls COPYING Documentation init Kbuild lib modules.builtin READMEsecuritySystem.map vmlinux [ perf record: Woken up 100 times to write data ] [ perf record: Captured and wrote 6.945 MB bts_ls/perf.data ] Copying kcore Done $ ~/libexec/perf-core/perf-with-kcore script bts_ls -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf script -i bts_ls/perf.data --kallsyms=bts_ls/kcore_dir/kallsyms -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls 2014-07-11 15:13:38.313702 Creating database... 2014-07-11 15:13:39.150277 Writing to intermediate files... 2014-07-11 15:13:40.182816 Copying to database... 2014-07-11 15:13:41.776815 Removing intermediate files... 2014-07-11 15:13:41.783443 Adding primary keys 2014-07-11 15:13:44.290786 Adding foreign keys 2014-07-11 15:13:44.974865 Done $ ~/libexec/perf-core/scripts/python/call-graph-from-postgresql.py bts_ls $ Regards Adrian -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
perf tools: Call graph from Intel BTS
Hi Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and extended perf tools to enable the creation of context-sensitive call graphs. A screen shot from an example call graph from tracing ls is shown here: http://git.infradead.org/~ahunter/call-graph-bts-ls.png The patches are here: http://git.infradead.org/users/ahunter/linux-perf.git There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner. The perf tools python API has been extended to facilitate exporting to databases, a script is provided to export to PostgreSQL, and another script reads the database and displays the call graph. The Intel BTS data provides branches (address from / to) but to create the call graph the type of instruction (call, return etc) is needed. That is found by reading the object code, but for the kernel that requires an accurate image. A copy of /proc/kcore is needed but it must be made under the same conditions as the trace. That has proven sufficiently awkward to warrant creating a script to do it. Here is how the ls call graph was created: $ sudo bash -c 'echo 0 /proc/sys/kernel/kptr_restrict' $ sudo bash -c 'echo -1 /proc/sys/kernel/perf_event_paranoid' $ sudo bash -c 'chmod go+rx /sys/kernel/debug' $ ~/libexec/perf-core/perf-with-kcore record bts_ls --per-thread -e intel_bts// -- ls Recording Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf record -o bts_ls/perf.data --per-thread -e intel_bts// -- ls CREDITSdrivers ipcMAINTAINERS modules.order REPORTING-BUGS signing_key.priv tools vmlinux.o arch cryptofirmware Kconfig Makefile Module.symvers samples signing_key.x509 usr x509.genkey block fsincludekernelmm net scripts sound virt bts_ls COPYING Documentation init Kbuild lib modules.builtin READMEsecuritySystem.map vmlinux [ perf record: Woken up 100 times to write data ] [ perf record: Captured and wrote 6.945 MB bts_ls/perf.data ] Copying kcore Done $ ~/libexec/perf-core/perf-with-kcore script bts_ls -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf script -i bts_ls/perf.data --kallsyms=bts_ls/kcore_dir/kallsyms -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls 2014-07-11 15:13:38.313702 Creating database... 2014-07-11 15:13:39.150277 Writing to intermediate files... 2014-07-11 15:13:40.182816 Copying to database... 2014-07-11 15:13:41.776815 Removing intermediate files... 2014-07-11 15:13:41.783443 Adding primary keys 2014-07-11 15:13:44.290786 Adding foreign keys 2014-07-11 15:13:44.974865 Done $ ~/libexec/perf-core/scripts/python/call-graph-from-postgresql.py bts_ls $ Regards Adrian -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu: There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner. We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like: commit 244c87b15b124914827f3ce28d8e70c8d147c9d0 Author: Adrian Hunter adrian.hun...@intel.com Date: Wed Jun 11 09:33:17 2014 +0300 perf tools: Fix the value used for unknown pids The value used for unknown pids cannot be zero because that is used by the idle task. Use -1 instead. Also handle the unknown pid case when creating map groups. Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence. Signed-off-by: Adrian Hunter adrian.hun...@intel.com But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch: [acme@ssdandy linux]$ find tools -name *.[ch] | xargs grep machine__findnew_thread | grep 0 tools/perf/util/session.c: thread = machine__findnew_thread(session-machines.host, 0, 0); tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0); tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3); [acme@ssdandy linux]$ So I think that one way to reduce the size of that branch is to do just that: start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled. You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat. - Arnaldo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote: Hi Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
On 11/07/2014 6:36 p.m., Peter Zijlstra wrote: On Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter wrote: Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later. Currently that driver does not allow kernel tracing. Last time I tried it with that restriction removed, it would lose data on larger workloads. Generally, any loss of data ruins the call graph. For me it does not make sense to try to support it, since I am primarily trying to pave the way for the Intel PT driver. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: perf tools: Call graph from Intel BTS
On 11/07/2014 6:18 p.m., Arnaldo Carvalho de Melo wrote: Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu: There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner. We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like: commit 244c87b15b124914827f3ce28d8e70c8d147c9d0 Author: Adrian Hunter adrian.hun...@intel.com Date: Wed Jun 11 09:33:17 2014 +0300 perf tools: Fix the value used for unknown pids The value used for unknown pids cannot be zero because that is used by the idle task. Use -1 instead. Also handle the unknown pid case when creating map groups. Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence. Signed-off-by: Adrian Hunter adrian.hun...@intel.com But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch: [acme@ssdandy linux]$ find tools -name *.[ch] | xargs grep machine__findnew_thread | grep 0 tools/perf/util/session.c: thread = machine__findnew_thread(session-machines.host, 0, 0); That's the idle thread. tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0); tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3); Those are valid pids for that test. [acme@ssdandy linux]$ So I think that one way to reduce the size of that branch is to do just that: start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled. You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat. Sounds good, thanks! It is currently based on tip/perf/core from a few days ago, so the current patches should be mostly ok. I will make a selection and check them again. -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
