This patch is based off of: https://lore.kernel.org/linux-trace-kernel/20250820180338.701352...@kernel.org And requires these patches to be enabled: https://lore.kernel.org/linux-trace-kernel/20250820190546.172023...@kernel.org/
To run this series, you can checkout this repo that has this series as well as the above: git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git unwind/perf-test This series implements the perf interface to use deferred user space stack tracing. The first 4 patches implement the kernel side of perf to do the deferred stack tracing, and the last 4 patches implement the perf user space side to read this new interface. Patch 1 adds a new API interface to the user unwinder logic to allow perf to get the current context cookie for it's task event tracing. Perf's task event tracing maps a single task per perf event buffer and it follows the task around, so it only needs to implement its own task_work to do the deferred stack trace. Because it can still suffer not knowing which user stack trace belongs to which kernel stack due to dropped events, having the cookie to create a unique identifier for each user space stack trace to know which kernel stack to append it to is useful. Patch 2 adds the per task deferred stack traces to perf. It adds a new event type called PERF_RECORD_CALLCHAIN_DEFERRED that is recorded when a task is about to go back to user space and happens in a location that pages may be faulted in. It also adds a new callchain context called PERF_CONTEXT_USER_DEFERRED that is used as a place holder in a kernel callchain to append the deferred user space stack trace to. Patch 3 adds the user stack trace context cookie in the kernel callchain right after the PERF_CONTEXT_USER_DEFERRED context so that the user space side can map the request to the deferred user space stack trace. Patch 4 adds support for the per CPU perf events that will allow the kernel to associate each of the per CPU perf event buffers to a single application. This is needed so that when a request for a deferred stack trace happens on a task that then migrates to another CPU, it will know which CPU buffer to use to record the stack trace on. It is possible to have more than one perf user tool running and a request made by one perf tool should have the deferred trace go to the same perf tool's perf CPU event buffer. A global list of all the descriptors representing each perf tool that is using deferred stack tracing is created to manage this. The last 4 patches implement the perf user space tooling side of this. Changes since v14: https://lore.kernel.org/linux-trace-kernel/20250718164119.089692...@kernel.org/ - Moved the clean up patches into their own series (mentioned at the beginning) - Added unwind_user_get_cookie() API to allow the task events to add cookies to differentiate which user stack belongs to which kernel stack in the event of dropped events. - Save the cookie in the kernel callchain right after the PERF_CONTEXT_USER_DEFERRED - Have the perf user space tooling match the cookies as well as the TID between the request and the user stack recording, to know which kernel stack gets the user space trace appended to it based on its context cookie. Josh Poimboeuf (1): perf: Support deferred user callchains Namhyung Kim (4): perf tools: Minimal CALLCHAIN_DEFERRED support perf record: Enable defer_callchain for user callchains perf script: Display PERF_RECORD_CALLCHAIN_DEFERRED perf tools: Merge deferred user callchains Steven Rostedt (3): unwind deferred: Add unwind_user_get_cookie() API perf: Have the deferred request record the user context cookie perf: Support deferred user callchains for per CPU events ---- include/linux/perf_event.h | 11 +- include/linux/unwind_deferred.h | 5 + include/uapi/linux/perf_event.h | 25 +- kernel/bpf/stackmap.c | 4 +- kernel/events/callchain.c | 14 +- kernel/events/core.c | 421 +++++++++++++++++++++++++++++- kernel/unwind/deferred.c | 21 ++ tools/include/uapi/linux/perf_event.h | 25 +- tools/lib/perf/include/perf/event.h | 8 + tools/perf/Documentation/perf-script.txt | 5 + tools/perf/builtin-script.c | 92 +++++++ tools/perf/util/callchain.c | 24 ++ tools/perf/util/callchain.h | 3 + tools/perf/util/event.c | 1 + tools/perf/util/evlist.c | 1 + tools/perf/util/evlist.h | 1 + tools/perf/util/evsel.c | 42 +++ tools/perf/util/evsel.h | 1 + tools/perf/util/machine.c | 1 + tools/perf/util/perf_event_attr_fprintf.c | 1 + tools/perf/util/sample.h | 4 +- tools/perf/util/session.c | 80 ++++++ tools/perf/util/tool.c | 2 + tools/perf/util/tool.h | 4 +- 24 files changed, 786 insertions(+), 10 deletions(-)
