alloc_retstack_tasklist() assigns a shadow stack to every thread when fgraph is enabled. It did so FTRACE_RETSTACK_ALLOC_SIZE (32) tasks at a time, and since for_each_process_thread() has no cursor, every sweep restarted from init_task and re-walked the tasks already served, making the ftrace_graph_active 0 -> 1 transition quadratic in the thread count. On a 60-core Sapphire Rapids machine with 400000 idle threads it took 227 s, inside a single bpf() syscall for the kprobe_multi case.
- 1/2 allocates inline with GFP_NOWAIT, which removes the batching: a single sweep serves every task, 227 s -> 0.092 s. It keeps the open-coded rcu_read_lock() for ease of backport. - 2/2 is the cleanup: convert that same walk to scoped_guard(rcu). No functional change, and not needed by stable. Both were tested on bare metal and in a VM, including forcing the GFP_NOWAIT failure path with failslab injection. v1: https://lore.kernel.org/all/[email protected]/ v2: https://lore.kernel.org/all/[email protected]/ v3: https://lore.kernel.org/all/[email protected]/ Changes since v3: - split into two patches: the fix keeps goto-based unwinding so it backports cleanly, and scoped_guard(rcu) moves to a separate cleanup, per Masami and Steve: https://lore.kernel.org/all/[email protected]/ - in the cleanup, -EAGAIN now exits via the existing free label rather than returning from inside the guard, so there is no longer a path that skips the free loop Changes since v2: - update the stale alloc_retstack_tasklist() comment, which still described the old batch-of-32 behaviour (bot+bpf-ci) Changes since v1: - replace both v1 patches with Peter's inline GFP_NOWAIT approach, which removes the quadratic behaviour rather than dividing it by a constant. With a single sweep there is no retry loop left, so the v1 cond_resched() patch has nothing to attach to and is dropped. Vineet Gupta (2): tracing: fgraph: allocate shadow stacks inline with GFP_NOWAIT tracing: fgraph: use scoped_guard() for the task list walk kernel/trace/fgraph.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222 -- 2.53.0-Meta
