On Sat Jun 6, 2026 at 5:39 AM PDT, Jiri Olsa wrote:
> Adding benchmark test that attaches to (almost) all allowed tracing
> functions and display attach/detach times.
>
> # ./test_progs -t tracing_multi_bench_attach -v
> bpf_testmod.ko is already unloaded.
> Loading bpf_testmod.ko...
> Successfully loaded bpf_testmod.ko.
> serial_test_tracing_multi_bench_attach:PASS:btf__load_vmlinux_btf 0 nsec
>
> serial_test_tracing_multi_bench_attach:PASS:tracing_multi_bench__open_and_load
> 0 nsec
> serial_test_tracing_multi_bench_attach:PASS:get_syms 0 nsec
>
> serial_test_tracing_multi_bench_attach:PASS:bpf_program__attach_tracing_multi
> 0 nsec
> serial_test_tracing_multi_bench_attach: found 51186 functions
> serial_test_tracing_multi_bench_attach: attached in 1.295s
> serial_test_tracing_multi_bench_attach: detached in 0.243s
...
> + if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms"))
> + goto cleanup;
> +
> + /* Get all ftrace 'safe' symbols.. */
> + for (i = 0; i < ksyms->filtered_cnt; i++) {
> + if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) {
> + ASSERT_FAIL("tsearch failed");
> + goto cleanup;
> + }
> + }
> +
> + /* ..and filter them through BTF and btf_type_is_traceable_func. */
> + nr = btf__type_cnt(btf);
> + for (type_id = 1; type_id < nr; type_id++) {
> + const struct btf_type *type;
> + const char *str;
> +
> + type = btf__type_by_id(btf, type_id);
> + if (!type)
> + break;
> +
> + if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
> + continue;
> +
> + str = btf__name_by_offset(btf, type->name_off);
> + if (!str)
> + break;
> +
> + if (!tfind(&str, &root, compare))
> + continue;
> +
> + if (!btf_type_is_traceable_func(btf, type))
> + continue;
> +
> + err = libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt
> + 1);
> + if (err)
> + goto cleanup;
> +
> + ids[cnt++] = type_id;
> + }
This filtering wasn't enough.
I've added removal of duplicates here while applying:
+ /*
+ * Collect names that are not unique in kallsyms. The kernel resolves a
+ * tracing-multi BTF id to an address with kallsyms_lookup_name(), which
+ * returns the first symbol of that name. For a duplicate name that may
+ * be a different (non-ftrace-able) instance than the ftrace-able one in
+ * available_filter_functions, so attaching to it by BTF id fails with
+ * -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by name,
+ * so equal names are adjacent.
+ */
+ for (i = 1; i < ksyms->sym_cnt; i++) {
+ if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name))
+ continue;
+ if (!tsearch(&ksyms->syms[i].name, &dups, compare)) {
+ ASSERT_FAIL("tsearch failed");
+ goto cleanup;
+ }
+ }
+ /* Skip names that are not unique in kallsyms, see above. */
+ if (tfind(&str, &dups, compare))
+ continue;
As claude explains it:
----
1. The kernel attaches tracing_multi by BTF id. To get an address it resolves
the BTF function name via kallsyms_lookup_name(tname) and requires
ftrace_location(addr) — kernel/bpf/verifier.c:19380:
addr = kallsyms_lookup_name(tname);
...
if (!addr || !ftrace_location(addr))
return -ENOENT;
2. t_start/t_next/t_stop each have 5 instances in this kernel. Only one is
ftrace-able — the copies in kernel/trace/* are built notrace (ftrace's Makefile
strips -pg), so only the unrelated copy is in available_filter_functions:
3. kallsyms_lookup_name() returns the lowest-address instance among equal names
(exact strcmp, lowest seq). That instance has no fentry → ftrace_location()
returns 0 → -ENOENT, which aborts the whole all-or-nothing bench attach.
Why the bench includes them: it intersects BTF FUNC names with
available_filter_functions names. Since some t_start is ftrace-able, the name
passes the filter — but the kernel resolves the wrong (non-ftrace-able)
t_start. The author's kernel apparently had the ftrace-able copy at the lowest
address, so it passed there.
This is a pre-existing limitation, not multi-specific: single fentry attach by
BTF id uses the same kallsyms_lookup_name(tname) path (verifier.c:19120) — you
can't reliably fentry-attach to any duplicate-named function on this kernel
either.
----
Maybe we should adjust bpf_get_ksyms() instead. Not sure.
After the fix in my gcc 15 kernel with KASAN:
serial_test_tracing_multi_bench_attach: found 49045 functions
serial_test_tracing_multi_bench_attach: attached in 4.115s
serial_test_tracing_multi_bench_attach: detached in 2.313s
