On Sun, Jun 07, 2026 at 11:13:32AM -0700, Alexei Starovoitov wrote:
> On Sat Jun 6, 2026 at 5:39 AM PDT, Jiri Olsa wrote:
> > Adding benchmark test that attaches to (almost) all allowed tracing
> > functions and display attach/detach times.
> >
> > # ./test_progs -t tracing_multi_bench_attach -v
> > bpf_testmod.ko is already unloaded.
> > Loading bpf_testmod.ko...
> > Successfully loaded bpf_testmod.ko.
> > serial_test_tracing_multi_bench_attach:PASS:btf__load_vmlinux_btf 0 nsec
> >
> > serial_test_tracing_multi_bench_attach:PASS:tracing_multi_bench__open_and_load
> > 0 nsec
> > serial_test_tracing_multi_bench_attach:PASS:get_syms 0 nsec
> >
> > serial_test_tracing_multi_bench_attach:PASS:bpf_program__attach_tracing_multi
> > 0 nsec
> > serial_test_tracing_multi_bench_attach: found 51186 functions
> > serial_test_tracing_multi_bench_attach: attached in 1.295s
> > serial_test_tracing_multi_bench_attach: detached in 0.243s
>
> ...
>
> > + if (!ASSERT_OK(bpf_get_ksyms(&ksyms, true), "get_syms"))
> > + goto cleanup;
> > +
> > + /* Get all ftrace 'safe' symbols.. */
> > + for (i = 0; i < ksyms->filtered_cnt; i++) {
> > + if (!tsearch(&ksyms->filtered_syms[i], &root, compare)) {
> > + ASSERT_FAIL("tsearch failed");
> > + goto cleanup;
> > + }
> > + }
> > +
> > + /* ..and filter them through BTF and btf_type_is_traceable_func. */
> > + nr = btf__type_cnt(btf);
> > + for (type_id = 1; type_id < nr; type_id++) {
> > + const struct btf_type *type;
> > + const char *str;
> > +
> > + type = btf__type_by_id(btf, type_id);
> > + if (!type)
> > + break;
> > +
> > + if (BTF_INFO_KIND(type->info) != BTF_KIND_FUNC)
> > + continue;
> > +
> > + str = btf__name_by_offset(btf, type->name_off);
> > + if (!str)
> > + break;
> > +
> > + if (!tfind(&str, &root, compare))
> > + continue;
> > +
> > + if (!btf_type_is_traceable_func(btf, type))
> > + continue;
> > +
> > + err = libbpf_ensure_mem((void **) &ids, &cap, sizeof(*ids), cnt
> > + 1);
> > + if (err)
> > + goto cleanup;
> > +
> > + ids[cnt++] = type_id;
> > + }
>
> This filtering wasn't enough.
> I've added removal of duplicates here while applying:
>
> + /*
> + * Collect names that are not unique in kallsyms. The kernel resolves
> a
> + * tracing-multi BTF id to an address with kallsyms_lookup_name(),
> which
> + * returns the first symbol of that name. For a duplicate name that
> may
> + * be a different (non-ftrace-able) instance than the ftrace-able one
> in
> + * available_filter_functions, so attaching to it by BTF id fails with
> + * -ENOENT (e.g. t_start/t_next/t_stop). ksyms->syms is sorted by
> name,
> + * so equal names are adjacent.
> + */
> + for (i = 1; i < ksyms->sym_cnt; i++) {
> + if (strcmp(ksyms->syms[i].name, ksyms->syms[i - 1].name))
> + continue;
> + if (!tsearch(&ksyms->syms[i].name, &dups, compare)) {
> + ASSERT_FAIL("tsearch failed");
> + goto cleanup;
> + }
> + }
>
>
> + /* Skip names that are not unique in kallsyms, see above. */
> + if (tfind(&str, &dups, compare))
> + continue;
>
>
> As claude explains it:
> ----
> 1. The kernel attaches tracing_multi by BTF id. To get an address it resolves
> the BTF function name via kallsyms_lookup_name(tname) and requires
> ftrace_location(addr) — kernel/bpf/verifier.c:19380:
> addr = kallsyms_lookup_name(tname);
> ...
> if (!addr || !ftrace_location(addr))
> return -ENOENT;
> 2. t_start/t_next/t_stop each have 5 instances in this kernel. Only one is
> ftrace-able — the copies in kernel/trace/* are built notrace (ftrace's
> Makefile
> strips -pg), so only the unrelated copy is in available_filter_functions:
> 3. kallsyms_lookup_name() returns the lowest-address instance among equal
> names
> (exact strcmp, lowest seq). That instance has no fentry → ftrace_location()
> returns 0 → -ENOENT, which aborts the whole all-or-nothing bench attach.
>
> Why the bench includes them: it intersects BTF FUNC names with
> available_filter_functions names. Since some t_start is ftrace-able, the name
> passes the filter — but the kernel resolves the wrong (non-ftrace-able)
> t_start. The author's kernel apparently had the ftrace-able copy at the lowest
> address, so it passed there.
>
> This is a pre-existing limitation, not multi-specific: single fentry attach by
> BTF id uses the same kallsyms_lookup_name(tname) path (verifier.c:19120) — you
> can't reliably fentry-attach to any duplicate-named function on this kernel
> either.
> ----
strange I never triggered that.. but makes sense
>
> Maybe we should adjust bpf_get_ksyms() instead. Not sure.
the only other user is test_kprobe_multi_bench_attach which does
not care about this, so I think at this point keeping this in the
serial_test_tracing_multi_bench_attach is enough for now
thanks,
jirka
