On 10/16/25 17:32, Song Liu wrote:
On Thu, Oct 16, 2025 at 3:56 AM Petr Mladek <[email protected]> wrote:
Added Song Liu and Steven into Cc because they are more familiar with
the ftrace trampolines.
On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote:
On 10/15/25 07:53, Petr Mladek wrote:
On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote:
Dear Upstream Livepatch team and Ubuntu Kernel team - I included you both in
this since the issue lies on the boundary between Ubuntu kernel and upstream.
According to official kernel documentation -
https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$
, section 7, Limitations -
1 - Kretprobes using the ftrace framework conflict with the patched functions.
2 - Kprobes in the original function are ignored when the code is redirected to
the new implementation.
Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable
kernel) machine, I have no problem applying Livepatch and then setting krpobes
and kretprobes on a patched function using bpftrace (or directly by coding a
BPF program with kprobe/kretprobe attachment)and can confirm both execute
without issues. Also the opposite works fine, running my krpobe and kretprobe
hooks doesn't prevent from livepatch to be applied successfully.
fentry/fexit probes do fail in in both directions - but this is expected
according to my understanding as coexistence of livepatching and ftrace based
BPF hooks are mutually exclusive until 6.0 based kernels
libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or resource busy
libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16
Please help me understand this contradiction about kprobes - is this because
the KPROBES are FTRACE based or any other reason ?
Heh, it seems that we have discussed this 10 years ago and I already
forgot most details.
Yes, the conflict is detected when KPROBES are using FTRACE
infrastructure. But it happens only when the KPROBE needs to redirect
the function call, namely when it needs to modify IP address which will be used
when all attached ftrace callbacks are proceed.
It is related to the FTRACE_OPS_FL_IPMODIFY flag.
I see, that explains my case as my probes are simple, print only probes that
defently don't that the ip pointer.
But i still don't understand limitation 2 above doesn't show itself - how my
kprobes and especially kretprobes, continue execute even after livepatch
applied to the function i am attached to ? The code execution is redirected
to a different function then to which i attached my probes...
The code is rather complicated and it is not obvious to me.
But I expect that kretprobes modify return addresses which are
stored on the stack. They replace them
with a helper function (trampoline) which would process
the kretprobe callback and jump to the original return address.
It works even with livepatching. The only requirement is that
the function returns using a "ret" instruction which reads
the return address from the stack.
Also - can you please confirm that as far as i checked, starting with kernel
6.0 fentry/fexit on x86 should not have any conflicts with livepatch per
merge of this RFC -
https://urldefense.com/v3/__https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html__;!!BmdzS3_lV9HdKG8!wlwzQrMFIQcM6TTsDw6Eso8D6cFjSA1lZeeRI7xStg-c4RFuGmqrg7B6GeRXLQ8M1l6HgitfnBCfMdSeleudJcNY$
?
My understanding is that the above mentioned patchset allowed to use
livepatching and BPF on the same function at the same time. BPF used
to redirect the function to a BPF trampoline. And the patchset allowed
to call the BPF trampoline/callback directly from the ftrace callback.
Yes, the patch set above allows BPF trampoline to work on the same
function at the same time with livepatch. After this patchset, BPF
trampoline no longer sets FTRACE_OPS_FL_IPMODIFY.
Thanks,
Song
I tested this bellow on 6.1.0 based Ubuntu kernel using bpftrace - in
the end as seen bellow, I wasn't able to attach using fentry/fexit (or
krpboe/kretprobe) as long as livepatch was present
ubuntu@ip-10-10-114-204:~$ uname -r
6.8.0-1008-aws
ubuntu@ip-10-10-114-204:~$ cat /proc/version_signature
Ubuntu 6.8.0-1008.8-aws 6.8.1
[AG] - See bellow the patched function I am to attach to -
ubuntu@ip-10-10-114-204:~$ sudo cat /proc/kallsyms | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
ffffffff81ee0420 T __pfx_begin_new_exec
ffffffff81ee0430 T begin_new_exec
ffffffff8374b3d8 r __ksymtab_begin_new_exec
ffffffffc07c9390 t begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
ffffffffc07c9380 t __pfx_begin_new_exec [lkp_Ubuntu_6_8_0_1008_8_aws_111]
[AG] - Attaching to begin_new_exec
ubuntu@ip-10-10-114-204:~$ sudo bpftool btf dump file
/sys/kernel/btf/vmlinux | grep begin_new_exec
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
[68920] FUNC 'begin_new_exec' type_id=48063 linkage=static
[AG] - , trying to attach to fentry and fexit and first error asks me to
explicitly resolve the ambiguity - which i guess make sense
ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec {
@start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n",
pid, comm); } fexit:begin_new_exec { $latency = nsecs - @start[tid];
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
stdin:1:1-22: ERROR: ambiguous attach point, please specify module
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec {
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm,
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
stdin:1:111-132: ERROR: ambiguous attach point, please specify module
containing the function 'begin_new_exec'
fentry:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC START
(fentry): PID %d, Comm %s\n", pid, comm); } fexit:begin_new_exec {
$latency = nsecs - @start[tid]; delete(@start[tid]); printf("<- EXEC END
(fexit): PID %d, Comm %s, Retval %d, Latency %d us\n", pid, comm,
retval, $latency / 1000); }
~~~~~~~~~~~~~~~~~~~~~
[AG] - Trying first to point him at the original function - but he
fails on the fexit I assume - which is strange, I assumed fexit
(kretfunc) and livepatch can coexist ?
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -e
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
START (fentry): PID %d, Comm %s\n", pid, comm); }
fexit:vmlinux:begin_new_exec { $latency = nsecs - @start[tid];
delete(@start[tid]); printf("<- EXEC END (fexit): PID %d, Comm %s,
Retval %d, Latency %d us\n", pid, comm, retval, $latency / 1000); }'
Attaching 2 probes...
ERROR: Error attaching probe: kretfunc:vmlinux:begin_new_exec
[AG] - Trying to skip the fexit and only do fentry - he still rejects
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -vvv -e
'fentry:vmlinux:begin_new_exec { @start[tid] = nsecs; printf("-> EXEC
START (fentry): PID %d, Comm %s\n", pid, comm); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
INFO: node count: 12
Attaching 1 probe...
Program ID: 295
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
peak_states 3 mark_read 1
Attaching kfunc:vmlinux:begin_new_exec
ERROR: Error attaching probe: kfunc:vmlinux:begin_new_exec
[AG] - Now trying to point him at the livepatch function but here he
complains that Uubntu doesn't have BTF info for the livepatch module
ubuntu@ip-10-10-114-204:~$
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e
'fentry:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { @start[tid] =
nsecs; printf("-> EXEC START (fentry): PID %d, Comm %s\n", pid, comm); }
fexit:lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec { $latency = nsecs
- @start[tid]; delete(@start[tid]); printf("<- EXEC END (fexit): PID %d,
Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency /
1000); }'
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
INFO: node count: 33
Attaching 2 probes...
ERROR: No BTF found for lkp_Ubuntu_6_8_0_1008_8_aws_111:begin_new_exec
[AG] - Disabling livepatch, rebooting to verify all livepatch symbols
are out of kalsym table and now everything fine
ubuntu@ip-10-10-114-204:~$ sudo bpftrace -v -e 'fentry:begin_new_exec
{ @start[tid] = nsecs; printf("-> EXEC START (fentry): PID %d, Comm
%s\n", pid, comm]; delete(@start[tid]); printf("<- EXEC END (fexit): PID
%d, Comm %s, Retval %d, Latency %d us\n", pid, comm, retval, $latency /
1000); }'
INFO: node count: 33
Attaching 2 probes...
Program ID: 33
The verifier log:
processed 76 insns (limit 1000000) max_states_per_insn 0 total_states 5
peak_states 5 mark_read 3
Attaching kretfunc:vmlinux:begin_new_exec
Program ID: 34
The verifier log:
processed 50 insns (limit 1000000) max_states_per_insn 0 total_states 3
peak_states 3 mark_read 1
Attaching kfunc:vmlinux:begin_new_exec
-> EXEC START (fentry): PID 1714, Comm bash
<- EXEC END (fexit): PID 1714, Comm sudo, Retval 0, Latency 99 us
ubuntu@ip-10-10-114-204:~$ sudo bpftool link
sudo: unable to resolve host ip-10-10-114-204: Temporary failure in name
resolution
1: tracing prog 2
prog_type tracing attach_type modify_return
target_obj_id 1 target_btf_id 119714
5: tracing prog 33
prog_type tracing attach_type trace_fexit
target_obj_id 1 target_btf_id 68920
6: tracing prog 34
prog_type tracing attach_type trace_fentry
target_obj_id 1 target_btf_id 68920
ubuntu@ip-10-10-114-204:~$
Thanks,
Andrey
Best Regards,
Petr
More details can be found in the discussion at
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$
I seems that I made some analyze when it worked and it did not work,
see
https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$
But I am not 100% sure that it was correct. Also it was before the
BPF-boom.
Also you might look at the selftest in the todays Linus' tree:
+
tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$
+ tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c
+ tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c
The parallel load fails when the Kprobe is using a post_handler.
Sigh, we should fix the livepatch documentation. The kretprobes
obviously work. register_kretprobe() even explicitely sets:
rp->kp.post_handler = NULL;
It seems that .post_handler is called after all ftrace handlers.
And it sets IP after the NOPs, see kprobe_ftrace_handler().
I am not sure about the use case.
Best Regards,
Petr
