Added Song Liu and Steven into Cc because they are more familiar with the ftrace trampolines.
On Wed 2025-10-15 17:11:31, Andrey Grodzovsky wrote: > On 10/15/25 07:53, Petr Mladek wrote: > > On Tue 2025-10-14 21:37:49, Andrey Grodzovsky wrote: > > > Dear Upstream Livepatch team and Ubuntu Kernel team - I included you > > > both in this since the issue lies on the boundary between Ubuntu kernel > > > and upstream. > > > > > > According to official kernel documentation - > > > https://urldefense.com/v3/__https://docs.kernel.org/livepatch/livepatch.html*livepatch__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5TeFxUxQ$ > > > , section 7, Limitations - > > > 1 - Kretprobes using the ftrace framework conflict with the patched > > > functions. > > > 2 - Kprobes in the original function are ignored when the code is > > > redirected to the new implementation. > > > > > > Yet, when testing on my Ubuntu 5.15.0-1005.7-aws (based on 5.15.30 stable > > > kernel) machine, I have no problem applying Livepatch and then setting > > > krpobes and kretprobes on a patched function using bpftrace (or directly > > > by coding a BPF program with kprobe/kretprobe attachment)and can confirm > > > both execute without issues. Also the opposite works fine, running my > > > krpobe and kretprobe hooks doesn't prevent from livepatch to be applied > > > successfully. > > > > > > fentry/fexit probes do fail in in both directions - but this is expected > > > according to my understanding as coexistence of livepatching and ftrace > > > based BPF hooks are mutually exclusive until 6.0 based kernels > > > > > > libbpf: prog 'begin_new_exec_fentry': failed to attach: Device or > > > resource busy > > > libbpf: prog 'begin_new_exec_fentry': failed to auto-attach: -16 > > > > > > > > > Please help me understand this contradiction about kprobes - is this > > > because the KPROBES are FTRACE based or any other reason ? > > Heh, it seems that we have discussed this 10 years ago and I already > > forgot most details. > > > > Yes, the conflict is detected when KPROBES are using FTRACE > > infrastructure. But it happens only when the KPROBE needs to redirect > > the function call, namely when it needs to modify IP address which will be > > used > > when all attached ftrace callbacks are proceed. > > > > It is related to the FTRACE_OPS_FL_IPMODIFY flag. > > > I see, that explains my case as my probes are simple, print only probes that > defently don't that the ip pointer. > > But i still don't understand limitation 2 above doesn't show itself - how my > kprobes and especially kretprobes, continue execute even after livepatch > applied to the function i am attached to ? The code execution is redirected > to a different function then to which i attached my probes... The code is rather complicated and it is not obvious to me. But I expect that kretprobes modify return addresses which are stored on the stack. They replace them with a helper function (trampoline) which would process the kretprobe callback and jump to the original return address. It works even with livepatching. The only requirement is that the function returns using a "ret" instruction which reads the return address from the stack. > Also - can you please confirm that as far as i checked, starting with kernel > 6.0 fentry/fexit on x86 should not have any conflicts with livepatch per > merge of this RFC - > https://lkml.iu.edu/hypermail/linux/kernel/2207.2/00858.html ? My understanding is that the above mentioned patchset allowed to use livepatching and BPF on the same function at the same time. BPF used to redirect the function to a BPF trampoline. And the patchset allowed to call the BPF trampoline/callback directly from the ftrace callback. Best Regards, Petr > > > > More details can be found in the discussion at > > https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/T/*re746846b6b16c49a55c89b4c63b7995fe5972111__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu6pjuIgig$ > > > > I seems that I made some analyze when it worked and it did not work, > > see > > https://urldefense.com/v3/__https://lore.kernel.org/all/[email protected]/T/*mffd8c8bf4325b473d89876f2805f42f1af7c82d7__;Iw!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5xbeoulA$ > > But I am not 100% sure that it was correct. Also it was before the > > BPF-boom. > > > > Also you might look at the selftest in the todays Linus' tree: > > > > + > > tools/testing/selftests/livepatch/https://urldefense.com/v3/__http://test-kprobe.sh__;!!BmdzS3_lV9HdKG8!z3Y4vlE7RcCriT3z4Hg7cVaojZPN-ysQTbjDJVXyO_MoRRlkKsymUTDP4PGvvPaV0TDVYhziOYMm9WnUGu5RXF-AnA$ > > + tools/testing/selftests/livepatch/test_modules/test_klp_kprobe.c > > + tools/testing/selftests/livepatch/test_modules/test_klp_livepatch.c > > > > The parallel load fails when the Kprobe is using a post_handler. > > > > Sigh, we should fix the livepatch documentation. The kretprobes > > obviously work. register_kretprobe() even explicitely sets: > > > > rp->kp.post_handler = NULL; > > > > It seems that .post_handler is called after all ftrace handlers. > > And it sets IP after the NOPs, see kprobe_ftrace_handler(). > > I am not sure about the use case. > > > > Best Regards, > > Petr > >
