Hello Andrey, On Wed, 19 Nov 2025, Andrey Grodzovsky wrote:
> Hello BPF and livepatch teams, > > This is somewhat a followup on > https://lists.ubuntu.com/archives/kernel-team/2025-October/163881.html as we > continue encounter issues and conflicts between BPF and livepatch. > > We've encountered an issue between BPF fentry/fexit trampolines and kernel > livepatching (kpatch/livepatch) on x86_64 systems with ORC unwinder enabled. > I'm reaching out to understand if this is a known limitation and to explore > potential solutions. I assume it's known as I see information along this lines > in https://www.kernel.org/doc/Documentation/livepatch/reliable-stacktrace.rst > > Problem Summary > > When BPF programs attach to kernel functions using fentry/fexit hooks, the > resulting JIT-compiled trampolines lack ORC unwind metadata. This causes > livepatch transition stall when threads are blocked in hooked functions, as > the stack becomes unreliable for unwinding purposes. > > In our case the environment is > > - RHEL 9.6 (kernel 5.14.0-570.17.1.el9_6.x86_64) > - CONFIG_UNWINDER_ORC=y > - CONFIG_BPF_JIT_ALWAYS_ON=y > - BPF fentry/fexit hooks on inet_recvmsg() > > Scenario: > 1. BPF program attached to inet_recvmsg via fentry/fexit (creates BPF > trampoline) > 2. CIFS filesystem mounted (creates cifsd kernel thread) > 3. cifsd thread blocks in inet_recvmsg → BPF trampoline is on the stack > 4. Attempt to load kpatch module > 5. Livepatch transition stalls indefinitely > > Error Message (repeated every ~1 second): > livepatch: klp_try_switch_task: cifsd:2886 has an unreliable stack > > Stack trace showing BPF trampoline: > cifsd D 0 2886 > Call Trace: > wait_woken+0x50/0x60 > sk_wait_data+0x176/0x190 > tcp_recvmsg_locked+0x234/0x920 > tcp_recvmsg+0x78/0x210 > inet_recvmsg+0x5c/0x140 > bpf_trampoline_6442469985+0x89/0x130 ← NO ORC metadata > sock_recvmsg+0x95/0xa0 > cifs_readv_from_socket+0x1ca/0x2d0 [cifs] > ... > > As far as I understand and please correct me if it's wrong - > > The failure occurs in arch/x86/kernel/unwind_orc.c > > orc = orc_find(state->signal ? state->ip : state->ip - 1); > if (!orc) { > /* > * As a fallback, try to assume this code uses a frame pointer. > * This is useful for generated code, like BPF, which ORC > * doesn't know about. This is just a guess, so the rest of > * the unwind is no longer considered reliable. > */ > orc = &orc_fp_entry; > state->error = true; // ← Marks stack as unreliable > } > > When orc_find() returns NULL for the BPF trampoline address, the unwinder > falls back to frame pointers and marks the stack unreliable. This causes > arch_stack_walk_reliable() to fail, which in turn causes livepatch's > klp_check_stack() to return -EINVAL before even checking if to-be-patched > functions are on the stack. > > Key observations: > 1. The kernel comment explicitly mentions "generated code, like BPF" > 2. Documentation/livepatch/reliable-stacktrace.rst lists "Dynamically > generated code (e.g. eBPF)" as causing unreliable stacks > 3. Native kernel functions have ORC metadata from objtool during build > 4. Ftrace trampolines have special ORC handling via orc_ftrace_find() > 5. BPF JIT trampolines have no such handling - Is this correct ? Yes, all you findings are correct and the above explains the situation really well. Thank you for summing it up. > Impact > > This affects production systems where: > - Security/observability tools use BPF fentry/fexit hooks > - Live kernel patching is required for security updates > - Kernel threads may be blocked in hooked network/storage functions > > The livepatch transition can stall for 60+ seconds before failing, blocking > critical security patches. Unfortunately yes. > Questions for the Community > > 1. Is this a known limitation (I assume yes) ? Yes. > 2. Runtime ORC generation? Could the BPF JIT generate ORC unwind entries for > trampolines, similar to how ftrace trampolines are handled? > 3. Trampoline registration? Could BPF trampolines register their address > ranges with the ORC unwinder to avoid the "unreliable" marking? > 4. Alternative unwinding? Could livepatch use an alternative unwinding method > when BPF trampolines are detected (e.g., frame pointers with validation)? > 5. Workarounds? I mention one bellow and I would be happy to hear if anyone > has a better idea to propose ? There is a parallel discussion going on under sframe unwiding enablement for arm64. See this subthread https://lore.kernel.org/all/cadbmgpwz32+shsa0swo8y4g-zw14ae-fcowrea_ptmf08mu...@mail.gmail.com/T/#u I would really welcome if it is solved eventually because it seems we will meet the described issue more and more often (Josh, I think this email shows that it happens in practice with the existing monitoring services based on BPF). Regards, Miroslav
