For now, the bpf trampoline is called by the "call" instruction. However, it break the RSB and introduce extra overhead in x86_64 arch.
For example, we hook the function "foo" with fexit, the call and return logic will be like this: call foo -> call trampoline -> call foo-body -> return foo-body -> return foo As we can see above, there are 3 call, but 2 return, which break the RSB balance. We can pseudo a "return" here, but it's not the best choice, as it will still cause once RSB miss: call foo -> call trampoline -> call foo-body -> return foo-body -> return dummy -> return foo The "return dummy" doesn't pair the "call trampoline", which can also cause the RSB miss. Therefore, we introduce the "jmp" mode for bpf trampoline, as advised by Alexei in [1]. And the logic will become this: call foo -> jmp trampoline -> call foo-body -> return foo-body -> return foo As we can see above, the RSB is totally balanced. After the modification, the performance of fexit increases from 76M/s to 130M/s. In this series, we introduce the FTRACE_OPS_FL_JMP for ftrace to make it use the "jmp" instruction instead of "call". And we also do some adjustment to bpf_arch_text_poke() to allow us specify the old and new poke_type. Link: https://lore.kernel.org/bpf/[email protected]/ Changes since v1: * change the bool parameter that we add to save_args() to "u32 flags" * rename bpf_trampoline_need_jmp() to bpf_trampoline_use_jmp() * add new function parameter to bpf_arch_text_poke instead of introduce bpf_arch_text_poke_type() * rename bpf_text_poke to bpf_trampoline_update_fentry * remove the BPF_TRAMP_F_JMPED and check the current mode with the origin flags instead. Link: https://lore.kernel.org/bpf/caadnvqlx54svi1oahrksilqjjajdm3tqjovrgu-lzimk6id...@mail.gmail.com/[1] Menglong Dong (6): ftrace: introduce FTRACE_OPS_FL_JMP x86/ftrace: implement DYNAMIC_FTRACE_WITH_JMP bpf: fix the usage of BPF_TRAMP_F_SKIP_FRAME bpf,x86: adjust the "jmp" mode for bpf trampoline bpf: specify the old and new poke_type for bpf_arch_text_poke bpf: implement "jmp" mode for trampoline arch/arm64/net/bpf_jit_comp.c | 14 +++--- arch/loongarch/net/bpf_jit.c | 9 ++-- arch/powerpc/net/bpf_jit_comp.c | 8 ++-- arch/riscv/net/bpf_jit_comp64.c | 11 +++-- arch/s390/net/bpf_jit_comp.c | 7 +-- arch/x86/Kconfig | 1 + arch/x86/kernel/ftrace.c | 7 ++- arch/x86/kernel/ftrace_64.S | 12 ++++- arch/x86/net/bpf_jit_comp.c | 55 +++++++++++++---------- include/linux/bpf.h | 18 +++++++- include/linux/ftrace.h | 33 ++++++++++++++ kernel/bpf/core.c | 5 ++- kernel/bpf/trampoline.c | 78 +++++++++++++++++++++++++-------- kernel/trace/Kconfig | 12 +++++ kernel/trace/ftrace.c | 9 +++- 15 files changed, 212 insertions(+), 67 deletions(-) -- 2.51.2
