An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Where Tasks RCU is built on reader-marked trampolines, such a trampoline must be a Tasks Trace reader across its call-out like the ftrace and BPF trampolines are, so document that in register_ftrace_direct().
That still leaves the few instructions before the reader is entered and after it is left. For BPF images those are in dynamically allocated text that rcu_tasks_trampoline_text() already treats as unmarked trampoline text, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag (under CONFIG_TASKS_RCU_TRAMPOLINE_READERS, its only consumer), set by every register/modify path when the direct address is module text, and have rcu_tasks_trampoline_text() treat a task interrupted anywhere in such a module as a potential holdout. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik <[email protected]> --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 18 +++++++++++++++--- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..82ca4f774725 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_trampoline_text(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index eb1388dd8a61..42ea6e0e61cb 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1006,6 +1006,8 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip) * deliberately does not ask is_ftrace_trampoline() and friends, since * text being torn down may already be unregistered there); * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(); + * - the text of a module that hosts an out-of-line ftrace direct-call + * trampoline (see ftrace_direct_mark_module()); * - the bytes after a kprobe that a pending jump optimization is about to * overwrite, the one synchronize_rcu_tasks() user with no trampoline. * @@ -1014,12 +1016,22 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip) */ bool rcu_tasks_trampoline_text(unsigned long ip) { + bool ret = true; + if (core_kernel_text(ip)) return arch_rcu_tasks_trampoline_text(ip) || kprobe_in_optimized_region(ip); - if (is_module_text_address(ip)) - return kprobe_in_optimized_region(ip); - return true; + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod = __module_text_address(ip); + + if (mod) + ret = READ_ONCE(mod->ftrace_direct_tramp) || + kprobe_in_optimized_region(ip); + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_trampoline_text); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..f69f71591358 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) ops->trampoline = 0; } +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_trampoline_text() recognises on its own (see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must be a Tasks + * Trace reader across its call-out (see register_ftrace_direct()); marking the + * owning module here covers the instructions before it enters that reader and + * after it leaves it, where a task interrupted in the module's text must not be + * counted as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed under + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#if defined(CONFIG_MODULES) && defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) + struct module *mod; + + guard(rcu)(); + mod = __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is + * executing it or is preempted in something it called. On architectures that + * select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a task if + * it is a Tasks Trace RCU reader, so the trampoline must enter one + * (rcu_read_lock_trace() or an assembly equivalent) before calling out and + * leave it before returning, just as that option requires of the in-kernel + * ftrace and BPF trampolines. The few instructions before and after are + * covered by the irq-exit check: automatically for trampolines outside kernel + * and module text (e.g. BPF images), and via ftrace_direct_mark_module() for + * trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) ops->flags |= MULTI_FLAGS; ops->trampoline = FTRACE_REGS_ADDR; ops->direct_call = addr; + ftrace_direct_mark_module(addr); err = register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) lockdep_assert_held_once(&direct_mutex); + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash = ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b tmp = __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct = entry->direct; } } -- 2.55.0
