kprobe_optimizer() is the one synchronize_rcu_tasks() user that is not about trampoline text: it waits for tasks that were interrupted on an instruction boundary inside the bytes it is about to overwrite with the optimized jump, so that none of them resumes into the middle of the new instruction. Those bytes are ordinary kernel or module text with no Tasks Trace reader around them, so on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels the irq-exit quiescent-state check has to be told about them.
Add kprobe_in_optimized_region(), a lockless and conservative form of get_optimized_kprobe() that reports whether any registered kprobe lies within MAX_OPTIMIZED_LENGTH before the given address regardless of its optimization state, and have rcu_tasks_trampoline_text() consult it for core and module text so that a task interrupted there becomes a holdout rather than a quiescent event. The hash walk only runs while kprobe_optimizer() is actually inside its synchronize_rcu_tasks(), tracked by a flag it sets around the call; otherwise the check is a single load. That check cannot see a task that was already preempted in the region before the flag went up (possibly before the kprobe even existed), and the new grace period does not otherwise wait for a preempted task to run again, so before synchronize_rcu_tasks() the optimizer calls rcu_tasks_wait_irq_preempted() to wait until no parked task's recorded irq-exit preemption IP is inside such a region; its leading synchronize_rcu() also publishes the flag to every (interrupts- disabled) check in flight. The kprobe hash is RCU-protected and every free path waits for a grace period after unhashing, so the lockless walk from the irq-exit path is safe. On other configurations the flag is set and cleared but nothing reads it and rcu_tasks_wait_irq_preempted() is a stub; the classic implementation already waits for such tasks. Assisted-by: LLM Signed-off-by: Josef Bacik <[email protected]> --- include/linux/kprobes.h | 8 +++++++- kernel/kprobes.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++++ kernel/rcu/tasks.h | 11 ++++++++--- 3 files changed, 65 insertions(+), 4 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index e6de7ae55bda..74cc48c04417 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long addr) } #endif /* !CONFIG_KPROBES */ -#ifndef CONFIG_OPTPROBES +#ifdef CONFIG_OPTPROBES +bool kprobe_in_optimized_region(unsigned long addr); +#else /* !CONFIG_OPTPROBES */ static inline bool is_kprobe_optinsn_slot(unsigned long addr) { return false; } +static inline bool kprobe_in_optimized_region(unsigned long addr) +{ + return false; +} #endif /* !CONFIG_OPTPROBES */ #ifdef CONFIG_KRETPROBES diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 6337da5cab9e..e460fba83e4a 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr) return NULL; } +/* + * True while kprobe_optimizer() is waiting for its Tasks RCU grace period. + * Only in that window can an interruption inside an optprobe's jump region + * matter to it, so kprobe_in_optimized_region() does no work otherwise. + */ +static bool kprobe_optimizer_waiting; + +/** + * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimized + * kprobe replaces? + * @addr: kernel text address, typically an interrupted instruction pointer + * + * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks that + * were interrupted on an instruction boundary inside the region about to be + * overwritten by the optimized jump. Where Tasks RCU is built on + * reader-marked trampolines that region has no reader, so the irq-exit + * quiescent-state check asks this instead (see rcu_tasks_trampoline_text()). + * This is the lockless, conservative form of get_optimized_kprobe(): it does + * not care whether the kprobe found is, or ever will be, optimized. May be + * called from any context with preemption disabled; the kprobe hash is + * RCU-protected and every free path waits for a grace period after unhashing. + * + * The hash walk only runs while the optimizer is actually waiting. A task + * that was preempted in such a region before the flag went up is invisible + * to that check, so the optimizer first waits those out by their recorded + * preemption IP (rcu_tasks_wait_irq_preempted(), whose leading + * synchronize_rcu() also publishes the flag to every check in flight). + */ +bool kprobe_in_optimized_region(unsigned long addr) +{ + int i; + + if (!READ_ONCE(kprobe_optimizer_waiting)) + return false; + + for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + if (get_kprobe((kprobe_opcode_t *)addr - i)) + return true; + return false; +} +NOKPROBE_SYMBOL(kprobe_in_optimized_region); + /* Optimization staging list, protected by 'kprobe_mutex' */ static LIST_HEAD(optimizing_list); static LIST_HEAD(unoptimizing_list); @@ -644,8 +686,16 @@ static void kprobe_optimizer(void) * to 2nd-Nth byte of jump instruction. This wait is for avoiding it. * Note that on non-preemptive kernel, this is transparently converted * to synchronoze_sched() to wait for all interrupts to have completed. + * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU + * recognise tasks interrupted in such a region while we wait, and + * rcu_tasks_wait_irq_preempted() (a no-op elsewhere) first waits + * out any that were preempted there before we said so; see + * kprobe_in_optimized_region(). */ + WRITE_ONCE(kprobe_optimizer_waiting, true); + rcu_tasks_wait_irq_preempted(kprobe_in_optimized_region); synchronize_rcu_tasks(); + WRITE_ONCE(kprobe_optimizer_waiting, false); /* Step 3: Optimize kprobes after quiesence period */ do_optimize_kprobes(); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 3a7c092361a6..866768462850 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1006,7 +1006,9 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip) * text being torn down may already be unregistered there); * - the .text..rcu_tramp section, C glue called directly from such * trampolines before it has entered the reader; - * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(). + * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(); + * - the bytes after a kprobe that a pending jump optimization is about to + * overwrite, the one synchronize_rcu_tasks() user with no trampoline. * * A false positive only makes the task a holdout until its next quiescent * event. Called with interrupts disabled from the irq-exit path. @@ -1017,9 +1019,12 @@ bool rcu_tasks_trampoline_text(unsigned long ip) if (ip >= (unsigned long)__rcu_tramp_text_start && ip < (unsigned long)__rcu_tramp_text_end) return true; - return arch_rcu_tasks_trampoline_text(ip); + return arch_rcu_tasks_trampoline_text(ip) || + kprobe_in_optimized_region(ip); } - return !is_module_text_address(ip); + if (is_module_text_address(ip)) + return kprobe_in_optimized_region(ip); + return true; } NOKPROBE_SYMBOL(rcu_tasks_trampoline_text); -- 2.55.0
