With the preceding patches every trampoline whose lifetime Tasks RCU guards on x86-64 and arm64 -- ftrace_caller and its copies, the optprobe template, BPF trampolines via their glue, and the sample direct-call trampolines -- is a Tasks Trace RCU reader around its call-out, and the text outside that reader is known to rcu_tasks_trampoline_text(). Select HAVE_RCU_TRAMPOLINE_READERS on both (x86-64 with SMP for Tree SRCU and DYNAMIC_FTRACE, which is where its ftrace_caller changes and arch_rcu_tasks_trampoline_text() live; arm64 with DYNAMIC_FTRACE_WITH_ARGS likewise), which switches CONFIG_TASKS_RCU to the implementation added earlier in the series: a Tasks RCU grace period becomes a per-CPU pass over context switches and irq-exit reschedules outside trampoline text plus a Tasks Trace grace period, bounded by a few jiffies and preempt-off latency instead of by the longest stretch any task runs without sleeping.
Other architectures keep the classic implementation. Update Documentation/RCU and the FORCE_TASKS_RCU help text to describe the variant and the obligation it places on trampolines. Assisted-by: LLM Signed-off-by: Josef Bacik <[email protected]> --- .../RCU/Design/Requirements/Requirements.rst | 20 ++++++++++++++++++++ Documentation/RCU/checklist.rst | 7 ++++++- arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + kernel/rcu/Kconfig | 6 ++++-- 5 files changed, 32 insertions(+), 3 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Documentation/RCU/Design/Requirements/Requirements.rst index 8101fe6229d5..34b81512cc5f 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2756,6 +2756,26 @@ synchronize_rcu(), and rcu_barrier(), respectively. In three APIs are therefore implemented by separate functions that check for voluntary context switches. +Architectures that select ``CONFIG_HAVE_RCU_TRAMPOLINE_READERS`` keep the +same three APIs but implement the grace period differently +(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose +lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical +section (rcu_read_lock_trace() or its assembly equivalent) before calling +out and leaves it before returning, so a task anywhere inside such a +call-out is an ordinary Tasks Trace reader whether or not it is +preempted. The few trampoline instructions outside that reader can only +be occupied by a task that was interrupted there, so the grace period +additionally waits for each CPU to pass through a context switch, and the +irq-exit preemption path, the only switch that can catch a task inside +such text (rcu_tasks_trampoline_text()), briefly makes such a task a +holdout until it is next seen elsewhere. On such kernels an involuntary +context switch outside trampoline text *is* a Tasks-RCU quiescent state, +a Tasks RCU grace period no longer depends on how long any task runs +without sleeping, cond_resched_tasks_rcu_qs() is unnecessary, and the +obligation moves to the trampolines: anything that relies on +synchronize_rcu_tasks() to protect code a task may be preempted in must +take the Tasks Trace reader (see register_ftrace_direct()). + Tasks Rude RCU ~~~~~~~~~~~~~~ diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.rst index 4b30f701225f..7082686cbd66 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -252,7 +252,12 @@ over a rather long period of time, but improvements are always welcome! a. If the updater uses synchronize_rcu_tasks() or call_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from - blocking. + blocking. On architectures that select + CONFIG_HAVE_RCU_TRAMPOLINE_READERS a reader must in + addition be a Tasks Trace RCU reader (that is what the + trampolines there do around their call-outs); an + arbitrary stretch of preemptible kernel code is not + protected. b. If the updater uses call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then the diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b5a51b0ef944..bf0e56006863 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -218,6 +218,7 @@ config ARM64 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_PREEMPT_DYNAMIC_KEY + select HAVE_RCU_TRAMPOLINE_READERS if DYNAMIC_FTRACE_WITH_ARGS select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE select HAVE_POSIX_CPU_TIMERS_TASK_WORK diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..64c3814eb745 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -288,6 +288,7 @@ config X86 select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK + select HAVE_RCU_TRAMPOLINE_READERS if X86_64 && SMP && DYNAMIC_FTRACE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if UNWINDER_ORC || STACK_VALIDATION select HAVE_FUNCTION_ARG_ACCESS_API diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index bbab14bc14c3..341b68b972ef 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -95,8 +95,10 @@ config FORCE_TASKS_RCU help This option force-enables a task-based RCU implementation that uses only voluntary context switch (not preemption!), - idle, and user-mode execution as quiescent states. Not for - manual selection in most cases. + idle, and user-mode execution as quiescent states, or, on + HAVE_RCU_TRAMPOLINE_READERS architectures, the variant built on + Tasks Trace RCU readers in trampolines. Not for manual + selection in most cases. config NEED_TASKS_RCU bool -- 2.55.0
