Since we offered to take the RCU/nocb piece, here is a concrete shape to
shoot at.
Constraint recap (kernel/rcu/tree_nocb.h): rcu_nocb_cpu_{offload,deoffload}()
require the CPU offline (they reject when cpu_online()) under cpus_read_lock +
nocb_mutex, and per Frederic the toggle really needs CPUHP_AP_IDLE_DEAD because
callbacks can still be enqueued before that. Reaching IDLE_DEAD pays the
stop_machine cost that disturbs other isolated partitions.
Observation: among the kernel-noise types, only RCU needs that deep offline;
tick, managed_irq and the watchdog reconfigure through the existing online-side
callbacks. So bundling RCU under HK_TYPE_KERNEL_NOISE makes every isolation
change pay the RCU cost.
Proposal:
1. Split HK_TYPE_RCU out of HK_TYPE_KERNEL_NOISE into its own
runtime-mutable type, plus an immutable HK_TYPE_RCU_BOOT snapshot.
HK_TYPE_KERNEL_NOISE keeps tick/timer/misc/wq.
2. Gate nocb changes behind an explicit, per-partition cpuset opt-in.
Default off: partition changes never touch rcu_nocb_mask, so no spike is
inflicted on other partitions. On: the isolated CPUs join/leave the nocb
set, paying the offline cost once at setup/teardown. This is Waiman's
admin-choice knob.
3. Do the toggle inside the serialized hotplug transition at
CPUHP_AP_IDLE_DEAD, rather than the remove_cpu() -> toggle -> add_cpu()
dance our v3 prototype used, which raced with concurrent hotplug.
Open questions:
- Frederic: is CPUHP_AP_IDLE_DEAD the right hook, and is a new cpuhp callback
there acceptable versus the current post-offline API?
- admin pre-offlines the target CPUs (your earlier suggestion) versus the
cpuset code cycling them via Thomas's per-CPU down/up primitive?
- does the "only RCU needs the deep offline" assumption hold for tick,
managed_irq and the watchdog?
If this shape sounds acceptable we will build it on top of your CPU down/up
primitives.
Thanks,
Jing Wu
Qiliang Yuan
