On 7/1/26 14:56, Waiman Long wrote:
> On 7/1/26 10:22 AM, Frederic Weisbecker wrote:
> > > I know RCU support changing the nocb mask for fully offline CPUs, I
> > > will need to find out if it possible to do that for partially
> > > offline CPUs.
> > No because callbacks can still be enqueued at this stage. But we could
> > manage to make it work with CPUHP_AP_IDLE_DEAD.
>
> If we can only go as high as CPUHP_AP_IDLE_DEAD, we may as well go down
> all the way to CPUHP_OFFLINE [...] we may have to break RCU out from
> HK_TYPE_KERNEL_NOISE and add a cpuset control switch [...]
A data point from the DHM side that corroborates this.
Our RCU/nocb prototype toggled the NOCB state by fully offlining each
affected CPU, one at a time:
remove_cpu(cpu);
rcu_nocb_cpu_offload(cpu) / rcu_nocb_cpu_deoffload(cpu);
add_cpu(cpu);
i.e. it went all the way to CPUHP_OFFLINE, which matches Frederic's
point that the nocb mask change needs the CPU at least at
CPUHP_AP_IDLE_DEAD - callbacks are still enqueued before that. It worked
functionally, but it also confirmed the cost Waiman describes: each
remove_cpu() pays the stop_machine price, so doing this while another
isolated partition is running latency-sensitive work is disruptive.
We are reworking that code precisely because doing it asynchronously
raced with concurrent CPU hotplug (a TOCTOU on cpu_online() and the nocb
state), so +1 that this has to be serialized against hotplug the way
Thomas outlined.
So decoupling RCU from HK_TYPE_KERNEL_NOISE and gating the "pay the
stop_machine spike" behaviour behind an explicit cpuset switch sounds
right to us. RCU seems to be the only kernel-noise type that needs to go
that deep; tick, managed_irq and the watchdog appear to only need the CPU
to cycle through the existing online-side callbacks, not a forced
IDLE_DEAD - please correct us if that is wrong.
On Frederic's idea of asking userspace to offline the target CPUs before
toggling isolation: that cleanly removes the kernel-internal offline for
the isolate direction. How would you see the de-isolate direction - the
admin brings the CPU back online and the housekeeping/nocb masks are
recomputed in the online path?
Waiman, since you mentioned you have not started on RCU and it carries
the deepest hotplug constraints, we are happy to take the RCU/nocb
decoupling piece and build it on top of your CPU down/up primitives,
following Frederic's CPUHP_AP_IDLE_DEAD guidance. That keeps the
subsystem split clean and avoids duplicating your tick/irq work.
Thanks,
Jing Wu
Qiliang Yuan
