On 7/4/25 11:37, Daniel Wagner wrote:
On Thu, Jul 03, 2025 at 08:43:01AM +0200, Hannes Reinecke wrote:
All of these drivers are not aware of CPU hotplug, and as such
will not be notified when the number of CPUs changes.
Ah, this explains this part.
But you use 'blk_mq_online_queue_affinity()' for all of these
drivers.
All these drivers are also using blk_mq_num_online_queue. When I only
used cpu_possible_mask the resulting mapping was not usable.
Yeah, I'd imagine so. Quite some drivers have 'interesting' ideas how
the firmware interface should look like.
But it also means that there is a very high likelyhood that these
drivers become inoperable under CPU hotplug.
Is there a way of disabling CPU hotplug when these drivers are in use?
Wouldn't 'blk_mq_possible_queue_affinit()' a better choice here
to insulate against CPU hotplug effects?
With this mask the queues will be distributed to all possible CPUs and
some of the hardware queues could be assigned to offline CPUs. I think
this would work but the question is, is this okay to leave some of the
perfomance on the road?
It really shouldn't be an issue when the cpus are distributed
'correctly' :-)
We have several possibilities:
-> #hwq > num_possible_cpus: easy, 1:1 mapping, no problem
-> num_online_cpu < #hwq < num_possible_cpus: Not as easy, but if we
ensure that each online cpu is mapped to a different hwq we don't
have a performance impact.
-> #hwq < num_online_cpu: If we ensure that a) the number of online cpus
per hwq is (roughly) identical we won't have a performance impact.
As a bonus we should strive to have the number of offline cpus
distributed equally on each hwq.
Of course, that doesn't take into accound NUMA locality; with NUMA
locality you would need to ensure to have at least one CPU per NUMA node
mapped to each hwq. Which actually would impose a lower limit on the
number (and granularity!) of hwqs (namely the number of NUMA nodes), but
that's fair, I guess.
But this really can be delegated to later patches; initially we really
should identify which drivers might have issues with CPU hotplug,
and at the very least issue a warning for these drivers.
I am not against this, just saying it would change the existing
behavior.
Oh, sure. No-one (except lpfc on Power) is testing CPU hotplug actively.
Also some drivers which are using irq affinity (eg aacraid, lpfc) are
missing from these conversions. Why?
I was not aware of aacraid. I started to work on lpfc and well let's put
it this way, it's complicated. lpfc needs a lot of work to make it
isolcpus aware.
Yeah, I know. Sorry ...
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
h...@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
