qemu: add configurable vCPU hotplug selection order
Count-based vCPU hotplug currently selects eligible vCPU hotplug
entities according to their vCPU IDs. For example:
virsh setvcpus VM COUNT --live
does not provide a way for an administrator to control which eligible
entities are added or removed first.
The existing ID-based policy is suitable for simple guests and for
deployments where the vCPU ID layout already represents the desired
placement policy. Domains that do not configure the new feature continue
to use the current ID-based behavior.
One use case is preserving a balanced guest NUMA CPU distribution during
count-based vCPU changes. Depending on how vCPU IDs are assigned to
guest NUMA cells, repeatedly adding or removing vCPUs by ID may
concentrate the change in one cell before affecting another, resulting
in an undesirable asymmetric guest CPU layout.
There is no single balancing policy that is correct for every guest. An
administrator may want to remove one hotplug entity from each guest NUMA
cell in a round-robin sequence. For a guest with a threaded CPU
topology, the preferred policy may instead remove a complete core, or
several threads from one cell, before proceeding to the next. The
appropriate sequence depends on the guest topology, workload,
architecture, and performance requirements.
The mechanism is not limited to NUMA balancing. It can also express
administrator-defined policies such as:
* retaining CPUs used for latency-sensitive, isolated, or
housekeeping workloads;
* implementing CPU capacity tiers;
* satisfying software licensing or application-affinity constraints;
* preserving a preferred socket, core, or thread population while
reducing capacity; or
* restoring vCPUs in a predictable order after an earlier reduction.
Rather than embedding any one placement policy in libvirt, this proposal
provides a general ordering mechanism. Administrators can describe the
desired selection sequence in the domain XML according to the guest and
deployment requirements.
The existing per-vCPU 'order' attribute has different semantics and
cannot be reused for this purpose. It is used to define the order and
grouping of vCPUs when they are enabled. It applies to online vCPUs and
is not a policy for selecting offline hotpluggable vCPUs.
In particular, 'order' is not used to choose which vCPUs are removed by
a count-based `setvcpus` reduction. Therefore, it cannot express an
administrator-defined hot-unplug sequence.
In contrast, 'hporder' is configured on disabled hotpluggable vCPUs and
is used when libvirt selects eligible hotplug entities for count-based
vCPU changes. It determines both which entities are removed during
reduction and the order in which ranked entities are restored during
expansion. It does not change the existing 'order' semantics, startup
behavior, or hotplug entity composition.
Introduce an optional positive per-vCPU 'hporder' XML attribute for QEMU
domains. If the attribute is omitted, the vCPU remains unranked.
All vCPUs belonging to the same QEMU hotplug entity must have the same
'hporder' value because they cannot be added or removed independently.
The configuration is rejected if the values within an entity differ.
For count-based operations, the QEMU driver applies the following
selection policy:
* On expansion, select unranked entities in ascending vCPU ID order,
followed by ranked entities in ascending 'hporder' order.
* On reduction, select ranked entities in descending 'hporder' order,
followed by unranked entities in descending vCPU ID order.
The ordering has two useful properties. First, domains without 'hporder'
retain the existing behavior. Second, among ranked entities, expansion
naturally restores entities in the reverse order in which reduction
removed them.
Unranked entities are selected first during expansion so that
configuring 'hporder' can be used primarily to describe a preferred
removal order without changing the normal expansion order of other
hotpluggable vCPUs.
The setting affects only the selection of eligible entities during
count-based vCPU operations. It does not change:
* the initial boot CPU order;
* the existing per-vCPU 'order' attribute;
* the composition of a QEMU hotplug entity;
* whether a vCPU is hotpluggable; or
* operations that explicitly address individual vCPUs.
The value is preserved in domain XML and exposed through:
virDomainSetVcpuHpOrder()
A positive API value configures the order. Passing zero removes the
configured value. Zero is not valid in domain XML and is therefore
omitted when the XML is formatted.
The corresponding virsh commands are:
setvcpuhporder DOMAIN VCPU HPORDER
delvcpuhporder DOMAIN VCPU
Proposed implementation:
1. Add the 'hporder' attribute to the per-vCPU domain XML.
2. Validate that all vCPUs belonging to the same QEMU hotplug entity use
the same 'hporder' value.
3. Use 'hporder' when selecting eligible vCPU hotplug entities for
count-based expansion and reduction.
4. Add a public API for configuring or removing the 'hporder' value.
5. Add virsh commands for setting and removing the value.
6. Update the domain XML and API documentation to describe the new
attribute, its valid values, its interaction with the existing
per-vCPU 'order' attribute, and the selection rules used for
expansion and reduction.
7. Add automated test coverage for XML parsing and formatting, invalid
zero values, hotplug-entity consistency, and count-based expansion
and reduction ordering.
RFC: Feedback is welcome on the ordering policy and interface. In
particular, feedback is requested on the XML/API naming, the handling of
unranked vCPUs during expansion, and whether this should remain a
QEMU-only feature.
Signed-off-by: Partha Sarathi Satapathy <[email protected]>
