Hi,
Below are the proof-of-concept results for the proposed configurable vCPU
hotplug selection order.
The test uses a guest with 32 vCPUs and two guest NUMA nodes:
* NUMA node 0: vCPUs 0-15
* NUMA node 1: vCPUs 16-31
The guest CPU topology is:
* 2 sockets
* 8 cores per socket
* 2 threads per core
Relevant domain XML:
<vcpu placement='static'>32</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='yes' hotpluggable='yes' order='4'/>
<vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/>
<vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/>
<vcpu id='6' enabled='yes' hotpluggable='yes' order='7'/>
<vcpu id='7' enabled='yes' hotpluggable='yes' order='8'/>
<vcpu id='8' enabled='yes' hotpluggable='yes' order='9'/>
<vcpu id='9' enabled='yes' hotpluggable='yes' order='10'/>
<vcpu id='10' enabled='yes' hotpluggable='yes' order='11'/>
<vcpu id='11' enabled='yes' hotpluggable='yes' order='12'/>
<vcpu id='12' enabled='yes' hotpluggable='yes' order='13'/>
<vcpu id='13' enabled='yes' hotpluggable='yes' order='14'/>
<vcpu id='14' enabled='yes' hotpluggable='yes' order='15'/>
<vcpu id='15' enabled='yes' hotpluggable='yes' order='16'/>
<vcpu id='16' enabled='yes' hotpluggable='yes' order='17'/>
<vcpu id='17' enabled='yes' hotpluggable='yes' order='18'/>
<vcpu id='18' enabled='yes' hotpluggable='yes' order='19'/>
<vcpu id='19' enabled='yes' hotpluggable='yes' order='20'/>
<vcpu id='20' enabled='yes' hotpluggable='yes' order='21'/>
<vcpu id='21' enabled='yes' hotpluggable='yes' order='22'/>
<vcpu id='22' enabled='yes' hotpluggable='yes' order='23'/>
<vcpu id='23' enabled='yes' hotpluggable='yes' order='24'/>
<vcpu id='24' enabled='yes' hotpluggable='yes' order='25'/>
<vcpu id='25' enabled='yes' hotpluggable='yes' order='26'/>
<vcpu id='26' enabled='yes' hotpluggable='yes' order='27'/>
<vcpu id='27' enabled='yes' hotpluggable='yes' order='28'/>
<vcpu id='28' enabled='yes' hotpluggable='yes' order='29'/>
<vcpu id='29' enabled='yes' hotpluggable='yes' order='30'/>
<vcpu id='30' enabled='yes' hotpluggable='yes' order='31'/>
<vcpu id='31' enabled='yes' hotpluggable='yes' order='32'/>
</vcpus>
<cpu mode='host-passthrough' check='none' migratable='on'>
<topology sockets='2' dies='1' clusters='1' cores='8' threads='2'/>
<cache mode='passthrough'/>
<maxphysaddr mode='passthrough'/>
<feature policy='require' name='arch-capabilities'/>
<numa>
<cell id='0' cpus='0-15' memory='20971520' unit='KiB'/>
<cell id='1' cpus='16-31' memory='20971520' unit='KiB'/>
</numa>
</cpu>
Initial state:
# virsh vcpucount testvm
maximum config 32
maximum live 32
current config 32
current live 32
Guest view :
NUMA node(s): 2
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
# Existing count-based behavior
Reducing the guest from 32 to 28 vCPUs:
# virsh setvcpus testvm 28 --live --config
causes the highest-numbered vCPUs to be removed:
smpboot: CPU 31 is now offline
smpboot: CPU 30 is now offline
smpboot: CPU 29 is now offline
smpboot: CPU 28 is now offline
The resulting guest topology is:
NUMA node(s): 2
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-27
This leaves 16 vCPUs in NUMA node 0 and 12 vCPUs in NUMA node 1.
The current count-based interface therefore creates an asymmetric guest
NUMA CPU distribution when vCPU IDs are grouped by NUMA node.
An administrator can avoid this by using direct per-vCPU operations such
as 'virsh setvcpu' and selecting the exact CPUs to remove. However, doing
this requires the management layer to understand each guest's topology and
calculate every individual vCPU operation. This requires additional
coordination and does not scale well across many VMs and hosts.
Restoring the guest to 32 vCPUs adds the same vCPUs back:
# virsh setvcpus testvm 32 --live --config
ACPI: CPU28 has been hot-added
ACPI: CPU29 has been hot-added
ACPI: CPU30 has been hot-added
ACPI: CPU31 has been hot-added
The guest NUMA topology returns to:
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
# Behavior with hporder
For the proof of concept, the following hotplug order was configured:
# virsh setvcpuhporder testvm 31 32 --live --config
# virsh setvcpuhporder testvm 30 31 --live --config
# virsh setvcpuhporder testvm 15 30 --live --config
# virsh setvcpuhporder testvm 14 29 --live --config
# virsh setvcpuhporder testvm 29 28 --live --config
# virsh setvcpuhporder testvm 28 27 --live --config
# virsh setvcpuhporder testvm 13 26 --live --config
# virsh setvcpuhporder testvm 12 25 --live --config
The resulting XML contains:
# virsh dumpxml testvm | grep hporder
<vcpu id='12' enabled='yes' hotpluggable='yes'
order='13' hporder='25'/> <vcpu id='13' enabled='yes' hotpluggable='yes'
order='14' hporder='26'/> <vcpu id='14' enabled='yes' hotpluggable='yes'
order='15' hporder='29'/> <vcpu id='15' enabled='yes' hotpluggable='yes'
order='16' hporder='30'/> <vcpu id='28' enabled='yes' hotpluggable='yes'
order='29' hporder='27'/> <vcpu id='29' enabled='yes' hotpluggable='yes'
order='30' hporder='28'/> <vcpu id='30' enabled='yes' hotpluggable='yes'
order='31' hporder='31'/> <vcpu id='31' enabled='yes' hotpluggable='yes'
order='32' hporder='32'/>
Initial vCPU count:
# virsh vcpucount testvm
maximum config 32
maximum live 32
current config 32
current live 32
Guest view:
NUMA node(s): 2
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
Reducing the guest from 32 to 28 vCPUs:
# virsh setvcpus testvm 28 --live --config
# virsh vcpucount testvm
maximum config 32
maximum live 32
current config 28
current live 28
removes the configured vCPUs in descending hporder:
smpboot: CPU 31 is now offline
smpboot: CPU 30 is now offline
smpboot: CPU 15 is now offline
smpboot: CPU 14 is now offline
The resulting guest topology is:
NUMA node(s): 2
NUMA node0 CPU(s): 0-13
NUMA node1 CPU(s): 16-29
Each NUMA node now contains 14 online vCPUs. The selected CPUs are removed
according to the configured priority.
Reducing the guest further to 26 vCPUs:
# virsh setvcpus testvm 26 --live --config
# virsh vcpucount testvm
maximum config 32
maximum live 32
current config 26
current live 26
removes the next configured vCPUs:
smpboot: CPU 29 is now offline
smpboot: CPU 28 is now offline
This demonstrates that repeated count-based reductions continue to follow
the administrator's configured selection order.
Expanding the guest back to 32 vCPUs:
# virsh setvcpus testvm 32 --live --config
# virsh vcpucount testvm
maximum config 32
maximum live 32
current config 32
current live 32
adds the removed vCPUs in reverse order:
ACPI: CPU28 has been hot-added
ACPI: CPU29 has been hot-added
ACPI: CPU14 has been hot-added
ACPI: CPU15 has been hot-added
ACPI: CPU30 has been hot-added
ACPI: CPU31 has been hot-added
The guest returns to the original balanced topology:
NUMA node(s): 2
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
# Summary
The proof of concept demonstrates that:
* the existing ID-based policy can produce an asymmetric guest NUMA CPU
distribution;
* hporder allows the administrator to define which eligible vCPUs are
removed first during a count-based reduction;
* the configured sequence can maintain a balanced guest NUMA CPU
distribution;
* repeated reductions continue to follow the configured priority; and
* a later expansion restores the ranked vCPUs in the reverse order of
their removal.
The same mechanism can support policies other than NUMA balancing. The
ordering remains administrator-defined and does not require libvirt to
implement a specific topology-balancing policy.
Regards,
Partha
