Public bug reported:
nova compute doesn't respect cpu_model_extra_flags while comparing CPU feature
flag sets of spawned VMs with HV
(https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714)
Environment:
OS: Almalinux 9.5
Openstack version: Caracal
openstack-nova-compute version 29.2.0
nova compute configuration:
```
cpu_mode = custom
cpu_models = Cascadelake-Server-noTSX
cpu_model_extra_flags = vmx,pcid
```
NOTE: Cascadelake-Server-noTSX custom model does not provide VMX flag
(needed for nested virtualization) and is compatible with used compute
node CPU
Test scenario 1 (cpu_model_extra_flags is not used):
1. Remove cpu_model_extra_flags from nova config, restart nova compute service
2. VMX flag does not appear in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 |
grep VMX
| COMPUTE_NET_VIF_MODEL_VMXNET3 |
```
3. Create a new flavor, or modify existing one with VMX trait requirement in
metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image
46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network
--os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group
13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM creation fails on scheduling stage
This behavior is expected since no HVs are reporting a compatible list
of CPU flags for this VM
Test scenario 2 (cpu_model_extra_flags is set to `vmx,pcid`)
1. Set cpu_model_extra_flags to `vmx,pcid`, restart nova compute service
2. VMX flag appears in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 |
grep VMX
| HW_CPU_X86_VMX |
| HW_CPU_X86_INTEL_VMX |
```
3. Create a new flavor, or modify existing one with VMX trait requirement in
metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image
46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network
--os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group
13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM is scheduled to correct HV (one with cpu_model_extra_flags set to
vmx,pcid)
6. VM spawn is failed on compute host due to CPU flags incompatibility:
```
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [None
req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf
16f06b95007248c98d6b1dec1581088b - - default default] [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Instance failed to spawn:
nova.exception.InvalidCPUInfo: No CPU model match traits, models:
['Cascadelake-Server-noTSX'], required flags: {'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Traceback (most recent call last):
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2885, in
_build_resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] yield resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2632, in
_build_and_run_instance
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] self.driver.spawn(context, instance,
image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4638, in
spawn
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] xml = self._get_guest_xml(context,
instance, network_info,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7817, in
_get_guest_xml
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] conf =
self._get_guest_config(instance, network_info, image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7297, in
_get_guest_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] guest.cpu =
self._get_guest_cpu_config(
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5709, in
_get_guest_cpu_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] cpu =
self._get_guest_cpu_model_config(flavor, arch)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5634, in
_get_guest_cpu_model_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] cpu.model =
self._match_cpu_model_by_flags(models, flags)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5704, in
_match_cpu_model_by_flags
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] raise exception.InvalidCPUInfo(msg)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] nova.exception.InvalidCPUInfo: No CPU
model match traits, models: ['Cascadelake-Server-noTSX'], required flags:
{'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979]
2025-06-25 13:00:48.129 2493382 INFO nova.compute.manager [None
req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf
16f06b95007248c98d6b1dec1581088b - - default default] [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Terminating instance
```
Expected behavior:
The VM is spawned without errors on the HV with cpu_model_extra_flags is set to
`vmx,pcid`
This happens in
https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714
namely because of requested `flags` set is not a subset `feature_names` which
is a flag set of a CPU model configured in nova.conf
According to nova docs cpu_model_extra_flags should allow adding flags to
pre-defined custom models, which it seemingly does not do
(https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-feature-flags)
Proposed patch is attached. It simply adds a flag set from
cpu_model_extra_flags to the list derived from baseline CPU and then
used for comparison, which seems consistent with documented logic
** Affects: nova
Importance: Undecided
Status: New
** Tags: nova nova-compute
** Patch added: "nova_extra_flags.patch"
https://bugs.launchpad.net/bugs/2116882/+attachment/5889712/+files/nova_extra_flags.patch
** Tags added: nova
** Tags added: nova-compute
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2116882
Title:
nova compute doesn't respect cpu_model_extra_flags
Status in OpenStack Compute (nova):
New
Bug description:
nova compute doesn't respect cpu_model_extra_flags while comparing CPU
feature flag sets of spawned VMs with HV
(https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714)
Environment:
OS: Almalinux 9.5
Openstack version: Caracal
openstack-nova-compute version 29.2.0
nova compute configuration:
```
cpu_mode = custom
cpu_models = Cascadelake-Server-noTSX
cpu_model_extra_flags = vmx,pcid
```
NOTE: Cascadelake-Server-noTSX custom model does not provide VMX flag
(needed for nested virtualization) and is compatible with used compute
node CPU
Test scenario 1 (cpu_model_extra_flags is not used):
1. Remove cpu_model_extra_flags from nova config, restart nova compute service
2. VMX flag does not appear in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 |
grep VMX
| COMPUTE_NET_VIF_MODEL_VMXNET3 |
```
3. Create a new flavor, or modify existing one with VMX trait requirement in
metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image
46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network
--os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group
13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM creation fails on scheduling stage
This behavior is expected since no HVs are reporting a compatible list
of CPU flags for this VM
Test scenario 2 (cpu_model_extra_flags is set to `vmx,pcid`)
1. Set cpu_model_extra_flags to `vmx,pcid`, restart nova compute service
2. VMX flag appears in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 |
grep VMX
| HW_CPU_X86_VMX |
| HW_CPU_X86_INTEL_VMX |
```
3. Create a new flavor, or modify existing one with VMX trait requirement in
metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image
46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network
--os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group
13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM is scheduled to correct HV (one with cpu_model_extra_flags set to
vmx,pcid)
6. VM spawn is failed on compute host due to CPU flags incompatibility:
```
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [None
req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf
16f06b95007248c98d6b1dec1581088b - - default default] [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Instance failed to spawn:
nova.exception.InvalidCPUInfo: No CPU model match traits, models:
['Cascadelake-Server-noTSX'], required flags: {'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Traceback (most recent call last):
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2885, in
_build_resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] yield resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2632, in
_build_and_run_instance
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] self.driver.spawn(context, instance,
image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4638, in
spawn
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] xml = self._get_guest_xml(context,
instance, network_info,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7817, in
_get_guest_xml
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] conf =
self._get_guest_config(instance, network_info, image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7297, in
_get_guest_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] guest.cpu =
self._get_guest_cpu_config(
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5709, in
_get_guest_cpu_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] cpu =
self._get_guest_cpu_model_config(flavor, arch)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5634, in
_get_guest_cpu_model_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] cpu.model =
self._match_cpu_model_by_flags(models, flags)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] File
"/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5704, in
_match_cpu_model_by_flags
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] raise exception.InvalidCPUInfo(msg)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] nova.exception.InvalidCPUInfo: No CPU
model match traits, models: ['Cascadelake-Server-noTSX'], required flags:
{'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979]
2025-06-25 13:00:48.129 2493382 INFO nova.compute.manager [None
req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf
16f06b95007248c98d6b1dec1581088b - - default default] [instance:
d747a173-2bb9-4794-8ea2-c5ab298b4979] Terminating instance
```
Expected behavior:
The VM is spawned without errors on the HV with cpu_model_extra_flags is set
to `vmx,pcid`
This happens in
https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714
namely because of requested `flags` set is not a subset `feature_names` which
is a flag set of a CPU model configured in nova.conf
According to nova docs cpu_model_extra_flags should allow adding flags to
pre-defined custom models, which it seemingly does not do
(https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-feature-flags)
Proposed patch is attached. It simply adds a flag set from
cpu_model_extra_flags to the list derived from baseline CPU and then
used for comparison, which seems consistent with documented logic
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2116882/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
