Reviewed: https://review.opendev.org/468203 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b06597418799b382d091bd2ff4b747e6f5a10d96 Submitter: Zuul Branch: master
commit b06597418799b382d091bd2ff4b747e6f5a10d96 Author: Chris Friesen <[email protected]> Date: Thu May 25 17:13:22 2017 -0600 hardware: Add validation for 'cpu_realtime_mask' It's possible to specify some strange values for the cpu realtime mask and the code won't currently complain, so let's make it a bit more strict. Change-Id: I0ba06529affe5b48af5ac37bc24242dffdac77d3 Closes-Bug: #1884231 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1884231 Title: 'hw:realtime_mask' extra spec is not validated Status in OpenStack Compute (nova): Fix Released Bug description: The 'hw:realtime_mask' extra spec is (currently) used to specify what cores in a host should *not* be part of the realtime set of cores on the host. Currently, this is mandatory and omitting it will cause a HTTP 400 error. For example: $ openstack flavor create --ram 512 --disk 1 --vcpus 2 \ --property hw:cpu_policy=dedicated --property hw:cpu_realtime=yes \ test.rt will fail with: Realtime policy needs vCPU(s) mask configured with at least 1 RT vCPU and 1 ordinary vCPU. See hw:cpu_realtime_mask or hw_cpu_realtime_mask Similarly, attempting to mask *all* values will result in a failure. For example: $ openstack flavor create --ram 512 --disk 1 --vcpus 2 \ --property hw:cpu_policy=dedicated --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask=^0-1 test.rt will also fail with: Realtime policy needs vCPU(s) mask configured with at least 1 RT vCPU and 1 ordinary vCPU. See hw:cpu_realtime_mask or hw_cpu_realtime_mask However, the value is otherwise unvalidated by nova, which can cause libvirt to explode when specific values are passed. For example, consider the following flavor: $ openstack flavor create --ram 512 --disk 1 --vcpus 2 \ --property hw:cpu_policy=dedicated --property hw:cpu_realtime=yes \ --property hw:cpu_realtime_mask='^2' \ test.rt This states that the instances should have two cores, and some imaginary third core (masks are 0-indexed) will be the non-realtime one. This is clearly nonsensical and, surely enough, creating an instance using this core causes things to go bang: Failed to build and run instance: libvirt.libvirtError: invalid argument: Failed to parse bitmap '' Traceback (most recent call last): File "/opt/stack/nova/nova/compute/manager.py", line 2378, in _build_and_run_instance accel_info=accel_info) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3702, in spawn cleanup_instance_disks=created_disks) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6664, in _create_domain_and_network cleanup_instance_disks=cleanup_instance_disks) File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line220, in __exit__ self.force_reraise() File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6633, in _create_domain_and_network post_xml_callback=post_xml_callback) File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6559, in _create_domain guest = libvirt_guest.Guest.create(xml, self._host) File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 127, in create encodeutils.safe_decode(xml)) File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line220, in __exit__ self.force_reraise() File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 123, in create guest = host.write_instance_config(xml) File "/opt/stack/nova/nova/virt/libvirt/host.py", line 1141, in write_instance_config domain = self.get_connection().defineXML(xml) File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190,in doit result = proxy_call(self._autowrap, f, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call rv = execute(f, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute six.reraise(c, e, tb) File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker rv = meth(*args, **kwargs) File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 4048, in defineXML if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self) libvirt.libvirtError: invalid argument: Failed to parse bitmap '' The error happens because libvirt is attempting to configure the set CPUs on which to pin emulators threads, which in the realtime case are all the non-realtime cores. However, since there are no cores set aside for non-realtime purposes - due to the invalid mask - we end up with an empty emulator thread set [1]. One *could* work around this by configuring an emulator thread policy. For example: openstack flavor create --ram 512 --disk 1 --vcpus 2 \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:emulator_threads_policy=isolate' \ --property 'hw:cpu_realtime=true' \ --property 'hw:cpu_realtime_mask=^2' \ test.rt Similarly, they could ensure at least one core in the range is valid: openstack flavor create --ram 512 --disk 1 --vcpus 2 \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:emulator_threads_policy=isolate' \ --property 'hw:cpu_realtime=true' \ --property 'hw:cpu_realtime_mask=^1-5' \ test.rt However, both cases are still wrong and the 'hw:cpu_realtime_mask' value is almost certainly user error. Nova should be validating things properly and rejecting invalid values. we could probably also look at dropping the requirement to specify 'hw:cpu_realtime_mask' if 'hw:emulator_threads_policy' is configured, however, that's more of a feature than a bug. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1884231/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
