Reviewed: https://review.opendev.org/c/openstack/nova/+/909795 Committed: https://opendev.org/openstack/nova/commit/0986d2bbe871d852075549eecea46f4b282c0842 Submitter: "Zuul (22348)" Branch: master
commit 0986d2bbe871d852075549eecea46f4b282c0842 Author: Artom Lifshitz <[email protected]> Date: Wed Feb 21 16:02:16 2024 -0500 Power on cores for isolated emulator threads Previously, with the `isolate` emulator threads policy and libvirt cpu power management enabled, we did not power on the cores to which the emulator threads were pin. Start doing that, and don't forget to power them down when the instance is stopped. Closes-bug: 2056612 Change-Id: I6e5383d8a0bf3f0ed8c870754cddae4e9163b4fd ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2056612 Title: libvirt CPU power management does not handle `isolate` emulator thread policy Status in OpenStack Compute (nova): Fix Released Bug description: Description =========== libvirt CPU power management does not handle `isolate` emulator thread policy. Steps to reproduce ================== 1. Turn on libvirt CPU power management 2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate Expected result =============== After the execution of the steps above, what should have happened if the issue wasn't present? Actual result ============= The instance doesn't start, with the following libvirt error in the nova-compute log: Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last): Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise e.with_traceback(tb) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed') Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Environment =========== This was originally noticed in a whitebox CI job [1] on devstack master. Additional info =============== When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on. [1] https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2056612/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
