Public bug reported: Description =========== libvirt CPU power management does not handle `isolate` emulator thread policy.
Steps to reproduce ================== 1. Turn on libvirt CPU power management 2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate Expected result =============== After the execution of the steps above, what should have happened if the issue wasn't present? Actual result ============= The instance doesn't start, with the following libvirt error in the nova-compute log: Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last): Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise e.with_traceback(tb) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed') Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Environment =========== This was originally noticed in a whitebox CI job [1] on devstack master. Additional info =============== When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on. [1] https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs ** Affects: nova Importance: Undecided Status: In Progress ** Description changed: Description =========== libvirt CPU power management does not handle `isolate` emulator thread policy. Steps to reproduce ================== - 1. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate - + 1. Turn on libvirt CPU power management + 2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate Expected result =============== After the execution of the steps above, what should have happened if the issue wasn't present? Actual result ============= The instance doesn't start, with the following libvirt error in the nova-compute log: Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last): Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise e.with_traceback(tb) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed') Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument - Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest + Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Environment =========== This was originally noticed in a whitebox CI job [1] on devstack master. Additional info =============== When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on. [1] https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2056612 Title: libvirt CPU power management does not handle `isolate` emulator thread policy Status in OpenStack Compute (nova): In Progress Bug description: Description =========== libvirt CPU power management does not handle `isolate` emulator thread policy. Steps to reproduce ================== 1. Turn on libvirt CPU power management 2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate Expected result =============== After the execution of the steps above, what should have happened if the issue wasn't present? Actual result ============= The instance doesn't start, with the following libvirt error in the nova-compute log: Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last): Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise e.with_traceback(tb) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed') Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument Environment =========== This was originally noticed in a whitebox CI job [1] on devstack master. Additional info =============== When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on. [1] https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2056612/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp