Public bug reported: Reproduced from rhbz#1686511 (https://bugzilla.redhat.com/show_bug.cgi?id=1686511)
When spawning an Openstack instance, this error is received: 2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming res = self.dispatcher.dispatch(message) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch result = func(ctxt, **new_args) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations dests = self.driver.select_destinations(ctxt, spec_obj) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations selected_hosts = self._schedule(context, spec_obj) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule spec_obj, index=num) File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts hosts, spec_obj, index) File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects list_objs = list(objs) File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all if self._filter_one(obj, spec_obj): File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one return self.host_passes(obj, spec) File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes pci_stats=host_state.pci_stats)) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host host_cell, instance_cell, limits) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell host_cell, instance_cell) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning max(map(len, host_cell.siblings))) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores itertools.chain(*sibling_set))) UnboundLocalError: local variable 'sibling_set' referenced before assignment 2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req- e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f- 8b7a-f2f3bc168b29] Setting instance to ERROR state. This issues appears to be because of: https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875 This works normally because of loop variables in Python are available outside of the scope of the loop: >>> for x in range(5): ... pass ... >>> print(x) 4 and because there's usually something in sibling_sets. However, this is presumably failing for this user because there are no free cores at all on the given host. This is likely the race condition between the nova- scheduler and nova-compute services. ** Affects: nova Importance: Undecided Status: New ** Description changed: - Reproduced from rhbz#1686511. + Reproduced from rhbz#1686511 + (https://bugzilla.redhat.com/show_bug.cgi?id=1686511) When spawning an Openstack instance, this error is received: + 2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment + Traceback (most recent call last): - 2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment - Traceback (most recent call last): + File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming + res = self.dispatcher.dispatch(message) - File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming - res = self.dispatcher.dispatch(message) + File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch + return self._do_dispatch(endpoint, method, ctxt, args) - File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch - return self._do_dispatch(endpoint, method, ctxt, args) + File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch + result = func(ctxt, **new_args) - File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch - result = func(ctxt, **new_args) + File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner + return func(*args, **kwargs) - File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner - return func(*args, **kwargs) + File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations + dests = self.driver.select_destinations(ctxt, spec_obj) - File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations - dests = self.driver.select_destinations(ctxt, spec_obj) + File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations + selected_hosts = self._schedule(context, spec_obj) - File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations - selected_hosts = self._schedule(context, spec_obj) + File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule + spec_obj, index=num) - File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule - spec_obj, index=num) + File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts + hosts, spec_obj, index) - File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts - hosts, spec_obj, index) + File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects + list_objs = list(objs) - File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects - list_objs = list(objs) + File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all + if self._filter_one(obj, spec_obj): - File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all - if self._filter_one(obj, spec_obj): + File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one + return self.host_passes(obj, spec) - File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one - return self.host_passes(obj, spec) + File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes + pci_stats=host_state.pci_stats)) - File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes - pci_stats=host_state.pci_stats)) + File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host + host_cell, instance_cell, limits) - File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host - host_cell, instance_cell, limits) + File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell + host_cell, instance_cell) - File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell - host_cell, instance_cell) + File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning + max(map(len, host_cell.siblings))) - File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning - max(map(len, host_cell.siblings))) + File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores + itertools.chain(*sibling_set))) - File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores - itertools.chain(*sibling_set))) - - UnboundLocalError: local variable 'sibling_set' referenced before + UnboundLocalError: local variable 'sibling_set' referenced before assignment - 2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req- + 2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req- e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f- 8b7a-f2f3bc168b29] Setting instance to ERROR state. - This issues appears to be because of: https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875 This works normally because of loop variables in Python are available outside of the scope of the loop: - >>> for x in range(5): - ... pass - ... - >>> print(x) - 4 + >>> for x in range(5): + ... pass + ... + >>> print(x) + 4 and because there's usually something in sibling_sets. However, this is presumably failing for this user because there are no free cores at all on the given host. This is likely the race condition between the nova- scheduler and nova-compute services. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1821733 Title: Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment Status in OpenStack Compute (nova): New Bug description: Reproduced from rhbz#1686511 (https://bugzilla.redhat.com/show_bug.cgi?id=1686511) When spawning an Openstack instance, this error is received: 2019-03-07 08:07:38.499 3124 WARNING nova.scheduler.utils [req-e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] Failed to compute_task_build_instances: local variable 'sibling_set' referenced before assignment Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming res = self.dispatcher.dispatch(message) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch result = func(ctxt, **new_args) File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations dests = self.driver.select_destinations(ctxt, spec_obj) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 53, in select_destinations selected_hosts = self._schedule(context, spec_obj) File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 113, in _schedule spec_obj, index=num) File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 576, in get_filtered_hosts hosts, spec_obj, index) File "/usr/lib/python2.7/site-packages/nova/filters.py", line 89, in get_filtered_objects list_objs = list(objs) File "/usr/lib/python2.7/site-packages/nova/filters.py", line 44, in filter_all if self._filter_one(obj, spec_obj): File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/__init__.py", line 44, in _filter_one return self.host_passes(obj, spec) File "/usr/lib/python2.7/site-packages/nova/scheduler/filters/numa_topology_filter.py", line 123, in host_passes pci_stats=host_state.pci_stats)) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 1297, in numa_fit_instance_to_host host_cell, instance_cell, limits) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 906, in _numa_fit_instance_cell host_cell, instance_cell) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 854, in _numa_fit_instance_cell_with_pinning max(map(len, host_cell.siblings))) File "/usr/lib/python2.7/site-packages/nova/virt/hardware.py", line 805, in _pack_instance_onto_cores itertools.chain(*sibling_set))) UnboundLocalError: local variable 'sibling_set' referenced before assignment 2019-03-07 08:07:38.500 3124 WARNING nova.scheduler.utils [req- e577cf31-7a58-420f-8ba5-3f962569ab08 0c90c8d8b42c42e883d2135cc733cac4 8b869a98a43e4fc48001e0ff6d149fe6 - - -] [instance: 5bca186a-5a36-4b0f- 8b7a-f2f3bc168b29] Setting instance to ERROR state. This issues appears to be because of: https://github.com/openstack/nova/blob/da9f9c962fe00dbfc9c8fe9c47e964816d67b773/nova/virt/hardware.py#L875 This works normally because of loop variables in Python are available outside of the scope of the loop: >>> for x in range(5): ... pass ... >>> print(x) 4 and because there's usually something in sibling_sets. However, this is presumably failing for this user because there are no free cores at all on the given host. This is likely the race condition between the nova-scheduler and nova-compute services. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1821733/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
