Public bug reported: I noticed this in a stable/pike functional test job run:
http://logs.openstack.org/46/560146/2/check/nova-tox-functional/4a9d1fd /job-output.txt.gz#_2018-04-10_21_37_20_943583 2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr: 2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~ 2018-04-10 21:37:20.945029 | ubuntu-xenial | Traceback (most recent call last): 2018-04-10 21:37:20.945231 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers 2018-04-10 21:37:20.945268 | ubuntu-xenial | timer() 2018-04-10 21:37:20.945467 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__ 2018-04-10 21:37:20.945513 | ubuntu-xenial | cb(*args, **kw) 2018-04-10 21:37:20.945598 | ubuntu-xenial | File "nova/utils.py", line 1030, in context_wrapper 2018-04-10 21:37:20.945650 | ubuntu-xenial | func(*args, **kwargs) 2018-04-10 21:37:20.945756 | ubuntu-xenial | File "nova/compute/manager.py", line 5620, in dispatch_live_migration 2018-04-10 21:37:20.945839 | ubuntu-xenial | self._do_live_migration(*args, **kwargs) 2018-04-10 21:37:20.945939 | ubuntu-xenial | File "nova/compute/manager.py", line 5599, in _do_live_migration 2018-04-10 21:37:20.945993 | ubuntu-xenial | clean_task_state=True) 2018-04-10 21:37:20.946194 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.946246 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.946452 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.946532 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.946679 | ubuntu-xenial | File "nova/compute/manager.py", line 5588, in _do_live_migration 2018-04-10 21:37:20.946764 | ubuntu-xenial | block_migration, migrate_data) 2018-04-10 21:37:20.946856 | ubuntu-xenial | File "nova/virt/fake.py", line 497, in live_migration 2018-04-10 21:37:20.946901 | ubuntu-xenial | migrate_data) 2018-04-10 21:37:20.947003 | ubuntu-xenial | File "nova/exception_wrapper.py", line 76, in wrapped 2018-04-10 21:37:20.947069 | ubuntu-xenial | function_name, call_dict, binary) 2018-04-10 21:37:20.947270 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.947322 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.947536 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.947619 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.947707 | ubuntu-xenial | File "nova/exception_wrapper.py", line 67, in wrapped 2018-04-10 21:37:20.947779 | ubuntu-xenial | return f(self, context, *args, **kw) 2018-04-10 21:37:20.947878 | ubuntu-xenial | File "nova/compute/manager.py", line 218, in decorated_function 2018-04-10 21:37:20.947950 | ubuntu-xenial | kwargs['instance'], e, sys.exc_info()) 2018-04-10 21:37:20.948160 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.948214 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.948433 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.948510 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.948607 | ubuntu-xenial | File "nova/compute/manager.py", line 206, in decorated_function 2018-04-10 21:37:20.948687 | ubuntu-xenial | return function(self, context, *args, **kwargs) 2018-04-10 21:37:20.948802 | ubuntu-xenial | File "nova/compute/manager.py", line 5824, in _post_live_migration 2018-04-10 21:37:20.948856 | ubuntu-xenial | instance, source_node) 2018-04-10 21:37:20.949000 | ubuntu-xenial | File "nova/compute/resource_tracker.py", line 1294, in delete_allocation_for_migrated_instance 2018-04-10 21:37:20.949111 | ubuntu-xenial | self._delete_allocation_for_moved_instance(instance, node, 'migrated') 2018-04-10 21:37:20.949244 | ubuntu-xenial | File "nova/compute/resource_tracker.py", line 1302, in _delete_allocation_for_moved_instance 2018-04-10 21:37:20.949319 | ubuntu-xenial | cn_uuid = self.compute_nodes[node].uuid 2018-04-10 21:37:20.949363 | ubuntu-xenial | KeyError: u'host3' But I see it's also happening on master: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d ** Affects: nova Importance: Medium Status: Confirmed ** Tags: functional testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1762876 Title: KeyError during move operation functional tests Status in OpenStack Compute (nova): Confirmed Bug description: I noticed this in a stable/pike functional test job run: http://logs.openstack.org/46/560146/2/check/nova-tox- functional/4a9d1fd/job-output.txt.gz#_2018-04-10_21_37_20_943583 2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr: 2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~ 2018-04-10 21:37:20.945029 | ubuntu-xenial | Traceback (most recent call last): 2018-04-10 21:37:20.945231 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers 2018-04-10 21:37:20.945268 | ubuntu-xenial | timer() 2018-04-10 21:37:20.945467 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__ 2018-04-10 21:37:20.945513 | ubuntu-xenial | cb(*args, **kw) 2018-04-10 21:37:20.945598 | ubuntu-xenial | File "nova/utils.py", line 1030, in context_wrapper 2018-04-10 21:37:20.945650 | ubuntu-xenial | func(*args, **kwargs) 2018-04-10 21:37:20.945756 | ubuntu-xenial | File "nova/compute/manager.py", line 5620, in dispatch_live_migration 2018-04-10 21:37:20.945839 | ubuntu-xenial | self._do_live_migration(*args, **kwargs) 2018-04-10 21:37:20.945939 | ubuntu-xenial | File "nova/compute/manager.py", line 5599, in _do_live_migration 2018-04-10 21:37:20.945993 | ubuntu-xenial | clean_task_state=True) 2018-04-10 21:37:20.946194 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.946246 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.946452 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.946532 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.946679 | ubuntu-xenial | File "nova/compute/manager.py", line 5588, in _do_live_migration 2018-04-10 21:37:20.946764 | ubuntu-xenial | block_migration, migrate_data) 2018-04-10 21:37:20.946856 | ubuntu-xenial | File "nova/virt/fake.py", line 497, in live_migration 2018-04-10 21:37:20.946901 | ubuntu-xenial | migrate_data) 2018-04-10 21:37:20.947003 | ubuntu-xenial | File "nova/exception_wrapper.py", line 76, in wrapped 2018-04-10 21:37:20.947069 | ubuntu-xenial | function_name, call_dict, binary) 2018-04-10 21:37:20.947270 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.947322 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.947536 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.947619 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.947707 | ubuntu-xenial | File "nova/exception_wrapper.py", line 67, in wrapped 2018-04-10 21:37:20.947779 | ubuntu-xenial | return f(self, context, *args, **kw) 2018-04-10 21:37:20.947878 | ubuntu-xenial | File "nova/compute/manager.py", line 218, in decorated_function 2018-04-10 21:37:20.947950 | ubuntu-xenial | kwargs['instance'], e, sys.exc_info()) 2018-04-10 21:37:20.948160 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-04-10 21:37:20.948214 | ubuntu-xenial | self.force_reraise() 2018-04-10 21:37:20.948433 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-04-10 21:37:20.948510 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2018-04-10 21:37:20.948607 | ubuntu-xenial | File "nova/compute/manager.py", line 206, in decorated_function 2018-04-10 21:37:20.948687 | ubuntu-xenial | return function(self, context, *args, **kwargs) 2018-04-10 21:37:20.948802 | ubuntu-xenial | File "nova/compute/manager.py", line 5824, in _post_live_migration 2018-04-10 21:37:20.948856 | ubuntu-xenial | instance, source_node) 2018-04-10 21:37:20.949000 | ubuntu-xenial | File "nova/compute/resource_tracker.py", line 1294, in delete_allocation_for_migrated_instance 2018-04-10 21:37:20.949111 | ubuntu-xenial | self._delete_allocation_for_moved_instance(instance, node, 'migrated') 2018-04-10 21:37:20.949244 | ubuntu-xenial | File "nova/compute/resource_tracker.py", line 1302, in _delete_allocation_for_moved_instance 2018-04-10 21:37:20.949319 | ubuntu-xenial | cn_uuid = self.compute_nodes[node].uuid 2018-04-10 21:37:20.949363 | ubuntu-xenial | KeyError: u'host3' But I see it's also happening on master: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1762876/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
