Public bug reported:

I noticed this in a stable/pike functional test job run:

http://logs.openstack.org/46/560146/2/check/nova-tox-functional/4a9d1fd
/job-output.txt.gz#_2018-04-10_21_37_20_943583

2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
2018-04-10 21:37:20.945029 | ubuntu-xenial |     Traceback (most recent call 
last):
2018-04-10 21:37:20.945231 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py",
 line 457, in fire_timers
2018-04-10 21:37:20.945268 | ubuntu-xenial |         timer()
2018-04-10 21:37:20.945467 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py",
 line 58, in __call__
2018-04-10 21:37:20.945513 | ubuntu-xenial |         cb(*args, **kw)
2018-04-10 21:37:20.945598 | ubuntu-xenial |       File "nova/utils.py", line 
1030, in context_wrapper
2018-04-10 21:37:20.945650 | ubuntu-xenial |         func(*args, **kwargs)
2018-04-10 21:37:20.945756 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5620, in dispatch_live_migration
2018-04-10 21:37:20.945839 | ubuntu-xenial |         
self._do_live_migration(*args, **kwargs)
2018-04-10 21:37:20.945939 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5599, in _do_live_migration
2018-04-10 21:37:20.945993 | ubuntu-xenial |         clean_task_state=True)
2018-04-10 21:37:20.946194 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
2018-04-10 21:37:20.946246 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.946452 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
2018-04-10 21:37:20.946532 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
2018-04-10 21:37:20.946679 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5588, in _do_live_migration
2018-04-10 21:37:20.946764 | ubuntu-xenial |         block_migration, 
migrate_data)
2018-04-10 21:37:20.946856 | ubuntu-xenial |       File "nova/virt/fake.py", 
line 497, in live_migration
2018-04-10 21:37:20.946901 | ubuntu-xenial |         migrate_data)
2018-04-10 21:37:20.947003 | ubuntu-xenial |       File 
"nova/exception_wrapper.py", line 76, in wrapped
2018-04-10 21:37:20.947069 | ubuntu-xenial |         function_name, call_dict, 
binary)
2018-04-10 21:37:20.947270 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
2018-04-10 21:37:20.947322 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.947536 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
2018-04-10 21:37:20.947619 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
2018-04-10 21:37:20.947707 | ubuntu-xenial |       File 
"nova/exception_wrapper.py", line 67, in wrapped
2018-04-10 21:37:20.947779 | ubuntu-xenial |         return f(self, context, 
*args, **kw)
2018-04-10 21:37:20.947878 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 218, in decorated_function
2018-04-10 21:37:20.947950 | ubuntu-xenial |         kwargs['instance'], e, 
sys.exc_info())
2018-04-10 21:37:20.948160 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
2018-04-10 21:37:20.948214 | ubuntu-xenial |         self.force_reraise()
2018-04-10 21:37:20.948433 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
2018-04-10 21:37:20.948510 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
2018-04-10 21:37:20.948607 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 206, in decorated_function
2018-04-10 21:37:20.948687 | ubuntu-xenial |         return function(self, 
context, *args, **kwargs)
2018-04-10 21:37:20.948802 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5824, in _post_live_migration
2018-04-10 21:37:20.948856 | ubuntu-xenial |         instance, source_node)
2018-04-10 21:37:20.949000 | ubuntu-xenial |       File 
"nova/compute/resource_tracker.py", line 1294, in 
delete_allocation_for_migrated_instance
2018-04-10 21:37:20.949111 | ubuntu-xenial |         
self._delete_allocation_for_moved_instance(instance, node, 'migrated')
2018-04-10 21:37:20.949244 | ubuntu-xenial |       File 
"nova/compute/resource_tracker.py", line 1302, in 
_delete_allocation_for_moved_instance
2018-04-10 21:37:20.949319 | ubuntu-xenial |         cn_uuid = 
self.compute_nodes[node].uuid
2018-04-10 21:37:20.949363 | ubuntu-xenial |     KeyError: u'host3'

But I see it's also happening on master:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: functional testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1762876

Title:
  KeyError during move operation functional tests

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  I noticed this in a stable/pike functional test job run:

  http://logs.openstack.org/46/560146/2/check/nova-tox-
  functional/4a9d1fd/job-output.txt.gz#_2018-04-10_21_37_20_943583

  2018-04-10 21:37:20.944928 | ubuntu-xenial | Captured stderr:
  2018-04-10 21:37:20.944966 | ubuntu-xenial | ~~~~~~~~~~~~~~~~
  2018-04-10 21:37:20.945029 | ubuntu-xenial |     Traceback (most recent call 
last):
  2018-04-10 21:37:20.945231 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/hub.py",
 line 457, in fire_timers
  2018-04-10 21:37:20.945268 | ubuntu-xenial |         timer()
  2018-04-10 21:37:20.945467 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/eventlet/hubs/timer.py",
 line 58, in __call__
  2018-04-10 21:37:20.945513 | ubuntu-xenial |         cb(*args, **kw)
  2018-04-10 21:37:20.945598 | ubuntu-xenial |       File "nova/utils.py", line 
1030, in context_wrapper
  2018-04-10 21:37:20.945650 | ubuntu-xenial |         func(*args, **kwargs)
  2018-04-10 21:37:20.945756 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5620, in dispatch_live_migration
  2018-04-10 21:37:20.945839 | ubuntu-xenial |         
self._do_live_migration(*args, **kwargs)
  2018-04-10 21:37:20.945939 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5599, in _do_live_migration
  2018-04-10 21:37:20.945993 | ubuntu-xenial |         clean_task_state=True)
  2018-04-10 21:37:20.946194 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
  2018-04-10 21:37:20.946246 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.946452 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
  2018-04-10 21:37:20.946532 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
  2018-04-10 21:37:20.946679 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5588, in _do_live_migration
  2018-04-10 21:37:20.946764 | ubuntu-xenial |         block_migration, 
migrate_data)
  2018-04-10 21:37:20.946856 | ubuntu-xenial |       File "nova/virt/fake.py", 
line 497, in live_migration
  2018-04-10 21:37:20.946901 | ubuntu-xenial |         migrate_data)
  2018-04-10 21:37:20.947003 | ubuntu-xenial |       File 
"nova/exception_wrapper.py", line 76, in wrapped
  2018-04-10 21:37:20.947069 | ubuntu-xenial |         function_name, 
call_dict, binary)
  2018-04-10 21:37:20.947270 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
  2018-04-10 21:37:20.947322 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.947536 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
  2018-04-10 21:37:20.947619 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
  2018-04-10 21:37:20.947707 | ubuntu-xenial |       File 
"nova/exception_wrapper.py", line 67, in wrapped
  2018-04-10 21:37:20.947779 | ubuntu-xenial |         return f(self, context, 
*args, **kw)
  2018-04-10 21:37:20.947878 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 218, in decorated_function
  2018-04-10 21:37:20.947950 | ubuntu-xenial |         kwargs['instance'], e, 
sys.exc_info())
  2018-04-10 21:37:20.948160 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 220, in __exit__
  2018-04-10 21:37:20.948214 | ubuntu-xenial |         self.force_reraise()
  2018-04-10 21:37:20.948433 | ubuntu-xenial |       File 
"/home/zuul/src/git.openstack.org/openstack/nova/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py",
 line 196, in force_reraise
  2018-04-10 21:37:20.948510 | ubuntu-xenial |         six.reraise(self.type_, 
self.value, self.tb)
  2018-04-10 21:37:20.948607 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 206, in decorated_function
  2018-04-10 21:37:20.948687 | ubuntu-xenial |         return function(self, 
context, *args, **kwargs)
  2018-04-10 21:37:20.948802 | ubuntu-xenial |       File 
"nova/compute/manager.py", line 5824, in _post_live_migration
  2018-04-10 21:37:20.948856 | ubuntu-xenial |         instance, source_node)
  2018-04-10 21:37:20.949000 | ubuntu-xenial |       File 
"nova/compute/resource_tracker.py", line 1294, in 
delete_allocation_for_migrated_instance
  2018-04-10 21:37:20.949111 | ubuntu-xenial |         
self._delete_allocation_for_moved_instance(instance, node, 'migrated')
  2018-04-10 21:37:20.949244 | ubuntu-xenial |       File 
"nova/compute/resource_tracker.py", line 1302, in 
_delete_allocation_for_moved_instance
  2018-04-10 21:37:20.949319 | ubuntu-xenial |         cn_uuid = 
self.compute_nodes[node].uuid
  2018-04-10 21:37:20.949363 | ubuntu-xenial |     KeyError: u'host3'

  But I see it's also happening on master:

  
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22cn_uuid%20%3D%20self.compute_nodes%5Bnode%5D.uuid%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1762876/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to