** Changed in: nova/victoria
Status: In Progress => Fix Released
** Changed in: nova/ussuri
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1914777
Title:
Possible race condition between n-cpu and n-api when deleting a
building instance
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) ussuri series:
Fix Released
Status in OpenStack Compute (nova) victoria series:
Fix Released
Bug description:
Description
===========
In tripleo, we have a ci test that tries to delete an instance while it's
building. Basically, we make sure that n-cpu can't boot it, so n-cpu destroys
and recreate it all the time, until n-api receives a delete.
When the delete happens in between spawn from n-cpu, it's failing.
https://bugs.launchpad.net/tripleo/+bug/1914174 comment #6 has all the
details.
Steps to reproduce
==================
* Create a server
* Delete right between the moment where n-cpu destroys it and respawns it.
Expected result
===============
We should delete
Actual result
=============
This traceback [1]
[1]
~~~
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
[req-c29af162-9e0a-4c1d-b55f-c11470e412ba 629d59dfb295418a93fbc75ef8a4c36e
2ea8047d0ed14eb48b8a6ee5070df67a - default default] Unexpected exception in API
method: AttributeError: 'NoneType' object has no attribute 'uuid'
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
Traceback (most recent call last):
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 655, in
destroy
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
hard_delete=hard_delete)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 549, in
instance_destroy
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
hard_delete=hard_delete)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 179, in
wrapper
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return f(*args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
ectxt.value = e.inner_exc
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in
__exit__
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
self.force_reraise()
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
six.reraise(self.type_, self.value, self.tb)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
raise value
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return f(*args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 222, in
wrapped
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return f(context, *args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 1166,
in instance_destroy
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
raise exception.ConstraintNotMet()
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova.exception.ConstraintNotMet: Constraint not met.
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
During handling of the above exception, another exception occurred:
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
Traceback (most recent call last):
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2170, in
_delete
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
if self._delete_while_booting(context, instance):
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2112, in
_delete_while_booting
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
instance.destroy()
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line
226, in wrapper
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return fn(self, *args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 659, in
destroy
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
reason='host changed')
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova.exception.ObjectActionError: Object action destroy failed because: host
changed
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
During handling of the above exception, another exception occurred:
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
Traceback (most recent call last):
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 671,
in wrapped
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return f(*args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py",
line 993, in delete
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
self._delete(req.environ['nova.context'], req, id)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py",
line 798, in _delete
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
self.compute_api.delete(context, instance)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 225, in inner
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return function(self, context, instance, *args, **kwargs)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 152, in inner
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
return f(self, context, instance, *args, **kw)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2492, in
delete
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
self._delete_instance(context, instance)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2484, in
_delete_instance
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
task_state=task_states.DELETING)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2207, in
_delete
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
self._local_delete_cleanup(context, instance)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2124, in
_local_delete_cleanup
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
context, instance.uuid)
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
AttributeError: 'NoneType' object has no attribute 'uuid'
nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi
~~~
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1914777/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
