Public bug reported: Description =========== In tripleo, we have a ci test that tries to delete an instance while it's building. Basically, we make sure that n-cpu can't boot it, so n-cpu destroys and recreate it all the time, until n-api receives a delete.
When the delete happens in between spawn from n-cpu, it's failing. https://bugs.launchpad.net/tripleo/+bug/1914174 comment #6 has all the details. Steps to reproduce ================== * Create a server * Delete right between the moment where n-cpu destroys it and respawns it. Expected result =============== We should delete Actual result ============= This traceback [1] [1] ~~~ nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi [req-c29af162-9e0a-4c1d-b55f-c11470e412ba 629d59dfb295418a93fbc75ef8a4c36e 2ea8047d0ed14eb48b8a6ee5070df67a - default default] Unexpected exception in API method: AttributeError: 'NoneType' object has no attribute 'uuid' nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 655, in destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi hard_delete=hard_delete) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 549, in instance_destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi hard_delete=hard_delete) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi ectxt.value = e.inner_exc nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self.force_reraise() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi raise value nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 222, in wrapped nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 1166, in instance_destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi raise exception.ConstraintNotMet() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova.exception.ConstraintNotMet: Constraint not met. nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred: nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2170, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi if self._delete_while_booting(context, instance): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2112, in _delete_while_booting nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi instance.destroy() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 659, in destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi reason='host changed') nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova.exception.ObjectActionError: Object action destroy failed because: host changed nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred: nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 993, in delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._delete(req.environ['nova.context'], req, id) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 798, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self.compute_api.delete(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 225, in inner nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 152, in inner nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2492, in delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._delete_instance(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2484, in _delete_instance nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi task_state=task_states.DELETING) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2207, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._local_delete_cleanup(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2124, in _local_delete_cleanup nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi context, instance.uuid) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi AttributeError: 'NoneType' object has no attribute 'uuid' nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi ~~~ ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1914777 Title: Possible race condition between n-cpu and n-api when deleting a building instance Status in OpenStack Compute (nova): New Bug description: Description =========== In tripleo, we have a ci test that tries to delete an instance while it's building. Basically, we make sure that n-cpu can't boot it, so n-cpu destroys and recreate it all the time, until n-api receives a delete. When the delete happens in between spawn from n-cpu, it's failing. https://bugs.launchpad.net/tripleo/+bug/1914174 comment #6 has all the details. Steps to reproduce ================== * Create a server * Delete right between the moment where n-cpu destroys it and respawns it. Expected result =============== We should delete Actual result ============= This traceback [1] [1] ~~~ nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi [req-c29af162-9e0a-4c1d-b55f-c11470e412ba 629d59dfb295418a93fbc75ef8a4c36e 2ea8047d0ed14eb48b8a6ee5070df67a - default default] Unexpected exception in API method: AttributeError: 'NoneType' object has no attribute 'uuid' nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 655, in destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi hard_delete=hard_delete) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/api.py", line 549, in instance_destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi hard_delete=hard_delete) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi ectxt.value = e.inner_exc nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self.force_reraise() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi raise value nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 222, in wrapped nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(context, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 1166, in instance_destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi raise exception.ConstraintNotMet() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova.exception.ConstraintNotMet: Constraint not met. nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred: nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2170, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi if self._delete_while_booting(context, instance): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2112, in _delete_while_booting nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi instance.destroy() nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return fn(self, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/objects/instance.py", line 659, in destroy nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi reason='host changed') nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova.exception.ObjectActionError: Object action destroy failed because: host changed nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi During handling of the above exception, another exception occurred: nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi Traceback (most recent call last): nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/wsgi.py", line 671, in wrapped nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 993, in delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._delete(req.environ['nova.context'], req, id) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/api/openstack/compute/servers.py", line 798, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self.compute_api.delete(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 225, in inner nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return function(self, context, instance, *args, **kwargs) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 152, in inner nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2492, in delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._delete_instance(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2484, in _delete_instance nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi task_state=task_states.DELETING) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2207, in _delete nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi self._local_delete_cleanup(context, instance) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.6/site-packages/nova/compute/api.py", line 2124, in _local_delete_cleanup nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi context, instance.uuid) nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi AttributeError: 'NoneType' object has no attribute 'uuid' nova-api.log.1:2021-02-02 08:51:20.093 19 ERROR nova.api.openstack.wsgi ~~~ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1914777/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
