Reviewed: https://review.openstack.org/543970 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6ba8a35825a7ec839b2d0aab7559351d573130ab Submitter: Zuul Branch: master
commit 6ba8a35825a7ec839b2d0aab7559351d573130ab Author: Dan Smith <[email protected]> Date: Tue Feb 13 07:16:57 2018 -0800 Lazy-load instance attributes with read_deleted=yes If we're doing a lazy-load of a generic attribute on instance, we should be using read_deleted=yes. Otherwise we just fail in the load process which is confusing and not helpful to a cleanup routine that needs to handle the deleted instance. This makes us load those things with read_deleted=yes. Change-Id: Ide6cc5bb1fce2c9aea9fa3efdf940e8308cd9ed0 Closes-Bug: #1745977 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1745977 Title: When source compute service up, will not destroy and clean up those instances which be evacuated then be deleted. Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: In Progress Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: In Progress Bug description: Description =========== When the instance evacuated to destination host successfully, then deleted this instance. The source host up will cleanup this instance failed. Steps to reproduce ================== 1.deploy a local instance in source host. 2.power off the source host. 3.evacuate the instance to destination host. 4.delete this instance. 5.power on the source host. Expected result =============== The source host nova-compute service cleanup this evacuated and deleted instance. Actual result ============= This instance still on source host. Environment =========== Openstack Pike Libvirt + KVM ovs network Logs & Configs ============== source host nova-compute log: 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service [req-7bdfe28f-0464-4af8-bdd0-2d433b25d84a - - - - -] Error starting thread.: InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch return getattr(target, method)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper result = fn(cls, context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid use_slave=use_slave) File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid columns_to_join=columns_to_join) File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped return f(context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid columns_to_join=columns_to_join) File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid raise exception.InstanceNotFound(instance_id=uuid) InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found. 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last): 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service service.start() 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 156, in start 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service self.manager.init_host() 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1173, in init_host 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service self._destroy_evacuated_instances(context) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 691, in _destroy_evacuated_instances 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service bdi, destroy_disks) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 909, in destroy 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service destroy_disks) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1032, in cleanup 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service attempts = int(instance.system_metadata.get('clean_attempts', 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service self.obj_load_attr(name) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1131, in obj_load_attr 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service self._load_generic(attrname) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 858, in _load_generic 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service expected_attrs=[attrname]) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service args, kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service args=args, kwargs=kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 167, in call 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service retry=self.retry) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service timeout=timeout, retry=retry) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service retry=retry) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service raise result 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound_Remote: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found. 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service Traceback (most recent call last): 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 125, in _object_dispatch 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service return getattr(target, method)(*args, **kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service result = fn(cls, context, *args, **kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 474, in get_by_uuid 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service use_slave=use_slave) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service return f(*args, **kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 466, in _db_instance_get_by_uuid 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service columns_to_join=columns_to_join) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 744, in instance_get_by_uuid 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 179, in wrapper 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service return f(*args, **kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service return f(context, *args, **kwargs) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1911, in instance_get_by_uuid 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service columns_to_join=columns_to_join) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1920, in _instance_get_by_uuid 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service raise exception.InstanceNotFound(instance_id=uuid) 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service 2018-01-29 10:28:48.664 9364 ERROR oslo_service.service InstanceNotFound: Instance 19022200-7abc-423d-90bd-e9dcd0887679 could not be found. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1745977/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
