Public bug reported: Description =========== After evacuation, nova-compute where VMs were evacuated could not restart emitting InstanceNotFound exception when init_host() called.
Steps to reproduce ================== 0) service nova-compute stop at 'compute01' 1) nova evacuate 'compute01' 2) nova delete 'server at compute01' 3) service nova-compute restart at 'compute01' Expected result =============== nova-compute normally started Actual result ============= nova-compute could not started Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ stable/ocata 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + kvm 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? Ceph 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) neutron with linux bridge Logs & Configs ============== 2019-02-26 16:24:29.015 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deleting instance files /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del 2019-02-26 16:24:29.017 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deletion of /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del complete 2019-02-26 16:24:29.020 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CALL msg_id: 9bbf4371e977450d98a519dac213592b exchange 'nova' topic 'conductor' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:550 2019-02-26 16:24:29.085 8901 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9bbf4371e977450d98a519dac213592b __call__ /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:407 2019-02-26 16:24:29.093 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CAST unique_id: d5a0d3b93fbf42b6a379b9507e9c2a9a FANOUT topic 'scheduler' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:539 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] Error starting thread. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last): 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service service.start() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/service.py", line 144, in start 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self.manager.init_host() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 1155, in init_host 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self._destroy_evacuated_instances(context) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 674, in _destroy_evacuated_instances 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service bdi, destroy_disks) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 915, in destroy 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service destroy_disks, migrate_data) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 1034, in cleanup 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service instance.save() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ctxt, self, fn.__name__, args, kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/rpcapi.py", line 244, in object_action 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service objmethod=objmethod, args=args, kwargs=kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=self.retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service timeout=timeout, retry=retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise result 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound_Remote: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last): 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/manager.py", line 93, in _object_dispatch 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return getattr(target, method)(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return fn(self, *args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/objects/instance.py", line 792, in save 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=_expected_cols(expected_attrs)) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/api.py", line 840, in instance_update_and_get_original 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service expected=expected) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ectxt.value = e.inner_exc 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(context, *args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 2714, in instance_update_and_get_original 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=columns_to_join) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 1910, in _instance_get_by_uuid 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise exception.InstanceNotFound(instance_id=uuid) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.102 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Acquired semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212 2019-02-26 16:24:29.103 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Releasing semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1817670 Title: nova-compute could not destroy evacuated instance which was deleted Status in OpenStack Compute (nova): New Bug description: Description =========== After evacuation, nova-compute where VMs were evacuated could not restart emitting InstanceNotFound exception when init_host() called. Steps to reproduce ================== 0) service nova-compute stop at 'compute01' 1) nova evacuate 'compute01' 2) nova delete 'server at compute01' 3) service nova-compute restart at 'compute01' Expected result =============== nova-compute normally started Actual result ============= nova-compute could not started Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ stable/ocata 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + kvm 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? Ceph 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) neutron with linux bridge Logs & Configs ============== 2019-02-26 16:24:29.015 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deleting instance files /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del 2019-02-26 16:24:29.017 8901 INFO nova.virt.libvirt.driver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] [instance: 25ab917a-1506-4154-9902-5b8b550ec929] Deletion of /var/lib/nova/instances/25ab917a-1506-4154-9902-5b8b550ec929_del complete 2019-02-26 16:24:29.020 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CALL msg_id: 9bbf4371e977450d98a519dac213592b exchange 'nova' topic 'conductor' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:550 2019-02-26 16:24:29.085 8901 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 9bbf4371e977450d98a519dac213592b __call__ /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:407 2019-02-26 16:24:29.093 8901 DEBUG oslo_messaging._drivers.amqpdriver [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] CAST unique_id: d5a0d3b93fbf42b6a379b9507e9c2a9a FANOUT topic 'scheduler' _send /opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:539 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service [req-dc2a5061-29aa-4097-9a3e-83de36804460 - - - - -] Error starting thread. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last): 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service service.start() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/service.py", line 144, in start 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self.manager.init_host() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 1155, in init_host 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service self._destroy_evacuated_instances(context) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/compute/manager.py", line 674, in _destroy_evacuated_instances 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service bdi, destroy_disks) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 915, in destroy 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service destroy_disks, migrate_data) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/virt/libvirt/driver.py", line 1034, in cleanup 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service instance.save() 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 210, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ctxt, self, fn.__name__, args, kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/rpcapi.py", line 244, in object_action 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service objmethod=objmethod, args=args, kwargs=kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=self.retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service timeout=timeout, retry=retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service retry=retry) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 557, in _send 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise result 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound_Remote: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service Traceback (most recent call last): 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/conductor/manager.py", line 93, in _object_dispatch 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return getattr(target, method)(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return fn(self, *args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/objects/instance.py", line 792, in save 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=_expected_cols(expected_attrs)) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/api.py", line 840, in instance_update_and_get_original 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service expected=expected) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 180, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service ectxt.value = e.inner_exc 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(*args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 251, in wrapped 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service return f(context, *args, **kwargs) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 2714, in instance_update_and_get_original 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service columns_to_join=columns_to_join) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service File "/opt/openstack/src/nova/nova/db/sqlalchemy/api.py", line 1910, in _instance_get_by_uuid 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service raise exception.InstanceNotFound(instance_id=uuid) 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service InstanceNotFound: Instance 25ab917a-1506-4154-9902-5b8b550ec929 could not be found. 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.097 8901 ERROR oslo_service.service 2019-02-26 16:24:29.102 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Acquired semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212 2019-02-26 16:24:29.103 8901 DEBUG oslo_concurrency.lockutils [req-67496f45-7809-41c4-a6a1-365574533016 - - - - -] Releasing semaphore "singleton_lock" lock /opt/openstack/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1817670/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
