Public bug reported: Seen here:
http://logs.openstack.org/43/635343/4/check/tempest-slow- py3/a2497ae/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_05_26_25_035103 Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [-] [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Post live migration at destination ubuntu-bionic-rax-iad-0003421804 failed: AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: res = self.dispatcher.dispatch(message) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return self._do_dispatch(endpoint, method, ctxt, args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: result = func(ctxt, **new_args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: function_name, call_dict, binary, tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return f(self, context, *args, **kw) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: kwargs['instance'], e, sys.exc_info()) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: context, instance, migration.dest_compute, migration=migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6846, in _post_live_migration Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance, block_migration, dest) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/rpcapi.py", line 716, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance=instance, block_migration=block_migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 178, in call Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=self.retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] call_monitor_timeout, retry=retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] raise result Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] res = self.dispatcher.dispatch(message) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return self._do_dispatch(endpoint, method, ctxt, args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] result = func(ctxt, **new_args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] function_name, call_dict, binary, tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return f(self, context, *args, **kw) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] kwargs['instance'], e, sys.exc_info()) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] context, instance, migration.dest_compute, migration=migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Apparently we don't hit this except block which would set the instance to ERROR state because I don't see that traceback and message in the logs: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6537 So we end up with something like this (failed tempest test): b'Details: (TestNetworkAdvancedServerOps:test_server_connectivity_live_migration) Server b78e10fb-44b8-4010-a72d-c86410950c15 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: MIGRATING. Current task state: migrating.' It looks like the error from post_live_migration_at_destination is logged and ignored here: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6844 And post_live_migration_at_destination is what sets the fields on the instance: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6971 So the task_state is never reset. This is a mess which will take some thought to clean up because if we get to _post_live_migration the guest is on the destination host, but the database is all messed up if post_live_migration_at_destination fails early. ** Affects: nova Importance: Medium Status: Triaged ** Tags: live-migration -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1818873 Title: When post_live_migration_at_destination fails the instance is not put into ERROR/None vm_state/task_state Status in OpenStack Compute (nova): Triaged Bug description: Seen here: http://logs.openstack.org/43/635343/4/check/tempest-slow- py3/a2497ae/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_05_26_25_035103 Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [-] [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Post live migration at destination ubuntu-bionic-rax-iad-0003421804 failed: AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: res = self.dispatcher.dispatch(message) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return self._do_dispatch(endpoint, method, ctxt, args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: result = func(ctxt, **new_args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: function_name, call_dict, binary, tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return f(self, context, *args, **kw) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: kwargs['instance'], e, sys.exc_info()) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: context, instance, migration.dest_compute, migration=migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6846, in _post_live_migration Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance, block_migration, dest) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/rpcapi.py", line 716, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] instance=instance, block_migration=block_migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/client.py", line 178, in call Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=self.retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/transport.py", line 128, in _send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] retry=retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 645, in send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] call_monitor_timeout, retry=retry) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 636, in _send Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] raise result Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Traceback (most recent call last): Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] res = self.dispatcher.dispatch(message) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return self._do_dispatch(endpoint, method, ctxt, args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] result = func(ctxt, **new_args) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 79, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] function_name, call_dict, binary, tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return f(self, context, *args, **kw) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/utils.py", line 1301, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 212, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] kwargs['instance'], e, sys.exc_info()) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] self.force_reraise() Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] six.reraise(self.type_, self.value, self.tb) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] return function(self, context, *args, **kwargs) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/compute/manager.py", line 6942, in post_live_migration_at_destination Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2815, in migrate_instance_finish Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] context, instance, migration.dest_compute, migration=migration) Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] AttributeError: 'dict' object has no attribute 'dest_compute' Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Mar 06 05:26:25.035103 ubuntu-bionic-rax-iad-0003421801 nova-compute[20242]: ERROR nova.compute.manager [instance: b78e10fb-44b8-4010-a72d-c86410950c15] Apparently we don't hit this except block which would set the instance to ERROR state because I don't see that traceback and message in the logs: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6537 So we end up with something like this (failed tempest test): b'Details: (TestNetworkAdvancedServerOps:test_server_connectivity_live_migration) Server b78e10fb-44b8-4010-a72d-c86410950c15 failed to reach ACTIVE status and task state "None" within the required time (196 s). Current status: MIGRATING. Current task state: migrating.' It looks like the error from post_live_migration_at_destination is logged and ignored here: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6844 And post_live_migration_at_destination is what sets the fields on the instance: https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/compute/manager.py#L6971 So the task_state is never reset. This is a mess which will take some thought to clean up because if we get to _post_live_migration the guest is on the destination host, but the database is all messed up if post_live_migration_at_destination fails early. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1818873/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
