*** This bug is a duplicate of bug 1397153 ***
https://bugs.launchpad.net/bugs/1397153
** This bug has been marked a duplicate of bug 1397153
instance stuck at migrating state when some unexpected error happened
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1412279
Title:
live migration, nova-conductor raise a exception named
'MigrationError', but nova-api can not receive any error reply except
a 'MessageTimeOut' exception occurs.
Status in OpenStack Compute (Nova):
Fix Committed
Bug description:
live migration, nova-api wait for a reply of rpc_call to nova-
conductor, nova-conductor raise a exception named 'MigrationError'.
The nova-api service can not receive any error reply from nova-
conductor service, and a 'MessageTimeOut' exception occurs until 60
seconds passed.
nova-api.log
2015-03-20 09:23:39.066 20276 ERROR
nova.api.openstack.compute.contrib.admin_actions
[req-3d2de7b2-5258-4ba0-935d-48eb84c877fc 7aba40236a4c4494aba4eb0b9365ffee
be568b8239d147e58f1ef16b6011c93d] Live migration of instance
ae701ba0-1eed-4d0f-a0c3-052b1f433553 to another host failed
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call
last):
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/admin_actions.py",
line 338, in _migrate_live
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions disk_over_commit, host)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 238, in _wrapped
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions return fn(self, context,
instance, *args, **kwargs)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 192, in inner
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions return f(self, context,
instance, *args, **kw)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3233, in
live_migrate
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions
disk_over_commit=disk_over_commit)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 387, in
live_migrate_instance
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions block_migration,
disk_over_commit, None)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 476, in
migrate_server
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions reservations=reservations)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in
call
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions wait_for_reply=True,
timeout=timeout)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in
_send
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions timeout=timeout)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
412, in send
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions return self._send(target,
ctxt, message, wait_for_reply, timeout)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
403, in _send
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions result =
self._waiter.wait(msg_id, timeout)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
267, in wait
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions reply, ending =
self._poll_connection(msg_id, timeout)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions File
"/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line
217, in _poll_connection
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions % msg_id)
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions MessagingTimeout: Timed out
waiting for a reply to message ID 20ee328ddd9446bda761fc162fb00d50
2015-03-20 09:23:39.066 20276 TRACE
nova.api.openstack.compute.contrib.admin_actions
nova-conductor.log
015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122,
in _do_dispatch
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher result
= getattr(endpoint, method)(ctxt, **new_args)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in
inner
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
func(*args, **kwargs)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in
migrate_server
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
block_migration, disk_over_commit)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 793, in
_live_migrate
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher raise
exception.MigrationError(reason=ex)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
MigrationError: Migration error: [Errno [Errno 2] No such file or directory:
'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122,
in _do_dispatch
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher result
= getattr(endpoint, method)(ctxt, **new_args)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher payload)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373, in
decorated_function
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in
decorated_function
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684, in
check_can_live_migrate_source
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
dest_check_data)
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4610, in
check_can_live_migrate_source
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
jsonutils.loads(self.get_instance_disk_info(instance['name'])))
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5293, in
get_instance_disk_info
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher dk_size
= int(os.path.getsize(path))
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.7/genericpath.py", line 49, in getsize
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher return
os.stat(filename).st_size
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher OSError:
[Errno 2] No such file or directory:
'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher ] No such
file or directory
2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher
2015-03-20 09:22:39.743 13849 ERROR oslo.messaging._drivers.common [-]
Returning exception Migration error: [Errno [Errno 2] No such file or
directory:
'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply
incoming.message))
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in
wrapped
payload)
File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in
wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373,
in decorated_function
e, sys.exc_info())
File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684,
in check_can_live_migrate_source
dest_check_data)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
4610, in check_can_live_migrate_source
jsonutils.loads(self.get_instance_disk_info(instance['name'])))
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5293, in get_instance_disk_info
dk_size = int(os.path.getsize(path))
File "/usr/lib64/python2.7/genericpath.py", line 49, in getsize
return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory:
'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
] No such file or directory to caller
2015-03-20 09:22:39.743 13849 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', '
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in
inner\n return func(*args, **kwargs)\n', ' File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in
migrate_server\n block_migration, disk_over_commit)\n', ' File
"/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 793, in
_live_migrate\n raise exception.MigrationError(reason=ex)\n
', 'MigrationError: Migration error: [Errno [Errno 2] No such file or
directory:
\'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config\'\nTraceback
(most recent call last):\n\n File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133,
in _dispatch_and_reply\n incoming.message))\n\n File
"/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176,
in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n\n
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line
122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n\n File "/usr/lib/python2.7/site-packages/nova/exception.py",
line 88, in wrapped\n payload)\n\n File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File
"/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n
retu
rn f(self, context, *args, **kw)\n\n File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373, in
decorated_function\n e, sys.exc_info())\n\n File
"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in
decorated_function\n return function(self, context, *args, **kwargs)\n\n
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684, in
check_can_live_migrate_source\n dest_check_data)\n\n File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4610, in
check_can_live_migrate_source\n
jsonutils.loads(self.get_instance_disk_info(instance[\'name\'])))\n\n File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5293, in
get_instance_disk_info\n dk_size = int(os.path.getsize(path))\n\n File
"/usr/lib64/python2.7/genericpath.py", line
49, in getsize\n return os.stat(filename).st_size\n\nOSError: [Errno 2] No
such file or directory:
\'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config\'\n]
No such file or directory\n']
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1412279/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
