This is basically a libvirt bug, which is fixed upstream. It is probably
not worth trying to mitigate this in Nova given the versions of libvirt
that distros are moving to.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1617073
Title:
Libvirt live block migration fails due to ['migrate-incoming': Failed
to bind socket: Address already in use]
Status in OpenStack Compute (nova):
Invalid
Bug description:
On the gate job gate-tempest-dsvm-multinode-live-migration, a Tempest
test test_live_block_migration fails because an actual host which a
server stays was different from the expected.
The tempest log is http://logs.openstack.org/50/349450/5/check/gate-
tempest-dsvm-multinode-live-
migration/60eb3b3/console.html#_2016-08-25_19_14_13_416233
2016-08-25 19:14:13.416233 | 2016-08-25 19:14:13.415 | Captured traceback:
2016-08-25 19:14:13.417870 | 2016-08-25 19:14:13.417 | ~~~~~~~~~~~~~~~~~~~
2016-08-25 19:14:13.419367 | 2016-08-25 19:14:13.419 | Traceback (most
recent call last):
2016-08-25 19:14:13.421015 | 2016-08-25 19:14:13.420 | File
"tempest/api/compute/admin/test_live_migration.py", line 132, in
test_live_block_migration
2016-08-25 19:14:13.422730 | 2016-08-25 19:14:13.422 |
self._test_live_migration()
2016-08-25 19:14:13.424314 | 2016-08-25 19:14:13.423 | File
"tempest/api/compute/admin/test_live_migration.py", line 128, in
_test_live_migration
2016-08-25 19:14:13.426047 | 2016-08-25 19:14:13.425 | msg)
2016-08-25 19:14:13.427470 | 2016-08-25 19:14:13.427 | File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py",
line 411, in assertEqual
2016-08-25 19:14:13.429269 | 2016-08-25 19:14:13.428 |
self.assertThat(observed, matcher, message)
2016-08-25 19:14:13.430954 | 2016-08-25 19:14:13.430 | File
"/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py",
line 498, in assertThat
2016-08-25 19:14:13.432623 | 2016-08-25 19:14:13.432 | raise
mismatch_error
2016-08-25 19:14:13.434426 | 2016-08-25 19:14:13.433 |
testtools.matchers._impl.MismatchError: !=:
2016-08-25 19:14:13.436041 | 2016-08-25 19:14:13.435 | reference =
u'ubuntu-xenial-2-node-rax-ord-3874987'
2016-08-25 19:14:13.437717 | 2016-08-25 19:14:13.437 | actual =
u'ubuntu-xenial-2-node-rax-ord-3874987-181073'
2016-08-25 19:14:13.439228 | 2016-08-25 19:14:13.438 | : Live Migration
failed. Migrations list for Instance 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0: [
2016-08-25 19:14:13.440771 | 2016-08-25 19:14:13.440 |
{u'migration_type': u'live-migration', u'instance_uuid':
u'0240bf1c-d2cf-407f-9ecf-7acfd891b4c0', u'status': u'error', u'updated_at':
u'2016-08-25T19:13:40.000000', u'created_at': u'2016-08-25T19:13:33.000000',
u'source_node': None, u'dest_node': None, u'dest_host': None, u'dest_compute':
u'ubuntu-xenial-2-node-rax-ord-3874987', u'source_compute':
u'ubuntu-xenial-2-node-rax-ord-3874987-181073', u'id': 1,
u'old_instance_type_id': 11, u'new_instance_type_id': 11}]
nova-cpu outputs the following log:
http://logs.openstack.org/50/349450/5/check/gate-tempest-dsvm-
multinode-live-
migration/60eb3b3/logs/subnode-2/screen-n-cpu.txt.gz#_2016-08-25_19_13_38_903
2016-08-25 19:13:38.904 25185 DEBUG nova.virt.libvirt.driver
[req-04d0773e-0085-4605-ad4e-bd98329b6c4d
tempest-LiveAutoBlockMigrationV225TestJSON-2087766141
tempest-LiveAutoBlockMigrationV225TestJSON-2087766141] [instance:
0240bf1c-d2cf-407f-9ecf-7acfd891b4c0] Migration operation thread notification
thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6273
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
457, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line
58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168,
in _do_send
waiter.switch(result)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line
214, in main
result = function(*args, **kwargs)
File "/opt/stack/new/nova/nova/utils.py", line 1066, in context_wrapper
return func(*args, **kwargs)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5874, in
_live_migration_operation
instance=instance)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
220, in __exit__
self.force_reraise()
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
196, in force_reraise
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5870, in
_live_migration_operation
bandwidth=CONF.libvirt.live_migration_bandwidth)
File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 568, in migrate
destination, params=params, flags=flags)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186,
in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144,
in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125,
in execute
six.reraise(c, e, tb)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83,
in tworker
rv = meth(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in
migrateToURI3
if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed',
dom=self)
libvirtError: internal error: unable to execute QEMU command
'migrate-incoming': Failed to bind socket: Address already in use
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1617073/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
