Public bug reported: On the gate job gate-tempest-dsvm-multinode-live-migration, a Tempest test test_live_block_migration fails because an actual host which a server stays was different from the expected.
The tempest log is http://logs.openstack.org/50/349450/5/check/gate- tempest-dsvm-multinode-live- migration/60eb3b3/console.html#_2016-08-25_19_14_13_416233 2016-08-25 19:14:13.416233 | 2016-08-25 19:14:13.415 | Captured traceback: 2016-08-25 19:14:13.417870 | 2016-08-25 19:14:13.417 | ~~~~~~~~~~~~~~~~~~~ 2016-08-25 19:14:13.419367 | 2016-08-25 19:14:13.419 | Traceback (most recent call last): 2016-08-25 19:14:13.421015 | 2016-08-25 19:14:13.420 | File "tempest/api/compute/admin/test_live_migration.py", line 132, in test_live_block_migration 2016-08-25 19:14:13.422730 | 2016-08-25 19:14:13.422 | self._test_live_migration() 2016-08-25 19:14:13.424314 | 2016-08-25 19:14:13.423 | File "tempest/api/compute/admin/test_live_migration.py", line 128, in _test_live_migration 2016-08-25 19:14:13.426047 | 2016-08-25 19:14:13.425 | msg) 2016-08-25 19:14:13.427470 | 2016-08-25 19:14:13.427 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual 2016-08-25 19:14:13.429269 | 2016-08-25 19:14:13.428 | self.assertThat(observed, matcher, message) 2016-08-25 19:14:13.430954 | 2016-08-25 19:14:13.430 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2016-08-25 19:14:13.432623 | 2016-08-25 19:14:13.432 | raise mismatch_error 2016-08-25 19:14:13.434426 | 2016-08-25 19:14:13.433 | testtools.matchers._impl.MismatchError: !=: 2016-08-25 19:14:13.436041 | 2016-08-25 19:14:13.435 | reference = u'ubuntu-xenial-2-node-rax-ord-3874987' 2016-08-25 19:14:13.437717 | 2016-08-25 19:14:13.437 | actual = u'ubuntu-xenial-2-node-rax-ord-3874987-181073' 2016-08-25 19:14:13.439228 | 2016-08-25 19:14:13.438 | : Live Migration failed. Migrations list for Instance 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0: [ 2016-08-25 19:14:13.440771 | 2016-08-25 19:14:13.440 | {u'migration_type': u'live-migration', u'instance_uuid': u'0240bf1c-d2cf-407f-9ecf-7acfd891b4c0', u'status': u'error', u'updated_at': u'2016-08-25T19:13:40.000000', u'created_at': u'2016-08-25T19:13:33.000000', u'source_node': None, u'dest_node': None, u'dest_host': None, u'dest_compute': u'ubuntu-xenial-2-node-rax-ord-3874987', u'source_compute': u'ubuntu-xenial-2-node-rax-ord-3874987-181073', u'id': 1, u'old_instance_type_id': 11, u'new_instance_type_id': 11}] nova-cpu outputs the following log: http://logs.openstack.org/50/349450/5/check/gate-tempest-dsvm-multinode- live- migration/60eb3b3/logs/subnode-2/screen-n-cpu.txt.gz#_2016-08-25_19_13_38_903 2016-08-25 19:13:38.904 25185 DEBUG nova.virt.libvirt.driver [req-04d0773e-0085-4605-ad4e-bd98329b6c4d tempest-LiveAutoBlockMigrationV225TestJSON-2087766141 tempest-LiveAutoBlockMigrationV225TestJSON-2087766141] [instance: 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6273 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers timer() File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__ cb(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send waiter.switch(result) File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main result = function(*args, **kwargs) File "/opt/stack/new/nova/nova/utils.py", line 1066, in context_wrapper return func(*args, **kwargs) File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5874, in _live_migration_operation instance=instance) File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5870, in _live_migration_operation bandwidth=CONF.libvirt.live_migration_bandwidth) File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 568, in migrate destination, params=params, flags=flags) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit result = proxy_call(self._autowrap, f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call rv = execute(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute six.reraise(c, e, tb) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker rv = meth(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'migrate-incoming': Failed to bind socket: Address already in use ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1617073 Title: Libvirt live block migration fails due to ['migrate-incoming': Failed to bind socket: Address already in use] Status in OpenStack Compute (nova): New Bug description: On the gate job gate-tempest-dsvm-multinode-live-migration, a Tempest test test_live_block_migration fails because an actual host which a server stays was different from the expected. The tempest log is http://logs.openstack.org/50/349450/5/check/gate- tempest-dsvm-multinode-live- migration/60eb3b3/console.html#_2016-08-25_19_14_13_416233 2016-08-25 19:14:13.416233 | 2016-08-25 19:14:13.415 | Captured traceback: 2016-08-25 19:14:13.417870 | 2016-08-25 19:14:13.417 | ~~~~~~~~~~~~~~~~~~~ 2016-08-25 19:14:13.419367 | 2016-08-25 19:14:13.419 | Traceback (most recent call last): 2016-08-25 19:14:13.421015 | 2016-08-25 19:14:13.420 | File "tempest/api/compute/admin/test_live_migration.py", line 132, in test_live_block_migration 2016-08-25 19:14:13.422730 | 2016-08-25 19:14:13.422 | self._test_live_migration() 2016-08-25 19:14:13.424314 | 2016-08-25 19:14:13.423 | File "tempest/api/compute/admin/test_live_migration.py", line 128, in _test_live_migration 2016-08-25 19:14:13.426047 | 2016-08-25 19:14:13.425 | msg) 2016-08-25 19:14:13.427470 | 2016-08-25 19:14:13.427 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual 2016-08-25 19:14:13.429269 | 2016-08-25 19:14:13.428 | self.assertThat(observed, matcher, message) 2016-08-25 19:14:13.430954 | 2016-08-25 19:14:13.430 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat 2016-08-25 19:14:13.432623 | 2016-08-25 19:14:13.432 | raise mismatch_error 2016-08-25 19:14:13.434426 | 2016-08-25 19:14:13.433 | testtools.matchers._impl.MismatchError: !=: 2016-08-25 19:14:13.436041 | 2016-08-25 19:14:13.435 | reference = u'ubuntu-xenial-2-node-rax-ord-3874987' 2016-08-25 19:14:13.437717 | 2016-08-25 19:14:13.437 | actual = u'ubuntu-xenial-2-node-rax-ord-3874987-181073' 2016-08-25 19:14:13.439228 | 2016-08-25 19:14:13.438 | : Live Migration failed. Migrations list for Instance 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0: [ 2016-08-25 19:14:13.440771 | 2016-08-25 19:14:13.440 | {u'migration_type': u'live-migration', u'instance_uuid': u'0240bf1c-d2cf-407f-9ecf-7acfd891b4c0', u'status': u'error', u'updated_at': u'2016-08-25T19:13:40.000000', u'created_at': u'2016-08-25T19:13:33.000000', u'source_node': None, u'dest_node': None, u'dest_host': None, u'dest_compute': u'ubuntu-xenial-2-node-rax-ord-3874987', u'source_compute': u'ubuntu-xenial-2-node-rax-ord-3874987-181073', u'id': 1, u'old_instance_type_id': 11, u'new_instance_type_id': 11}] nova-cpu outputs the following log: http://logs.openstack.org/50/349450/5/check/gate-tempest-dsvm- multinode-live- migration/60eb3b3/logs/subnode-2/screen-n-cpu.txt.gz#_2016-08-25_19_13_38_903 2016-08-25 19:13:38.904 25185 DEBUG nova.virt.libvirt.driver [req-04d0773e-0085-4605-ad4e-bd98329b6c4d tempest-LiveAutoBlockMigrationV225TestJSON-2087766141 tempest-LiveAutoBlockMigrationV225TestJSON-2087766141] [instance: 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6273 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers timer() File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__ cb(*args, **kw) File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send waiter.switch(result) File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main result = function(*args, **kwargs) File "/opt/stack/new/nova/nova/utils.py", line 1066, in context_wrapper return func(*args, **kwargs) File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5874, in _live_migration_operation instance=instance) File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5870, in _live_migration_operation bandwidth=CONF.libvirt.live_migration_bandwidth) File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 568, in migrate destination, params=params, flags=flags) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit result = proxy_call(self._autowrap, f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call rv = execute(f, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute six.reraise(c, e, tb) File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker rv = meth(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'migrate-incoming': Failed to bind socket: Address already in use To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1617073/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
