Public bug reported:
### Steps to reproduce
I have two compute node, source com and dest com.
I did execute "live migration" command.
$ nova live-migration <instance id> <dest com>
### Expected result
The instance is migrated to dest com host and the status change from MIGRATING
to ACTIVE.
### Actual result
During _post_livemigration function, the multipath command somehow returned
"code 1" , I think it's a temporary error, on source com.
And the status remains MIGRATING
I think retrying multipath command can avoid this problem.
### Environment
1. [version] My environment is actually Kilo
(openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation
is not much changed on the latest version.
2. [hypervisor] Libvirt + KVM (RHEL7.1)
3. [storage] FC Multipath
4. [network] openvswitch vlan
### Logs
nova-compute.log on source com:
Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath
-l 123456789012345678901234567890123 execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
123456789012345678901234567890123" returned: 1 in 0.109s execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
123456789012345678901234567890123' failed. Not Retrying. execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
Multipath call failed exit (1)
Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration:
'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5708, in _live_migration
dom, finish_event)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5638, in _live_migration_monitor
migrate_data)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
payload)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
__exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in
decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
__exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in
decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355,
in _post_live_migration
migrate_data)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5983, in post_live_migration
self._disconnect_volume(connection_info, disk_dev)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
1023, in _disconnect_volume
return driver.disconnect_volume(connection_info, disk_dev)
File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line
445, in inner
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line
1402, in disconnect_volume
devices = mdev_info['devices']
TypeError: 'NoneType' object has no attribute '__getitem__'
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is
all done _live_migration
/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1686868
Title:
live-migration failed when multipath command returned code 1
Status in OpenStack Compute (nova):
New
Bug description:
### Steps to reproduce
I have two compute node, source com and dest com.
I did execute "live migration" command.
$ nova live-migration <instance id> <dest com>
### Expected result
The instance is migrated to dest com host and the status change from
MIGRATING to ACTIVE.
### Actual result
During _post_livemigration function, the multipath command somehow returned
"code 1" , I think it's a temporary error, on source com.
And the status remains MIGRATING
I think retrying multipath command can avoid this problem.
### Environment
1. [version] My environment is actually Kilo
(openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation
is not much changed on the latest version.
2. [hypervisor] Libvirt + KVM (RHEL7.1)
3. [storage] FC Multipath
4. [network] openvswitch vlan
### Logs
nova-compute.log on source com:
Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
multipath -l 123456789012345678901234567890123 execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
123456789012345678901234567890123" returned: 1 in 0.109s execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l
123456789012345678901234567890123' failed. Not Retrying. execute
/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
Multipath call failed exit (1)
Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration:
'NoneType' object has no attribute '__getitem__'
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5708, in _live_migration
dom, finish_event)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5638, in _live_migration_monitor
migrate_data)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in
wrapped
payload)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
__exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in
wrapped
return f(self, context, *args, **kw)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355,
in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
__exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343,
in decorated_function
return function(self, context, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355,
in _post_live_migration
migrate_data)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
5983, in post_live_migration
self._disconnect_volume(connection_info, disk_dev)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
1023, in _disconnect_volume
return driver.disconnect_volume(connection_info, disk_dev)
File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line
445, in inner
return f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line
1402, in disconnect_volume
devices = mdev_info['devices']
TypeError: 'NoneType' object has no attribute '__getitem__'
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is
all done _live_migration
/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1686868/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
