Public bug reported: ### Steps to reproduce I have two compute node, source com and dest com. I did execute "live migration" command. $ nova live-migration <instance id> <dest com>
### Expected result The instance is migrated to dest com host and the status change from MIGRATING to ACTIVE. ### Actual result During _post_livemigration function, the multipath command somehow returned "code 1" , I think it's a temporary error, on source com. And the status remains MIGRATING I think retrying multipath command can avoid this problem. ### Environment 1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation is not much changed on the latest version. 2. [hypervisor] Libvirt + KVM (RHEL7.1) 3. [storage] FC Multipath 4. [network] openvswitch vlan ### Logs nova-compute.log on source com: Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444 Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199 CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123" returned: 1 in 0.109s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225 u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258 Multipath call failed exit (1) Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456 [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration: 'NoneType' object has no attribute '__getitem__' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5708, in _live_migration dom, finish_event) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5638, in _live_migration_monitor migrate_data) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped payload) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped return f(self, context, *args, **kw) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in decorated_function kwargs['instance'], e, sys.exc_info()) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355, in _post_live_migration migrate_data) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5983, in post_live_migration self._disconnect_volume(connection_info, disk_dev) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1023, in _disconnect_volume return driver.disconnect_volume(connection_info, disk_dev) File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1402, in disconnect_volume devices = mdev_info['devices'] TypeError: 'NoneType' object has no attribute '__getitem__' [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1686868 Title: live-migration failed when multipath command returned code 1 Status in OpenStack Compute (nova): New Bug description: ### Steps to reproduce I have two compute node, source com and dest com. I did execute "live migration" command. $ nova live-migration <instance id> <dest com> ### Expected result The instance is migrated to dest com host and the status change from MIGRATING to ACTIVE. ### Actual result During _post_livemigration function, the multipath command somehow returned "code 1" , I think it's a temporary error, on source com. And the status remains MIGRATING I think retrying multipath command can avoid this problem. ### Environment 1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation is not much changed on the latest version. 2. [hypervisor] Libvirt + KVM (RHEL7.1) 3. [storage] FC Multipath 4. [network] openvswitch vlan ### Logs nova-compute.log on source com: Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444 Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199 CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123" returned: 1 in 0.109s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225 u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258 Multipath call failed exit (1) Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456 [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration: 'NoneType' object has no attribute '__getitem__' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5708, in _live_migration dom, finish_event) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5638, in _live_migration_monitor migrate_data) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped payload) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped return f(self, context, *args, **kw) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in decorated_function kwargs['instance'], e, sys.exc_info()) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355, in _post_live_migration migrate_data) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5983, in post_live_migration self._disconnect_volume(connection_info, disk_dev) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1023, in _disconnect_volume return driver.disconnect_volume(connection_info, disk_dev) File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1402, in disconnect_volume devices = mdev_info['devices'] TypeError: 'NoneType' object has no attribute '__getitem__' [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1686868/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp