** Also affects: cloud-archive/juno Importance: Undecided Status: New
** Also affects: cloud-archive/icehouse Importance: Undecided Status: New ** Also affects: cloud-archive/kilo Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1374999 Title: iSCSI volume detach does not correctly remove the multipath device descriptors Status in Ubuntu Cloud Archive: New Status in Ubuntu Cloud Archive icehouse series: New Status in Ubuntu Cloud Archive juno series: New Status in Ubuntu Cloud Archive kilo series: New Status in OpenStack Compute (nova): In Progress Status in nova package in Ubuntu: Triaged Status in nova source package in Trusty: Triaged Bug description: iSCSI volume detach does not correctly remove the multipath device descriptors tested environment: nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300. I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths. Here is the terminal log for last volume detach. openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 | None | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l Fri Sep 19 21:38:13 JST 2014 360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=-1 status=active | |- 4:0:0:42 sdb 8:16 active undef running | |- 5:0:0:42 sdd 8:48 active undef running | |- 6:0:0:42 sdf 8:80 active undef running | `- 7:0:0:42 sdh 8:112 active undef running `-+- policy='round-robin 0' prio=-1 status=enabled |- 11:0:0:42 sdp 8:240 active undef running |- 8:0:0:42 sdj 8:144 active undef running |- 9:0:0:42 sdl 8:176 active undef running `- 10:0:0:42 sdn 8:208 active undef running openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session Fri Sep 19 21:38:19 JST 2014 tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7 tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7 tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4 tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5 tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6 tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4 tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5 tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6 openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e 56a63288-5cc0-4f5c-9197-cde731172dd8 openstack@W1DEV103:~/devstack$ openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 | None | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e| +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+ openstack@W1DEV103:~/devstack$ openstack@W1DEV103:~/devstack$ cinder list +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 | None | false | +--------------------------------------+-----------+------+------+-------------+----------+-------------+ | openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l Fri Sep 19 21:39:23 JST 2014 360060160cf0036002d1475f6e73fe411 dm-2 , size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw |-+- policy='round-robin 0' prio=-1 status=active | |- #:#:#:# - #:# active undef running | |- #:#:#:# - #:# active undef running | |- #:#:#:# - #:# active undef running | `- #:#:#:# - #:# active undef running `-+- policy='round-robin 0' prio=-1 status=enabled |- #:#:#:# - #:# active undef running |- #:#:#:# - #:# active undef running |- #:#:#:# - #:# active undef running `- #:#:#:# - #:# active undef running openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session Fri Sep 19 21:39:27 JST 2014 iscsiadm: No active sessions. Then I manually removed the multipaths, openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411 openstack@W1CN103:/etc/iscsi$ sudo multipath -l openstack@W1CN103:/etc/iscsi$ I think the problem is in, virt/libvirt/volume.py:LibvirtISCSIVolumeDriver def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device): End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return. self._remove_multipath_device_descriptor(multipath_device) return Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_multipath_device_descriptor due to following if statement. if not in_use: # disconnect if no other multipath devices with same iqn self._disconnect_mpath(iscsi_properties, ips_iqns) return It just disconnect them but not remove them. One of the reasons why we have to remove them is, https://bugs.launchpad.net/nova/+bug/1223975 IMO, we should call _remove_multipath_device_descriptor in above if statement before return. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1374999/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp