Regarding the cleanup in detach, I found that the python-os-brick package in Ubuntu 16 is rather old: 1.2.0
I found that the function find_multipath_device in os_brick/initiator/linuxscsi.py was refactored her https://github.com/openstack/os-brick/commit/3ea86f7d605d7109f4e13c1566ba1a14dc856443 I partially applied those changes and could make it work (clean up) On Fri, Aug 12, 2016 at 5:47 PM, Gustavo Randich <[email protected]> wrote: > Hi, > > Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work > OK for us in Ubuntu 14 / Icehouse seem to be broken. > > Basically, the WWID of the volume is not automatically placed in > /etc/multipath/wwids, and if we add it manually, we can attach with > multipath, but the detach never cleans up iscsi sessions nor device mapper > multipath maps. > > (using multipath-tools 0.5.0+git1.656f8865-5ubuntu2.1, and only one path > for testing purposes) > > > > - first attach, gives "can't get udev device" error -> > https://bugs.launchpad.net/os-brick/+bug/1585940 > > Failed to execute command multipath -l > /dev/disk/by-path/ip-10.167.15.101:3260-iscsi-iqn.1992-08.co > m.netapp:sn.1873815289-lun-0 > > root@dev-e01-host07:~# multipath -l /dev/disk/by-path/ip-10.167.15.101: > 3260-iscsi-iqn.1992-08.com.netapp:sn.1873815289-lun-0 > Aug 12 15:38:30 | ip-10.167.15.101:3260-iscsi-iq > n.1992-08.com.netapp:sn.1873815289-lun-0: can't get udev device > > > - once patched, we get a non-multipath attach: in nova-compute.log we see > "couldn't find a valid multipath device path for <id>" > > issuing "multipath -v3", we get "wwid <id> not in wwids file, > skipping <device>" > > root@dev-e01-host07:~# multipath -v3 > Aug 12 15:39:47 | libdevmapper version 1.02.110 (2015-10-30) > Aug 12 15:39:47 | DM multipath kernel driver v1.11.0 > Aug 12 15:39:47 | sdb: prio = ontap controller setting) > Aug 12 15:39:47 | sdb: prio args = "" controller setting) > Aug 12 15:39:47 | sdb: ontap prio = 2 > ... > ... > Aug 12 15:39:47 | dm-0: blacklisted, udev property missing > ===== paths list ===== > uuid hcil dev dev_t pri dm_st chk_st > vend/prod > 360a9800041763061792b492f4e564179 3:0:0:0 sdb 8:16 2 undef faulty > NETAPP,LU > *Aug 12 15:39:47 | wwid 360a9800041763061792b492f4e564179 not in wwids > file, skipping sdb* > Aug 12 15:39:47 | sdb: orphan path, only one path > Aug 12 15:39:47 | ontap prioritizer refcount 1 > Aug 12 15:39:47 | tur checker refcount 1 > > > - if we manually add the WWID to /etc/multipath/wwids, the attach is done > with multipath: > > root@dev-e01-host07:~# virsh domblklist 10 > Target Source > ------------------------------------------------ > vda /var/lib/nova/instances/8440e7a8-92d9-448d-ac17- > 5f04f032849c/disk > vdb /dev/disk/by-id/dm-uuid-mpath-360a9800041763061792b492f4e564230 > > root@dev-e01-host07:~# multipath -ll > 360a9800041763061792b492f4e564230 dm-1 NETAPP,LUN > size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' > hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=2 status=active > `- 3:0:0:0 sdb 8:16 active ready running > > > - but the detach operation doesn't remove the multipath map and doesn't > logout of iscsi sessions > > [1]+ Done nova volume-detach > 8440e7a8-92d9-448d-ac17-5f04f032849c > 2f717d41-894d-472a-851c-aaa03b593364 > > root@dev-e01-host07:~# multipath -ll > 360a9800041763061792b492f4e564230 dm-1 ##,## > size=1.0G features='2 queue_if_no_path retain_attached_hw_handler' > hwhandler='0' wp=rw > `-+- policy='round-robin 0' prio=0 status=enabled > `- #:#:#:# sdb 8:16 failed undef running > > root@dev-e01-host07:~# iscsiadm -m session > tcp: [2] 10.167.15.101:3260,2000 iqn.1992-08.com.netapp:sn.1873815289 > (non-flash) > > > > in nova-compute.log, we see "Couldn't find multipath device /dev/mapper/ > 360a9800041763061792b492f4e564230" > > > > instance: 8440e7a8-92d9-448d-ac17-5f04f032849c] Detach volume > 951264a3-19d9-4f7e-8bbb-a2eabdcae044 from mountpoint /dev/vdb > -] Generated XML ('<disk type="block" device="disk">\n <driver > name="qemu" type="raw" cache="none" io="native"/>\n <source > dev="/dev/disk/by-id/dm-uuid-mpath-360a9 > -] Waiting for function nova.virt.libvirt.guest._do_wait_and_retry_detach > to return. func /usr/lib/python2.7/dist-packages/oslo_service/ > loopingcall.py:384 > 83 - - -] calling os-brick to detach iSCSI Volume disconnect_volume > /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:95 > - -] Lock "connect_volume" acquired by > "os_brick.initiator.connector.disconnect_volume" > :: waited 0.000s inner /usr/lib/python2.7/dist- > packages/oslo_concurrency/lock > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.py:3 > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r" > returned: 0 in 0.167s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.py > - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230 > undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0' > wp=undef#012`-+- > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf multipath -ll /dev/sdb execute > /usr/lib/python2.7/dist-packages/oslo_concurrency/process > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll > /dev/sdb" returned: 0 in 0.191s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/proce > - - -] multipath ['-ll', u'/dev/sdb']: stdout= > 360a9800041763061792b492f4e564230 dm-1 NETAPP,LUN#012size=1.0G > features='2 queue_if_no_path retain_attached_hw_handler' > - - -] remove multipath device /dev/sdb remove_multipath_device > /usr/lib/python2.7/dist-packages/os_brick/initiator/linuxscsi.py:123 > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf multipath -l /dev/sdb execute > /usr/lib/python2.7/dist-packages/oslo_concurrency/processu > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l > /dev/sdb" returned: 0 in 0.156s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/proces > 83 - - -] Couldn't find multipath device /dev/mapper/ > 360a9800041763061792b492f4e564230 > - - -] Disconnect multipath device /dev/mapper/ > 360a9800041763061792b492f4e564230 _disconnect_volume_multipath_iscsi > /usr/lib/python2.7/dist-packages/os_brick/initiat > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf multipath -ll execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.py: > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll" > returned: 0 in 0.158s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.p > - - -] multipath ['-ll']: stdout=360a9800041763061792b492f4e564230 dm-1 > NETAPP,LUN#012size=1.0G features='2 queue_if_no_path > retain_attached_hw_handler' hwhandler='0 > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p > 10.167.15.101:3260 execute /usr/lib/python2.7/di > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m > discovery -t sendtargets -p 10.167.15.101:3260" returned: 0 in 0.168s > execute /usr/lib/python2.7/ > - - -] iscsiadm ['-m', 'discovery', '-t', 'sendtargets', '-p', > u'10.167.15.101:3260']: stdout=10.167.15.101:3260,2000 > iqn.1992-08.com.netapp:sn.1873815289#012 stderr > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf multipath -r execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.py:3 > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -r" > returned: 0 in 0.165s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processutils.py > - - -] multipath ['-r']: stdout=reload: 360a9800041763061792b492f4e564230 > undef NETAPP,LUN#012size=1.0G features='1 queue_if_no_path' hwhandler='0' > wp=undef#012`-+- > - -] Lock "connect_volume" released by > "os_brick.initiator.connector.disconnect_volume" > :: held 1.022s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/ > lockut > 83 - - -] Disconnected iSCSI Volume vdb disconnect_volume > /usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume/iscsi.py:101 > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf cat /etc/iscsi/initiatorname.iscsi execute > /usr/lib/python2.7/dist-packages/oslo_concurr > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf cat > /etc/iscsi/initiatorname.iscsi" returned: 0 in 0.186s execute > /usr/lib/python2.7/dist-packages/oslo_concu > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf systool -c fc_host -v execute > /usr/lib/python2.7/dist-packages/oslo_concurrency/processu > 3 - - -] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c > fc_host -v" returned: 96 in 0.140s execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/proce > 3 - - -] u'sudo nova-rootwrap /etc/nova/rootwrap.conf systool -c fc_host > -v' failed. Not Retrying. execute /usr/lib/python2.7/dist- > packages/oslo_concurrency/processut > - - -] systool is not installed > 3 - - -] Running cmd (subprocess): sudo nova-rootwrap > /etc/nova/rootwrap.conf systool -c fc_host -v execute > /usr/lib/python2.7/dist-packages/oslo_concurrency/processu > > > > Any idea of what could be happening? > > Thanks > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
