On Thursday, 15 July 2021 14:49:45 CEST Gianluca Cecchi wrote: > On Fri, Apr 23, 2021 at 7:15 PM Nir Soffer <nsof...@redhat.com> wrote: > > >> > 1) Is this the expected behavior? > > >> > > >> yes, before removing multipath devices, you need to unzone LUN on > > > > storage > > > > >> server. As oVirt doesn't manage storage server in case of iSCSI, it has > > > > to be > > > > >> done by storage sever admin and therefore oVirt cannot manage whole > > > > flow. > > > > > Thank you for the information. Perhaps you can expand then on how the > > > > volumes are picked up once mapped from the Storage system? Traditionally > > when mapping storage from an iSCSI or Fibre Channel storage we have to > > initiate a LIP or iSCSI login. How is it that oVirt doesn't need to do > > this?> > > >> > 2) Are we supposed to go to each KVM host and manually remove the > > >> > underlying multipath devices? > > >> > > >> oVirt provides ansible script for it: > > >> > > >> https://github.com/oVirt/ovirt-ansible-collection/blob/master/examples/ > > >> remove_mpath_device.yml > > >> > > >> Usage is as follows: > > >> > > >> ansible-playbook --extra-vars "lun=<LUN_ID>" remove_mpath_device.yml > > I had to decommission one iSCSI based storage domain, after having added > one new iSCSI one (with another portal) and moved all the objects into the > new one (vm disks, template disks, iso disks, leases). > The Environment is based on 4.4.6, with 3 hosts, external engine. > So I tried the ansible playbook way to verify it. > > Initial situation is this below; the storage domain to decommission is the > ovsd3750, based on the 5Tb LUN. > > $ sudo multipath -l > 364817197c52f98316900666e8c2b0b2b dm-13 EQLOGIC,100E-00 > size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw > `-+- policy='round-robin 0' prio=0 status=active > > |- 16:0:0:0 sde 8:64 active undef running > > `- 17:0:0:0 sdf 8:80 active undef running > 36090a0d800851c9d2195d5b837c9e328 dm-2 EQLOGIC,100E-00 > size=5.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw > `-+- policy='round-robin 0' prio=0 status=active > > |- 13:0:0:0 sdb 8:16 active undef running > > `- 14:0:0:0 sdc 8:32 active undef running > > Connections are using iSCSI multipathing (iscsi1 and iscs2 in web admin > gui), so that I have two paths to each LUN: > > $sudo iscsiadm -m node > 10.10.100.7:3260,1 > iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 > 10.10.100.7:3260,1 > iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 > 10.10.100.9:3260,1 > iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920 > 10.10.100.9:3260,1 > iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920 > > $ sudo iscsiadm -m session > tcp: [1] 10.10.100.7:3260,1 > iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 > (non-flash) > tcp: [2] 10.10.100.7:3260,1 > iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 > (non-flash) > tcp: [4] 10.10.100.9:3260,1 > iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920 > (non-flash) > tcp: [5] 10.10.100.9:3260,1 > iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920 > (non-flash) > > One point not taken in consideration inside the previously opened bugs in > my opinion is the deletion of iSCSI connections and node at host side > (probably to be done by the os admin, but it could be taken in charge by > the ansible playbook...) > The bugs I'm referring are: > Bug 1310330 - [RFE] Provide a way to remove stale LUNs from hypervisors > Bug 1928041 - Stale DM links after block SD removal > > Actions done: > put storage domain into maintenance > detach storage domain > remove storage domain > remove access from equallogic admin gui > > I have a group named ovirt in ansible inventory composed by my 3 hosts: > ov200, ov300 and ov301 > executed > $ ansible-playbook -b -l ovirt --extra-vars > "lun=36090a0d800851c9d2195d5b837c9e328" remove_mpath_device.yml > > it went all ok with ov200 and ov300, but for ov301 I got > > fatal: [ov301: FAILED! => {"changed": true, "cmd": "multipath -f > \"36090a0d800851c9d2195d5b837c9e328\"", "delta": "0:00:00.009003", "end": > "2021-07-15 11:17:37.340584", "msg": "non-zero return code", "rc": 1, > "start": "2021-07-15 11:17:37.331581", "stderr": "Jul 15 11:17:37 | > 36090a0d800851c9d2195d5b837c9e328: map in use", "stderr_lines": ["Jul 15 > 11:17:37 | 36090a0d800851c9d2195d5b837c9e328: map in use"], "stdout": "", > "stdout_lines": []} > > the complete output: > > $ ansible-playbook -b -l ovirt --extra-vars > "lun=36090a0d800851c9d2195d5b837c9e328" remove_mpath_device.yml > > PLAY [Cleanly remove unzoned storage devices (LUNs)] > ************************************************************* > > TASK [Gathering Facts] > **************************************************************************** > *************** ok: [ov200] > ok: [ov300] > ok: [ov301] > > TASK [Get underlying disks (paths) for a multipath device and turn them > into a list.] **************************** > changed: [ov300] > changed: [ov200] > changed: [ov301] > > TASK [Remove from multipath device.] > **************************************************************************** > * changed: [ov200] > changed: [ov300] > fatal: [ov301]: FAILED! => {"changed": true, "cmd": "multipath -f > \"36090a0d800851c9d2195d5b837c9e328\"", "delta": "0:00:00.009003", "end": > "2021-07-15 11:17:37.340584", "msg": "non-zero return code", "rc": 1, > "start": "2021-07-15 11:17:37.331581", "stderr": "Jul 15 11:17:37 | > 36090a0d800851c9d2195d5b837c9e328: map in use", "stderr_lines": ["Jul 15 > 11:17:37 | 36090a0d800851c9d2195d5b837c9e328: map in use"], "stdout": "", > "stdout_lines": []} > > TASK [Remove each path from the SCSI subsystem.] > ***************************************************************** > changed: [ov300] => (item=sdc) > changed: [ov300] => (item=sdb) > changed: [ov200] => (item=sdc) > changed: [ov200] => (item=sdb) > > PLAY RECAP > **************************************************************************** > *************************** ov200 : ok=4 changed=3 unreachable=0 > failed=0 skipped=0 rescued=0 ignored=0 > ov300 : ok=4 changed=3 unreachable=0 failed=0 skipped=0 > rescued=0 ignored=0 > ov301 : ok=2 changed=1 unreachable=0 failed=1 skipped=0 > rescued=0 ignored=0 > > Indeed going to the server I get: > > root@ov301 ~]# multipath -f 36090a0d800851c9d2195d5b837c9e328 > Jul 15 11:24:37 | 36090a0d800851c9d2195d5b837c9e328: map in use > [root@ov301 ~]# > > the dm device under the multipath one is dm-2 and > [root@ov301 ~]# ll /dev/dm-2 > brw-rw----. 1 root disk 253, 2 Jul 15 11:28 /dev/dm-2 > [root@ov301 ~]# > > [root@ov301 ~]# lsof | grep "253,2" > > I get no lines, but other devices with minor beginning with 2 (eg. 24, 25, > 27..) > . . . > qemu-kvm 10638 10653 vnc_worke qemu 84u BLK > 253,24 0t0 112027277 /dev/dm-24 > qemu-kvm 11479 qemu 43u BLK > 253,27 0t0 112135384 /dev/dm-27 > qemu-kvm 11479 qemu 110u BLK > 253,25 0t0 112140523 /dev/dm-25 > > so nothing for dm-2 > > What to do to crosscheck what is using the device and so preventing the > "-f" to complete?
can you try
dmsetup info /dev/mapper/36090a0d800851c9d2195d5b837c9e328
and check "Open count" filed to see if there is still anything open?
Also, you can try
fuser /dev/dm-2
to see which process is using the device
> Now I get
>
> # multipath -l
> 364817197c52f98316900666e8c2b0b2b dm-14 EQLOGIC,100E-00
> size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
> `-+- policy='round-robin 0' prio=0 status=active
>
> |- 16:0:0:0 sde 8:64 active undef running
>
> `- 17:0:0:0 sdf 8:80 active undef running
> 36090a0d800851c9d2195d5b837c9e328 dm-2 ##,##
> size=5.0T features='0' hwhandler='0' wp=rw
>
> Another thing to improve perhaps in the ansible playbook is that usually
> when in general I remove FC or iSCSI LUNs under multipath on a Linux
> system, after the "multipath -f" command and before the "echo 1 > ...
> /device/delete" one I run also, for safeness:
>
> blockdev --flushbufs /dev/$i
> where $i loops over the devices composing the multipath.
>
> I see that inside the web admin Gui under Datacenter--> iSCSI multipath
> iscsi1
> iscsi2
> there is no more the connection to the removed SD.
> But at the host side nothing changed from the iSCSI point of view.
> So I executed:
>
> log out from the sessions:
> [root@ov300 ~]# iscsiadm -m session -r 1 -u
> Logging out of session [sid: 1, target:
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750,
> portal: 10.10.100.7,3260]
> Logout of [sid: 1, target:
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750,
> portal: 10.10.100.7,3260] successful.
> [root@ov300 ~]# iscsiadm -m session -r 2 -u
> Logging out of session [sid: 2, target:
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750,
> portal: 10.10.100.7,3260]
> Logout of [sid: 2, target:
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750,
> portal: 10.10.100.7,3260] successful.
> [root@ov300 ~]#
>
> and then removal of the node
> [root@ov300 ~]# iscsiadm -m node -T
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 -o
> delete
> [root@ov300 ~]# ll /var/lib/iscsi/nodes/
> total 4
> drw-------. 3 root root 4096 Jul 13 11:18
> iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920
> [root@ov300 ~]#
>
> while previously I had:
> [root@ov300 ~]# ll /var/lib/iscsi/nodes/
> total 8
> drw-------. 3 root root 4096 Jan 12 2021
> iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750
> drw-------. 3 root root 4096 Jul 13 11:18
> iqn.2001-05.com.equallogic:4-771816-31982fc59-2b0b2b8c6e660069-ovsd3920
> [root@ov301 ~]#
>
> Otherwise I think that at reboot the host will try to reconnect to the no
> more existing portal...
>
> Comments welcome
>
> Gianluca
signature.asc
Description: This is a digitally signed message part.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4GYVBKZJ23R3OERD6SE3BUZVPTYKE2MY/
