Hello ovirt gurus,
I'd like to bother you with a long lasting issue I face.
My (new) test setup (not production one) is running ovirt 4.04 hosted engine on
a Dell R510 server connected to SAN Compellent SC040 via iscsi.
All systems installed from scratch following guidelines.
Ovirt hosted engine installed using ovirt-appliance.
Everything is working "fine".
BUT
We are unable to perform a clear LUN removal.
We can perform all actions from web interface (storage add, maintenance,
detach, delete) with no error.
However, underlying device remains mapped to server. As a result, we are unable
to unmap the LUN otherwise multipath fails, server fails, ovirt fails etc.
Steps to reproduce:
1) Create LUN on SAN, map it to server -> OK
2) Log in ovirt web interface, add a new storage targeting LUN -> OK
3) Create disk, create VM setup VM, (optional) -> OK
4) Shutdown VM -> OK
5) Ovirt: Put storage in maintenance -> OK
6) Ovirt: Detach storage -> OK
7) Ovirt: Delete storage -> OK
Expected result:
Ovirt unmaps and remove device from multipath so that it can be destroyed at
SAN storage level.
It should be possible to perform this action even when host is not in
maintenance.
Current result: volume remain locked in multipath (map in use).
Trying vgchange -a n <DEVICE> and then multipath -f <DEVICE> removes device,
but then it is automatically re-added after a while (???)
Please note: this is the same behavior we always faced also with our ovirt
production environment.
Details attached. Volume to be removed: LUN 000064
Any ideas?
Thanks
AG
[root@barbera log]# multipath -ll
36000d310004edc000000000000000064 dm-0 COMPELNT,Compellent Vol
size=100G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 18:0:0:2 sdk 8:160 active ready running
|- 19:0:0:2 sdm 8:192 active ready running
|- 22:0:0:2 sdq 65:0 active ready running
|- 23:0:0:2 sds 65:32 active ready running
|- 24:0:0:2 sdu 65:64 active ready running
|- 28:0:0:2 sdaa 65:160 active ready running
|- 29:0:0:2 sdab 65:176 active ready running
|- 5:0:0:2 sdc 8:32 active ready running
`- 6:0:0:2 sdg 8:96 active ready running
36000d310004edc000000000000000062 dm-1 COMPELNT,Compellent Vol
size=1.0T features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 5:0:0:1 sdb 8:16 active ready running
|- 6:0:0:1 sdd 8:48 active ready running
|- 19:0:0:1 sdl 8:176 active ready running
|- 18:0:0:1 sdj 8:144 active ready running
|- 22:0:0:1 sdp 8:240 active ready running
|- 23:0:0:1 sdr 65:16 active ready running
|- 24:0:0:1 sdt 65:48 active ready running
|- 29:0:0:1 sdz 65:144 active ready running
`- 28:0:0:1 sdy 65:128 active ready running
36000d310004edc000000000000000061 dm-19 COMPELNT,Compellent Vol
size=120G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 8:0:0:1 sdf 8:80 active ready running
|- 7:0:0:1 sde 8:64 active ready running
|- 11:0:0:1 sdh 8:112 active ready running
|- 12:0:0:1 sdi 8:128 active ready running
|- 20:0:0:1 sdn 8:208 active ready running
|- 21:0:0:1 sdo 8:224 active ready running
|- 25:0:0:1 sdv 65:80 active ready running
|- 26:0:0:1 sdw 65:96 active ready running
`- 27:0:0:1 sdx 65:112 active ready running
[root@barbera log]#
[root@barbera log]# lsscsi
[0:0:0:0] disk SEAGATE ST3300657SS ES66 -
[0:0:1:0] disk SEAGATE ST3300657SS ES66 -
[0:1:0:0] disk Dell Virtual Disk 1028 /dev/sda
[3:0:0:0] cd/dvd PLDS DVD+-RW DS-8A8SH KD51 /dev/sr0
[5:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdb
[5:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdc
[6:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdd
[6:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdg
[7:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sde
[8:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdf
[11:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdh
[12:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdi
[18:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdj
[18:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdk
[19:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdl
[19:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdm
[20:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdn
[21:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdo
[22:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdp
[22:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdq
[23:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdr
[23:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sds
[24:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdt
[24:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdu
[25:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdv
[26:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdw
[27:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdx
[28:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdy
[28:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdaa
[29:0:0:1] disk COMPELNT Compellent Vol 0606 /dev/sdz
[29:0:0:2] disk COMPELNT Compellent Vol 0606 /dev/sdab
[root@barbera log]#
[root@barbera log]# vgs
VG #PV #LV #SN Attr VSize VFree
40907462-5b58-40c2-ade7-3031c8a01949 1 9 0 wz--n- 99.62g 35.50g
4e4afb2f-2c86-44ee-90ed-58220664c8c4 1 12 0 wz--n- 119.62g 104.25g
62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97 1 8 0 wz--n- 1023.62g 1019.50g
[root@barbera log]# lvs
LV VG
Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
397c0f40-ed70-4670-af89-55c35de929d6 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 128.00m
aac32cf9-af0d-42c4-84f7-6c62fa62f23c 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 60.00g
d5065ccd-c56a-42bb-accd-6e09c8a44ebe 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 128.00m
ids 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 128.00m
inbox 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 128.00m
leases 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 2.00g
master 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 1.00g
metadata 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 512.00m
outbox 40907462-5b58-40c2-ade7-3031c8a01949
-wi-a----- 128.00m
12b6b2f0-111c-441f-ba85-20996a049378 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 1.00g
3240e1f4-f16b-476a-80ea-ec4f14bdab85 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-ao---- 128.00m
7a65e0e2-05f5-40bf-9a40-09ac50ab0673 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 128.00m
9d1ed3fd-d6ef-4bbc-b222-7687134b8fc3 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 128.00m
b851077b-2f2d-4970-b433-427cad796d46 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-ao---- 10.00g
ecc47e78-b052-40ff-8eaf-4145b41df851 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 128.00m
ids 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-ao---- 128.00m
inbox 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 128.00m
leases 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 2.00g
master 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 1.00g
metadata 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 512.00m
outbox 4e4afb2f-2c86-44ee-90ed-58220664c8c4
-wi-a----- 128.00m
4287f4ac-c6c4-4445-ae4d-18b9e8791db6 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi------- 128.00m
8f5b90e7-f1b3-479a-83ae-8cba03849f8e 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi------- 128.00m
ids 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-ao---- 128.00m
inbox 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-a----- 128.00m
leases 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-a----- 2.00g
master 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-ao---- 1.00g
metadata 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-a----- 512.00m
outbox 62d8cfdd-ee80-415b-b2c3-a7e6bda0dd97
-wi-a----- 128.00m
[root@barbera log]#
[root@barbera log]# dmsetup info
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-3240e1f4--f16b--476a--80ea--ec4f14bdab85
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 26
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxGLa8MxXLjGZriVzIwZih8nVG3wbyf0jn
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-leases
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 13
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL27g2GG67P0NfY9E09hhj1hXuKfBqQjIb3
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-b851077b--2f2d--4970--b433--427cad796d46
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 18
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxG8DRH566pPlz0Lar6UuLSzoTaNoMJAIz
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-ecc47e78--b052--40ff--8eaf--4145b41df851
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 9
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxquWV3DOttM2D0lgyXgYeHNG8bcCeOoTO
Name: 40907462--5b58--40c2--ade7--3031c8a01949-metadata
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 2
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76LWvXyJijO14BqZpP19VZ2km0RfGUVJqKL
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-12b6b2f0--111c--441f--ba85--20996a049378
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 17
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxLwq3f2sEngWy31oTxj2rf9B1aXmhBK8h
Name: 40907462--5b58--40c2--ade7--3031c8a01949-leases
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 4
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76LDFZh3dtMYL2VMRLQm7HMwAl43k8OmMFo
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-metadata
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 11
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL2QThGptQOESGafvmrlj39yyZXStYEO0Rm
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-7a65e0e2--05f5--40bf--9a40--09ac50ab0673
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 10
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxS3Gsav6apOymEDSYkLpQEkBQX3x8T11V
Name: 36000d310004edc000000000000000064
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 9
Event number: 1
Major, minor: 253, 0
Number of targets: 1
UUID: mpath-36000d310004edc000000000000000064
Name:
40907462--5b58--40c2--ade7--3031c8a01949-d5065ccd--c56a--42bb--accd--6e09c8a44ebe
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 29
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76LiB2BSfDXxFgTkV7l3hqMsoLKxjT2YvXI
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-master
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 16
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL2vybGSWFcZI95Z268UUF15A0jxcgNW6LR
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-ids
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 14
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL2l3I9N8Qrao5ry2MGG3LhUOWFf13k7VqL
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-metadata
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 20
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxtCdgCf6sXMUkVA01a3GLESAV323jGlGt
Name: 40907462--5b58--40c2--ade7--3031c8a01949-master
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 7
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76L5B9CnacrMh9fN73MO0mWFAWnfZnMfrIv
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-outbox
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 12
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL2YTgdC79cv8nEgDTjgl42EoIqznPB0zlm
Name: 36000d310004edc000000000000000062
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 6
Event number: 0
Major, minor: 253, 1
Number of targets: 1
UUID: mpath-36000d310004edc000000000000000062
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-leases
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 22
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxH3yeLl2ff2q5saOE5Zh50VxLJ1xvKjnx
Name: 40907462--5b58--40c2--ade7--3031c8a01949-inbox
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 6
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76LdfwVhnwOn3jKuzwPzFz3YD89ITiLfESP
Name: 62d8cfdd--ee80--415b--b2c3--a7e6bda0dd97-inbox
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 15
Number of targets: 1
UUID: LVM-cziL8hR6PJcuVInDnPxEo0eBZIhWBRL2dcB9aVZUIYEFdTEycDZ6dzFUKAZOILr0
Name: 40907462--5b58--40c2--ade7--3031c8a01949-outbox
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 3
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76Ly7B9hY5Cv1R3OZMBvvdOeJYmjfVqTOYA
Name: 36000d310004edc000000000000000061
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 12
Event number: 0
Major, minor: 253, 19
Number of targets: 1
UUID: mpath-36000d310004edc000000000000000061
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-inbox
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 24
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxDXw6Io0J3keGR3UX05yvMXRvW0uXeqVj
Name:
40907462--5b58--40c2--ade7--3031c8a01949-397c0f40--ed70--4670--af89--55c35de929d6
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 28
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76L1qnbIZZK5hI4rifrjF67A0jWV40AbDpD
Name: 40907462--5b58--40c2--ade7--3031c8a01949-ids
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 5
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76LnXcQhEZXHTC8BBDcXESXpOseg10dloSL
Name:
4e4afb2f--2c86--44ee--90ed--58220664c8c4-9d1ed3fd--d6ef--4bbc--b222--7687134b8fc3
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 8
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxiNB5ctRaxdWcoMKkehCgD63pjdAmyAVY
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-master
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 25
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3Vx5GIf3Zz87c6QCwXQavDW8oCAreNOSksj
Name:
40907462--5b58--40c2--ade7--3031c8a01949-aac32cf9--af0d--42c4--84f7--6c62fa62f23c
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 27
Number of targets: 1
UUID: LVM-NjRHVsQwHNrlrPMpsAeyphSlGuJDZ76L7ZBz0vA0dbJBqhVnzVajz917BxpilUbc
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-ids
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 253, 23
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3Vx1CWyOwQbKm26vS9YBrDoS1R7CLomler8
Name: 4e4afb2f--2c86--44ee--90ed--58220664c8c4-outbox
State: ACTIVE
Read Ahead: 8192
Tables present: LIVE
Open count: 0
Event number: 0
Major, minor: 253, 21
Number of targets: 1
UUID: LVM-khBMu21R1pt4KdUHcTIQQjKAIXVEq3VxM3nQgfdZ2UXRTZGpZrvOgwPAFORBId38
[root@barbera log]# cat /etc/multipath.conf
# VDSM REVISION 1.3
defaults {
polling_interval 5
no_path_retry fail
user_friendly_names no
flush_on_last_del yes
fast_io_fail_tmo 5
dev_loss_tmo 30
max_fds 4096
}
# Remove devices entries when overrides section is available.
devices {
device {
# These settings overrides built-in devices settings. It does not apply
# to devices without built-in settings (these use the settings in the
# "defaults" section), or to devices defined in the "devices" section.
# Note: This is not available yet on Fedora 21. For more info see
# https://bugzilla.redhat.com/1253799
all_devs yes
no_path_retry fail
}
}
# Enable when this section is available on all supported platforms.
# Options defined here override device specific options embedded into
# multipathd.
#
# overrides {
# no_path_retry fail
# }
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
