Re: [ovirt-users] [HELP] unmapping a deleted storage domain triggers crash ovirt

2016-11-21 Thread Andrea Ghelardi 
Hello Nir,

Actually Ovirt3.5 host crashes even if the storage domain has been destroyed / 
removed from ovirt (hence ovirt has no knowledge of this storage and its mpath 
device and don’t bother about it). But I guess this is what you refer to when 
you write about “handling of unmapped luns”.

However, now that I caught you online, could you confirm that deleting an 
entity ( storage domain) from Ovirt should release the sanlock?
In other words: what is the supported way to delete an iscsi SAN volume?
My current approach is:

1)  Maintenance storage domain from ovirt4

2)  Detach+remove or just destroy storage domain from ovirt4

3)  Wait ~10 minutes

4)  Access SAN control panel and unmap+delete  associated volume.

5)  Eventually, access every host server and remove mpath device manually 
(multipath -f )

SAN is a DELL Compellent SC040
Cheers
AG

From: Nir Soffer [mailto:nsof...@redhat.com]
Sent: Thursday, November 17, 2016 18:59
To: Andrea Ghelardi <a.ghela...@iontrading.com>
Cc: users@ovirt.org
Subject: Re: [ovirt-users] [HELP] unmapping a deleted storage domain triggers 
crash ovirt

On Thu, Nov 17, 2016 at 5:48 PM, Andrea Ghelardi 
<a.ghela...@iontrading.com<mailto:a.ghela...@iontrading.com>> wrote:
Hello again,

Just to provide an update for future reference, I’ve been unable to sort this 
problem out.
However, Ovirt4 does not show this failure.

Handling of unmapped luns was fixed in 3.6.

The best way is to upgrade to 3.6, and then upgrade to 4, and I think this is 
also
the only supported upgrade path.


My current approach to move VMs from one installation to another is:
1)Shutdown VM in ovirt3.
2)Maintenance VM storage - do you mean the storage server?
3)Clone VM storage at LUN level
4)Import cloned storage in ovirt4
5)Import VM in ovirt4
If everything is ok:

You must put the storage domain to maintenance here.

Since we don't support yet removal of pv from a storage domain, you cannot
remove a lun from a storage, you must remove the entire storage domain.

6a)  detach + remove original VM storage in ovirt3
7a)  remove mappings for original VM storage LUN in ovirt3; this imply 
cluster instability (sanlock crash on hosts);
8a)  recover cluster; this implies shutting down host (and its VM) one 
by one
9a)  delete unmapped original LUN from storage manager

When Sanlock freeze on host, the only solution found so far is to reboot it. 
This has to be done on all host experiencing this problem (which basically 
means: all hosts in the cluster)

If sanlock freezed it means that you unmapped a lun that was used by sanlock.
This is not supported.


Cheers
AG


From: Andrea Ghelardi
Sent: Tuesday, November 08, 2016 16:40
To: users@ovirt.org<mailto:users@ovirt.org>
Subject: [HELP] unmapping a deleted storage domain triggers crash ovirt

Hello people,

something’s not right in my ovirt infrastructure.
I currently have two different ovirt installation:
Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt 4.0.4

I’m currently moving my guests from ovirt3 to ovirt4.
Since iscsi storage is linked to both installations, my high level approach is:

1)  Shutdown VM in ovirt3.

2)  Maintenance + detach + remove VM storage in ovirt3

3)  Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4

4)  Import storage in ovirt4

5)  Import VM in ovirt4

6)  Run and cheers with high grade liquor. GOTO step 1 for different VM.

Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes crazy 
and eventually forces me to reboot all hosts one by one.
I tried different low level approaches to unmap LUN mpath’ed from hosts with 
inconsistent results.
A notable error log extract is:
Nov  8 15:30:52 sovana vdsm root ERROR Process failed with rc=1 out='\nudevadm 
settle - timeout of 5 seconds reached, the event queue contains:\n  
/sys/devices/virtual/block/dm-39 (8603)\n  /sys/devices/virtual/block/dm-39 
(8604)\n  /sys
/devices/virtual/block/dm-39 (8605)\n  /sys/devices/virtual/block/dm-39 
(8606)\n  /sys/devices/virtual/block/dm-39 (8607)\n  
/sys/devices/virtual/block/dm-39 (8608)\n  /sys/devices/virtual/block/dm-39 
(8609)\n  /sys/devices/virtual/block
/dm-39 (8610)\n  /sys/devices/virtual/block/dm-39 (8611)\n  
/sys/devices/virtual/block/dm-39 (8612)\n  /sys/devices/virtual/block/dm-39 
(8613)\n  /sys/devices/virtual/block/dm-39 (8614)\n  
/sys/devices/virtual/block/dm-39 (8615)\n  /sys/
devices/virtual/block/dm-39 (8616)\n  /sys/devices/virtual/block/dm-39 (8617)\n 
 /sys/devices/virtual/block/dm-39 (8618)\n  /sys/devices/virtual/block/dm-39 
(8619)\n  /sys/devices/virtual/block/dm-39 (8620)\n  /sys/devices/virtual/block/
dm-39 (8621)\n  /sys/devices/virtual/block/dm-39 (8622)\n  
/sys/devices/virtual/block/dm-39 (8623)\n  /sys/devices/virtual/blo

Re: [ovirt-users] [HELP] unmapping a deleted storage domain triggers crash ovirt

2016-11-17 Thread Nir Soffer 
On Thu, Nov 17, 2016 at 5:48 PM, Andrea Ghelardi <a.ghela...@iontrading.com>
wrote:

> Hello again,
>
>
>
> Just to provide an update for future reference, I’ve been unable to sort
> this problem out.
>
> However, Ovirt4 does not show this failure.
>

Handling of unmapped luns was fixed in 3.6.

The best way is to upgrade to 3.6, and then upgrade to 4, and I think this
is also
the only supported upgrade path.


>
>
> My current approach to move VMs from one installation to another is:
>
> 1)Shutdown VM in ovirt3.
>
> 2)Maintenance VM storage - do you mean the storage server?
>
> 3)Clone VM storage at LUN level
>
> 4)Import cloned storage in ovirt4
>
> 5)Import VM in ovirt4
>
> If everything is ok:
>

You must put the storage domain to maintenance here.

Since we don't support yet removal of pv from a storage domain, you cannot
remove a lun from a storage, you must remove the entire storage domain.


> 6a)  detach + remove original VM storage in ovirt3
>
> 7a)  remove mappings for original VM storage LUN in ovirt3; this
> imply cluster instability (sanlock crash on hosts);
>
> 8a)  recover cluster; this implies shutting down host (and its VM)
> one by one
>
> 9a)  delete unmapped original LUN from storage manager
>

>
> When Sanlock freeze on host, the only solution found so far is to reboot
> it. This has to be done on all host experiencing this problem (which
> basically means: all hosts in the cluster)
>

If sanlock freezed it means that you unmapped a lun that was used by
sanlock.
This is not supported.


>
>
> Cheers
>
> AG
>
>
>
>
>
> *From:* Andrea Ghelardi
> *Sent:* Tuesday, November 08, 2016 16:40
> *To:* users@ovirt.org
> *Subject:* [HELP] unmapping a deleted storage domain triggers crash ovirt
>
>
>
> Hello people,
>
>
>
> something’s not right in my ovirt infrastructure.
>
> I currently have two different ovirt installation:
>
> Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
>
> Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt
> 4.0.4
>
>
>
> I’m currently moving my guests from ovirt3 to ovirt4.
>
> Since iscsi storage is linked to both installations, my high level
> approach is:
>
> 1)  Shutdown VM in ovirt3.
>
> 2)  Maintenance + detach + remove VM storage in ovirt3
>
> 3)  Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4
>
> 4)  Import storage in ovirt4
>
> 5)  Import VM in ovirt4
>
> 6)  Run and cheers with high grade liquor. GOTO step 1 for different
> VM.
>
>
>
> Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes
> crazy and eventually forces me to reboot all hosts one by one.
>
> I tried different low level approaches to unmap LUN mpath’ed from hosts
> with inconsistent results.
>
> A notable error log extract is:
>
> Nov  8 15:30:52 sovana vdsm root ERROR Process failed with rc=1
> out='\nudevadm settle - timeout of 5 seconds reached, the event queue
> contains:\n  /sys/devices/virtual/block/dm-39 (8603)\n
> /sys/devices/virtual/block/dm-39 (8604)\n  /sys
>
> /devices/virtual/block/dm-39 (8605)\n  /sys/devices/virtual/block/dm-39
> (8606)\n  /sys/devices/virtual/block/dm-39 (8607)\n
> /sys/devices/virtual/block/dm-39 (8608)\n  /sys/devices/virtual/block/dm-39
> (8609)\n  /sys/devices/virtual/block
>
> /dm-39 (8610)\n  /sys/devices/virtual/block/dm-39 (8611)\n
> /sys/devices/virtual/block/dm-39 (8612)\n  /sys/devices/virtual/block/dm-39
> (8613)\n  /sys/devices/virtual/block/dm-39 (8614)\n
> /sys/devices/virtual/block/dm-39 (8615)\n  /sys/
>
> devices/virtual/block/dm-39 (8616)\n  /sys/devices/virtual/block/dm-39
> (8617)\n  /sys/devices/virtual/block/dm-39 (8618)\n
> /sys/devices/virtual/block/dm-39 (8619)\n  /sys/devices/virtual/block/dm-39
> (8620)\n  /sys/devices/virtual/block/
>
> dm-39 (8621)\n  /sys/devices/virtual/block/dm-39 (8622)\n
> /sys/devices/virtual/block/dm-39 (8623)\n  /sys/devices/virtual/block/dm-39
> (8624)\n  /sys/devices/virtual/block/dm-39 (8625)\n
> /sys/devices/virtual/block/dm-39 (8626)\n  /sys/d
>
> evices/virtual/block/dm-39 (8627)\n  /sys/devices/virtual/block/dm-39
> (8628)\n  /sys/devices/virtual/block/dm-39 (8629)\n
> /sys/devices/virtual/block/dm-39 (8630)\n  /sys/devices/virtual/block/dm-39
> (8631)\n  /sys/devices/virtual/block/d
>
> m-39 (8632)\n  /sys/devices/virtual/block/dm-39 (8633)\n
> /sys/devices/virtual/block/dm-39 (8634)\n  /sys/devices/virtual/block/dm-39
> (8635)\n  /sys/devices/virtual/block/dm-39 (8636)\n
> /sys/devices/virtual/block/

Re: [ovirt-users] [HELP] unmapping a deleted storage domain triggers crash ovirt

2016-11-17 Thread Andrea Ghelardi 
Hello again,

Just to provide an update for future reference, I've been unable to sort this 
problem out.
However, Ovirt4 does not show this failure.

My current approach to move VMs from one installation to another is:
1)Shutdown VM in ovirt3.
2)Maintenance VM storage
3)Clone VM storage at LUN level
4)Import cloned storage in ovirt4
5)Import VM in ovirt4
If everything is ok:
6a)  detach + remove original VM storage in ovirt3
7a)  remove mappings for original VM storage LUN in ovirt3; this imply 
cluster instability (sanlock crash on hosts);
8a)  recover cluster; this implies shutting down host (and its VM) one 
by one
9a)  delete unmapped original LUN from storage manager

When Sanlock freeze on host, the only solution found so far is to reboot it. 
This has to be done on all host experiencing this problem (which basically 
means: all hosts in the cluster)

Cheers
AG


From: Andrea Ghelardi
Sent: Tuesday, November 08, 2016 16:40
To: users@ovirt.org
Subject: [HELP] unmapping a deleted storage domain triggers crash ovirt

Hello people,

something's not right in my ovirt infrastructure.
I currently have two different ovirt installation:
Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt 4.0.4

I'm currently moving my guests from ovirt3 to ovirt4.
Since iscsi storage is linked to both installations, my high level approach is:

1)  Shutdown VM in ovirt3.

2)  Maintenance + detach + remove VM storage in ovirt3

3)  Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4

4)  Import storage in ovirt4

5)  Import VM in ovirt4

6)  Run and cheers with high grade liquor. GOTO step 1 for different VM.

Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes crazy 
and eventually forces me to reboot all hosts one by one.
I tried different low level approaches to unmap LUN mpath'ed from hosts with 
inconsistent results.
A notable error log extract is:
Nov  8 15:30:52 sovana vdsm root ERROR Process failed with rc=1 out='\nudevadm 
settle - timeout of 5 seconds reached, the event queue contains:\n  
/sys/devices/virtual/block/dm-39 (8603)\n  /sys/devices/virtual/block/dm-39 
(8604)\n  /sys
/devices/virtual/block/dm-39 (8605)\n  /sys/devices/virtual/block/dm-39 
(8606)\n  /sys/devices/virtual/block/dm-39 (8607)\n  
/sys/devices/virtual/block/dm-39 (8608)\n  /sys/devices/virtual/block/dm-39 
(8609)\n  /sys/devices/virtual/block
/dm-39 (8610)\n  /sys/devices/virtual/block/dm-39 (8611)\n  
/sys/devices/virtual/block/dm-39 (8612)\n  /sys/devices/virtual/block/dm-39 
(8613)\n  /sys/devices/virtual/block/dm-39 (8614)\n  
/sys/devices/virtual/block/dm-39 (8615)\n  /sys/
devices/virtual/block/dm-39 (8616)\n  /sys/devices/virtual/block/dm-39 (8617)\n 
 /sys/devices/virtual/block/dm-39 (8618)\n  /sys/devices/virtual/block/dm-39 
(8619)\n  /sys/devices/virtual/block/dm-39 (8620)\n  /sys/devices/virtual/block/
dm-39 (8621)\n  /sys/devices/virtual/block/dm-39 (8622)\n  
/sys/devices/virtual/block/dm-39 (8623)\n  /sys/devices/virtual/block/dm-39 
(8624)\n  /sys/devices/virtual/block/dm-39 (8625)\n  
/sys/devices/virtual/block/dm-39 (8626)\n  /sys/d
evices/virtual/block/dm-39 (8627)\n  /sys/devices/virtual/block/dm-39 (8628)\n  
/sys/devices/virtual/block/dm-39 (8629)\n  /sys/devices/virtual/block/dm-39 
(8630)\n  /sys/devices/virtual/block/dm-39 (8631)\n  
/sys/devices/virtual/block/d
m-39 (8632)\n  /sys/devices/virtual/block/dm-39 (8633)\n  
/sys/devices/virtual/block/dm-39 (8634)\n  /sys/devices/virtual/block/dm-39 
(8635)\n  /sys/devices/virtual/block/dm-39 (8636)\n  
/sys/devices/virtual/block/dm-39 (8637)\n  /sys/de
vices/virtual/block/dm-39 (8638)\n  /sys/devices/virtual/block/dm-39 (8639)\n  
/sys/devices/virtual/block/dm-39 (8640)\n  /sys/devices/virtual/block/dm-39 
(8641)\n  /sys/devices/virtual/block/dm-39 (8642)\n  
/sys/devices/virtual/block/dm
-39 (8643)\n  /sys/devices/virtual/block/dm-39 (8644)\n  
/sys/devices/virtual/block/dm-39 (8645)\n  /sys/devices/virtual/block/dm-39 
(8646)\n  /sys/devices/virtual/block/d

I really need your help to sort this out as I'm actually blocked in my task.

Why mapping changes triggers ovirt crash on a storage which ovirt should not 
care about?
Thanks


Andrea Ghelardi

+39 050 2203 71 | www.iongroup.com<http://www.iongroup.com/> | 
a.ghela...@iontrading.com<mailto:a.ghela...@iontrading.com>
Via San Martino, 52 - 56125 Pisa - ITALY

This email and any attachments may contain information which is confidential 
and/or privileged. The information is intended exclusively for the addressee 
and the views expressed may not be official policy, but the personal views of 
the originator. If you are not the intended recipient, be aware that any 
disclosure, copying, distribution or use of the contents is prohibited. If you 
have received this email

[ovirt-users] [HELP] unmapping a deleted storage domain triggers crash ovirt

2016-11-08 Thread Andrea Ghelardi 
Hello people,

something's not right in my ovirt infrastructure.
I currently have two different ovirt installation:
Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt 4.0.4

I'm currently moving my guests from ovirt3 to ovirt4.
Since iscsi storage is linked to both installations, my high level approach is:

1)  Shutdown VM in ovirt3.

2)  Maintenance + detach + remove VM storage in ovirt3

3)  Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4

4)  Import storage in ovirt4

5)  Import VM in ovirt4

6)  Run and cheers with high grade liquor. GOTO step 1 for different VM.

Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes crazy 
and eventually forces me to reboot all hosts one by one.
I tried different low level approaches to unmap LUN mpath'ed from hosts with 
inconsistent results.
A notable error log extract is:
Nov  8 15:30:52 sovana vdsm root ERROR Process failed with rc=1 out='\nudevadm 
settle - timeout of 5 seconds reached, the event queue contains:\n  
/sys/devices/virtual/block/dm-39 (8603)\n  /sys/devices/virtual/block/dm-39 
(8604)\n  /sys
/devices/virtual/block/dm-39 (8605)\n  /sys/devices/virtual/block/dm-39 
(8606)\n  /sys/devices/virtual/block/dm-39 (8607)\n  
/sys/devices/virtual/block/dm-39 (8608)\n  /sys/devices/virtual/block/dm-39 
(8609)\n  /sys/devices/virtual/block
/dm-39 (8610)\n  /sys/devices/virtual/block/dm-39 (8611)\n  
/sys/devices/virtual/block/dm-39 (8612)\n  /sys/devices/virtual/block/dm-39 
(8613)\n  /sys/devices/virtual/block/dm-39 (8614)\n  
/sys/devices/virtual/block/dm-39 (8615)\n  /sys/
devices/virtual/block/dm-39 (8616)\n  /sys/devices/virtual/block/dm-39 (8617)\n 
 /sys/devices/virtual/block/dm-39 (8618)\n  /sys/devices/virtual/block/dm-39 
(8619)\n  /sys/devices/virtual/block/dm-39 (8620)\n  /sys/devices/virtual/block/
dm-39 (8621)\n  /sys/devices/virtual/block/dm-39 (8622)\n  
/sys/devices/virtual/block/dm-39 (8623)\n  /sys/devices/virtual/block/dm-39 
(8624)\n  /sys/devices/virtual/block/dm-39 (8625)\n  
/sys/devices/virtual/block/dm-39 (8626)\n  /sys/d
evices/virtual/block/dm-39 (8627)\n  /sys/devices/virtual/block/dm-39 (8628)\n  
/sys/devices/virtual/block/dm-39 (8629)\n  /sys/devices/virtual/block/dm-39 
(8630)\n  /sys/devices/virtual/block/dm-39 (8631)\n  
/sys/devices/virtual/block/d
m-39 (8632)\n  /sys/devices/virtual/block/dm-39 (8633)\n  
/sys/devices/virtual/block/dm-39 (8634)\n  /sys/devices/virtual/block/dm-39 
(8635)\n  /sys/devices/virtual/block/dm-39 (8636)\n  
/sys/devices/virtual/block/dm-39 (8637)\n  /sys/de
vices/virtual/block/dm-39 (8638)\n  /sys/devices/virtual/block/dm-39 (8639)\n  
/sys/devices/virtual/block/dm-39 (8640)\n  /sys/devices/virtual/block/dm-39 
(8641)\n  /sys/devices/virtual/block/dm-39 (8642)\n  
/sys/devices/virtual/block/dm
-39 (8643)\n  /sys/devices/virtual/block/dm-39 (8644)\n  
/sys/devices/virtual/block/dm-39 (8645)\n  /sys/devices/virtual/block/dm-39 
(8646)\n  /sys/devices/virtual/block/d

I really need your help to sort this out as I'm actually blocked in my task.

Why mapping changes triggers ovirt crash on a storage which ovirt should not 
care about?
Thanks


Andrea Ghelardi

+39 050 2203 71 | www.iongroup.com | 
a.ghela...@iontrading.com
Via San Martino, 52 - 56125 Pisa - ITALY

This email and any attachments may contain information which is confidential 
and/or privileged. The information is intended exclusively for the addressee 
and the views expressed may not be official policy, but the personal views of 
the originator. If you are not the intended recipient, be aware that any 
disclosure, copying, distribution or use of the contents is prohibited. If you 
have received this email and any file transmitted with it in error, please 
notify the sender by telephone or return email immediately and delete the 
material from your computer. Internet communications are not secure and ION 
Trading is not responsible for their abuse by third parties, nor for any 
alteration or corruption in transmission, nor for any damage or loss caused by 
any virus or other defect. ION Trading accepts no liability or responsibility 
arising out of or in any way connected to this email.

[iON_HBlu_small]
Automation through innovation

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users