Re: [ovirt-users] best way to remove SAN lun

2017-03-05 Thread Nir Soffer
On Sun, Mar 5, 2017 at 2:40 PM, Pavel Gashev <p...@acronis.com> wrote:
> Please also consider a case when a single iSCSI target has several LUNs and 
> you remove one of them.
> In this case you should not logout.

Right. ovirt manages connections to storage. When you remove the last
usage of a connection, we should disconnect from the target.

If this is not the case, this is ovirt-engine bug.

>
> -Original Message-
> From: <users-boun...@ovirt.org> on behalf of Nelson Lameiras 
> <nelson.lamei...@lyra-network.com>
> Date: Friday, 3 March 2017 at 20:55
> To: Nir Soffer <nsof...@redhat.com>
> Cc: users <users@ovirt.org>
> Subject: Re: [ovirt-users] best way to remove SAN lun
>
> Hello Nir,
>
> I think I was not clear in my explanations, let me try again :
>
> we have a oVirt 4.0.5.5 cluster with multiple hosts (centos 7.2).
> In this cluster, we added a SAN volume (iscsi) a few months ago directly in 
> GUI
> Later we had to remove a DATA volume (SAN iscsi). Below the steps we have 
> taken :
>
> 1- we migrated all disks outside the volume (oVirt)
> 2- we put volume on maintenance (oVirt)
> 3- we detach volume (oVirt)
> 4- we removed/destroyed volume (oVirt)
>
> In SAN :
> 5- we put it offline on SAN
> 6- we delete it from SAN
>
> We thought this would be enough, but later we has a serious incident when log 
> partition went full (partially our fault) :
> /var/log/messages was continuously logging that it was still trying to reach 
> the SAN volumes (we have since taken care of the issue of log space issue => 
> more aggressive logrotate, etc)
>
> The real solution was to add two more steps, using shell in ALL hosts :
> 4a - logout from SAN : iscsiadm -m node --logout -T iqn.
> 4b - remove iscsi targets : rm -fr /var/lib/iscsi/nodes/iqn.X
>
> This effectively solved our problem, but was fastidious since we had to do it 
> manually in all hosts (imagine if we had hundreds of hosts...)
>
> So my question was : shouldn't it be oVirt job to system "logout" and "remove 
> iscsi targets" automatically when a volume is removed from oVirt ? maybe not, 
> and I'm missing something?
>
> cordialement, regards,
>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70
> nelson.lamei...@lyra-network.com
>
> www.lyra-network.com | www.payzen.eu
>
>
>
>
>
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
> - Original Message -
> From: "Nir Soffer" <nsof...@redhat.com>
> To: "Nelson Lameiras" <nelson.lamei...@lyra-network.com>
> Cc: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
> <ali...@redhat.com>, "users" <users@ovirt.org>
> Sent: Wednesday, February 22, 2017 8:27:26 AM
> Subject: Re: [ovirt-users] best way to remove SAN lun
>
> On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras
> <nelson.lamei...@lyra-network.com> wrote:
>> Hello,
>>
>> Not sure it is the same issue, but we have had a "major" issue recently in 
>> our production system when removing a ISCSI volume from oVirt, and then 
>> removing it from SAN.
>
> What version? OS version?
>
> The order must be:
>
> 1. remove the LUN from storage domain
> will be available in next 4.1 release. in older versions you have
> to remove the storage domain
>
> 2. unzone the LUN on the server
>
> 3. remove the multipath devices and the paths on the nodes
>
>> The issue being that each host was still trying to access regularly to the 
>> SAN volume in spite of not being completely removed from oVirt.
>
> What do you mean by "not being completely removed"?
>
> Who was accessing the volume?
>
>> This led to an massive increase of error logs, which filled completely 
>> /var/log partition,
>
> Which log was full with errors?
>
>> which snowballed into crashing vdsm and other nasty consequences.
>
> You should have big enough /var/log to avoid such issues.
>
>>
>> Anyway, the solution was to manually logout from SAN (in each host) with 
>> iscsiadm and manually remove iscsi targets (again in each host). It was not 
>> difficult once the problem was found because currently we only have 3 hosts 
>> in this cluster, but I'm wondering what would happen if we had hundreds of 
>> hosts ?
>>
>> Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE 
>> still waiting to be included on this subject or should I write one ?
>
> We have RFE for this here:
> https://bugzilla.redhat.com/13

Re: [ovirt-users] best way to remove SAN lun

2017-03-05 Thread Pavel Gashev
Please also consider a case when a single iSCSI target has several LUNs and you 
remove one of them.
In this case you should not logout.

-Original Message-
From: <users-boun...@ovirt.org> on behalf of Nelson Lameiras 
<nelson.lamei...@lyra-network.com>
Date: Friday, 3 March 2017 at 20:55
To: Nir Soffer <nsof...@redhat.com>
Cc: users <users@ovirt.org>
Subject: Re: [ovirt-users] best way to remove SAN lun

Hello Nir,

I think I was not clear in my explanations, let me try again :

we have a oVirt 4.0.5.5 cluster with multiple hosts (centos 7.2).
In this cluster, we added a SAN volume (iscsi) a few months ago directly in GUI
Later we had to remove a DATA volume (SAN iscsi). Below the steps we have taken 
:

1- we migrated all disks outside the volume (oVirt)
2- we put volume on maintenance (oVirt)
3- we detach volume (oVirt)
4- we removed/destroyed volume (oVirt)

In SAN :
5- we put it offline on SAN
6- we delete it from SAN

We thought this would be enough, but later we has a serious incident when log 
partition went full (partially our fault) :
/var/log/messages was continuously logging that it was still trying to reach 
the SAN volumes (we have since taken care of the issue of log space issue => 
more aggressive logrotate, etc)

The real solution was to add two more steps, using shell in ALL hosts :
4a - logout from SAN : iscsiadm -m node --logout -T iqn.
4b - remove iscsi targets : rm -fr /var/lib/iscsi/nodes/iqn.X

This effectively solved our problem, but was fastidious since we had to do it 
manually in all hosts (imagine if we had hundreds of hosts...)

So my question was : shouldn't it be oVirt job to system "logout" and "remove 
iscsi targets" automatically when a volume is removed from oVirt ? maybe not, 
and I'm missing something?

cordialement, regards, 

Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Nir Soffer" <nsof...@redhat.com>
To: "Nelson Lameiras" <nelson.lamei...@lyra-network.com>
Cc: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
<ali...@redhat.com>, "users" <users@ovirt.org>
Sent: Wednesday, February 22, 2017 8:27:26 AM
Subject: Re: [ovirt-users] best way to remove SAN lun

On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras
<nelson.lamei...@lyra-network.com> wrote:
> Hello,
>
> Not sure it is the same issue, but we have had a "major" issue recently in 
> our production system when removing a ISCSI volume from oVirt, and then 
> removing it from SAN.

What version? OS version?

The order must be:

1. remove the LUN from storage domain
will be available in next 4.1 release. in older versions you have
to remove the storage domain

2. unzone the LUN on the server

3. remove the multipath devices and the paths on the nodes

> The issue being that each host was still trying to access regularly to the 
> SAN volume in spite of not being completely removed from oVirt.

What do you mean by "not being completely removed"?

Who was accessing the volume?

> This led to an massive increase of error logs, which filled completely 
> /var/log partition,

Which log was full with errors?

> which snowballed into crashing vdsm and other nasty consequences.

You should have big enough /var/log to avoid such issues.

>
> Anyway, the solution was to manually logout from SAN (in each host) with 
> iscsiadm and manually remove iscsi targets (again in each host). It was not 
> difficult once the problem was found because currently we only have 3 hosts 
> in this cluster, but I'm wondering what would happen if we had hundreds of 
> hosts ?
>
> Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE 
> still waiting to be included on this subject or should I write one ?

We have RFE for this here:
https://bugzilla.redhat.com/1310330

But you must understand that ovirt does not control your storage server,
you are responsible to add devices on the storage server, and remove
them. We are only consuming the devices.

Even we we provide a way to remove devices on all hosts, you will have
to remove the device on the storage server before removing it from
hosts. If not, ovirt will find the removed devices again in the next
scsi rescan,
and we do lot of these to support automatic discovery of new devices
or resized devices.

Nir

>
> cordialement, regards,
>
>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70
> nelson.lamei...@lyra-network.com
>
> www.lyra-network.com | www.payzen.eu
>
>
>
>
>

Re: [ovirt-users] best way to remove SAN lun

2017-03-03 Thread Nelson Lameiras
Hello Nir,

I think I was not clear in my explanations, let me try again :

we have a oVirt 4.0.5.5 cluster with multiple hosts (centos 7.2).
In this cluster, we added a SAN volume (iscsi) a few months ago directly in GUI
Later we had to remove a DATA volume (SAN iscsi). Below the steps we have taken 
:

1- we migrated all disks outside the volume (oVirt)
2- we put volume on maintenance (oVirt)
3- we detach volume (oVirt)
4- we removed/destroyed volume (oVirt)

In SAN :
5- we put it offline on SAN
6- we delete it from SAN

We thought this would be enough, but later we has a serious incident when log 
partition went full (partially our fault) :
/var/log/messages was continuously logging that it was still trying to reach 
the SAN volumes (we have since taken care of the issue of log space issue => 
more aggressive logrotate, etc)

The real solution was to add two more steps, using shell in ALL hosts :
4a - logout from SAN : iscsiadm -m node --logout -T iqn.
4b - remove iscsi targets : rm -fr /var/lib/iscsi/nodes/iqn.X

This effectively solved our problem, but was fastidious since we had to do it 
manually in all hosts (imagine if we had hundreds of hosts...)

So my question was : shouldn't it be oVirt job to system "logout" and "remove 
iscsi targets" automatically when a volume is removed from oVirt ? maybe not, 
and I'm missing something?

cordialement, regards, 

Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Nir Soffer" <nsof...@redhat.com>
To: "Nelson Lameiras" <nelson.lamei...@lyra-network.com>
Cc: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
<ali...@redhat.com>, "users" <users@ovirt.org>
Sent: Wednesday, February 22, 2017 8:27:26 AM
Subject: Re: [ovirt-users] best way to remove SAN lun

On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras
<nelson.lamei...@lyra-network.com> wrote:
> Hello,
>
> Not sure it is the same issue, but we have had a "major" issue recently in 
> our production system when removing a ISCSI volume from oVirt, and then 
> removing it from SAN.

What version? OS version?

The order must be:

1. remove the LUN from storage domain
will be available in next 4.1 release. in older versions you have
to remove the storage domain

2. unzone the LUN on the server

3. remove the multipath devices and the paths on the nodes

> The issue being that each host was still trying to access regularly to the 
> SAN volume in spite of not being completely removed from oVirt.

What do you mean by "not being completely removed"?

Who was accessing the volume?

> This led to an massive increase of error logs, which filled completely 
> /var/log partition,

Which log was full with errors?

> which snowballed into crashing vdsm and other nasty consequences.

You should have big enough /var/log to avoid such issues.

>
> Anyway, the solution was to manually logout from SAN (in each host) with 
> iscsiadm and manually remove iscsi targets (again in each host). It was not 
> difficult once the problem was found because currently we only have 3 hosts 
> in this cluster, but I'm wondering what would happen if we had hundreds of 
> hosts ?
>
> Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE 
> still waiting to be included on this subject or should I write one ?

We have RFE for this here:
https://bugzilla.redhat.com/1310330

But you must understand that ovirt does not control your storage server,
you are responsible to add devices on the storage server, and remove
them. We are only consuming the devices.

Even we we provide a way to remove devices on all hosts, you will have
to remove the device on the storage server before removing it from
hosts. If not, ovirt will find the removed devices again in the next
scsi rescan,
and we do lot of these to support automatic discovery of new devices
or resized devices.

Nir

>
> cordialement, regards,
>
>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70
> nelson.lamei...@lyra-network.com
>
> www.lyra-network.com | www.payzen.eu
>
>
>
>
>
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
> ----- Original Message -----
> From: "Nir Soffer" <nsof...@redhat.com>
> To: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
> <ali...@redhat.com>
> Cc: "users" <users@ovirt.org>
> Sent: Tuesday, February 21, 2017 6:32:18 PM
> Subject: Re: [ovirt-users] best wa

Re: [ovirt-users] best way to remove SAN lun

2017-02-22 Thread Yaniv Kaul
On Wed, Feb 22, 2017 at 9:27 AM Nir Soffer <nsof...@redhat.com> wrote:

> On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras
> <nelson.lamei...@lyra-network.com> wrote:
> > Hello,
> >
> > Not sure it is the same issue, but we have had a "major" issue recently
> in our production system when removing a ISCSI volume from oVirt, and then
> removing it from SAN.
>
> What version? OS version?
>
> The order must be:
>
> 1. remove the LUN from storage domain
> will be available in next 4.1 release. in older versions you have
> to remove the storage domain
>
> 2. unzone the LUN on the server
>
> 3. remove the multipath devices and the paths on the nodes
>
> > The issue being that each host was still trying to access regularly to
> the SAN volume in spite of not being completely removed from oVirt.
>
> What do you mean by "not being completely removed"?
>
> Who was accessing the volume?
>
> > This led to an massive increase of error logs, which filled completely
> /var/log partition,
>
> Which log was full with errors?
>
> > which snowballed into crashing vdsm and other nasty consequences.
>
> You should have big enough /var/log to avoid such issues.
>

- Log rotation should be set better not to consume excessive amounts of
space.
I'm seeing /etc/vdsm/logrotate/vdsm - not sure why it's not under
/etc/logrotate.d . Looking at the file, seems like there's a 15M limit and
100 files, which translates to 1.5GB - and it is supposed to be compressed
(not sure XZ is a good choice - it's very CPU intensive).

Others (Gluster?) do not seem to have a size limit, just weekly. Need to
look at other components as well.
- At least on ovirt-node, we'd like to separate some directories to
different partitions. So for example core dumps (which should be limited as
well) on /var/core do not fill the same partition as /var/log is and thus
render the host unusable.
And again, looking at file, we have a 'size 0' on /var/log/core/*.dump -
and 'rotate 1' - not sure what it means - but it should not be in
/var/log/core, but /var/core, I reckon.
Y.


> >
> > Anyway, the solution was to manually logout from SAN (in each host) with
> iscsiadm and manually remove iscsi targets (again in each host). It was not
> difficult once the problem was found because currently we only have 3 hosts
> in this cluster, but I'm wondering what would happen if we had hundreds of
> hosts ?
> >
> > Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE
> still waiting to be included on this subject or should I write one ?
>
> We have RFE for this here:
> https://bugzilla.redhat.com/1310330
>
> But you must understand that ovirt does not control your storage server,
> you are responsible to add devices on the storage server, and remove
> them. We are only consuming the devices.
>
> Even we we provide a way to remove devices on all hosts, you will have
> to remove the device on the storage server before removing it from
> hosts. If not, ovirt will find the removed devices again in the next
> scsi rescan,
> and we do lot of these to support automatic discovery of new devices
> or resized devices.
>
> Nir
>
> >
> > cordialement, regards,
> >
> >
> > Nelson LAMEIRAS
> > Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> > Tel: +33 5 32 09 09 70 <+33%205%2032%2009%2009%2070>
> > nelson.lamei...@lyra-network.com
> >
> > www.lyra-network.com | www.payzen.eu
> >
> >
> >
> >
> >
> > Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
> >
> > - Original Message -
> > From: "Nir Soffer" <nsof...@redhat.com>
> > To: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" <
> ali...@redhat.com>
> > Cc: "users" <users@ovirt.org>
> > Sent: Tuesday, February 21, 2017 6:32:18 PM
> > Subject: Re: [ovirt-users] best way to remove SAN lun
> >
> > On Tue, Feb 21, 2017 at 7:25 PM, Gianluca Cecchi
> > <gianluca.cec...@gmail.com> wrote:
> >> On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer <nsof...@redhat.com> wrote:
> >>>
> >>> This is caused by active lvs on the remove storage domains that were
> not
> >>> deactivated during the removal. This is a very old known issue.
> >>>
> >>> You have remove the remove device mapper entries - you can see the
> devices
> >>> using:
> >>>
> >>> dmsetup status
> >>>
> >>> Then you can remove the mapping using:
> >>>
> >>> dmsetup remove device-nam

Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Nir Soffer
On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras
<nelson.lamei...@lyra-network.com> wrote:
> Hello,
>
> Not sure it is the same issue, but we have had a "major" issue recently in 
> our production system when removing a ISCSI volume from oVirt, and then 
> removing it from SAN.

What version? OS version?

The order must be:

1. remove the LUN from storage domain
will be available in next 4.1 release. in older versions you have
to remove the storage domain

2. unzone the LUN on the server

3. remove the multipath devices and the paths on the nodes

> The issue being that each host was still trying to access regularly to the 
> SAN volume in spite of not being completely removed from oVirt.

What do you mean by "not being completely removed"?

Who was accessing the volume?

> This led to an massive increase of error logs, which filled completely 
> /var/log partition,

Which log was full with errors?

> which snowballed into crashing vdsm and other nasty consequences.

You should have big enough /var/log to avoid such issues.

>
> Anyway, the solution was to manually logout from SAN (in each host) with 
> iscsiadm and manually remove iscsi targets (again in each host). It was not 
> difficult once the problem was found because currently we only have 3 hosts 
> in this cluster, but I'm wondering what would happen if we had hundreds of 
> hosts ?
>
> Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE 
> still waiting to be included on this subject or should I write one ?

We have RFE for this here:
https://bugzilla.redhat.com/1310330

But you must understand that ovirt does not control your storage server,
you are responsible to add devices on the storage server, and remove
them. We are only consuming the devices.

Even we we provide a way to remove devices on all hosts, you will have
to remove the device on the storage server before removing it from
hosts. If not, ovirt will find the removed devices again in the next
scsi rescan,
and we do lot of these to support automatic discovery of new devices
or resized devices.

Nir

>
> cordialement, regards,
>
>
> Nelson LAMEIRAS
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer
> Tel: +33 5 32 09 09 70
> nelson.lamei...@lyra-network.com
>
> www.lyra-network.com | www.payzen.eu
>
>
>
>
>
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE
>
> - Original Message -
> From: "Nir Soffer" <nsof...@redhat.com>
> To: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
> <ali...@redhat.com>
> Cc: "users" <users@ovirt.org>
> Sent: Tuesday, February 21, 2017 6:32:18 PM
> Subject: Re: [ovirt-users] best way to remove SAN lun
>
> On Tue, Feb 21, 2017 at 7:25 PM, Gianluca Cecchi
> <gianluca.cec...@gmail.com> wrote:
>> On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer <nsof...@redhat.com> wrote:
>>>
>>> This is caused by active lvs on the remove storage domains that were not
>>> deactivated during the removal. This is a very old known issue.
>>>
>>> You have remove the remove device mapper entries - you can see the devices
>>> using:
>>>
>>> dmsetup status
>>>
>>> Then you can remove the mapping using:
>>>
>>> dmsetup remove device-name
>>>
>>> Once you removed the stale lvs, you will be able to remove the multipath
>>> device and the underlying paths, and lvm will not complain about read
>>> errors.
>>>
>>> Nir
>>
>>
>> OK Nir, thanks for advising.
>>
>> So what I run with success on the 2 hosts
>>
>> [root@ovmsrv05 vdsm]# for dev in $(dmsetup status | grep
>> 900b1853--e192--4661--a0f9--7c7c396f6f49 | cut -d ":" -f 1)
>> do
>>dmsetup remove $dev
>> done
>> [root@ovmsrv05 vdsm]#
>>
>> and now I can run
>>
>> [root@ovmsrv05 vdsm]# multipath -f 3600a0b8000299902cd3c5501458f
>> [root@ovmsrv05 vdsm]#
>>
>> Also, with related names depending on host,
>>
>> previous maps to single devices were for example in ovmsrv05:
>>
>> 3600a0b8000299902cd3c5501458f dm-4 IBM ,1814  FAStT
>> size=2.0T features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw
>> |-+- policy='service-time 0' prio=0 status=enabled
>> | |- 0:0:0:2 sdb8:16  failed undef running
>> | `- 1:0:0:2 sdh8:112 failed undef running
>> `-+- policy='service-time 0' prio=0 status=enabled
>>   |- 0:0:1:2 sdg8:96  failed undef running
>>   `- 1:0:1:2 sdn8:208 failed undef running
>>
>> And removal of single path devices:
>>
>> [root@ovmsrv05 root]# for dev in sdb sdh sdg sdn
>> do
>>   echo 1 > /sys/block/${dev}/device/delete
>> done
>> [root@ovmsrv05 vdsm]#
>>
>> All clean now... ;-)
>
> Great!
>
> I think we should have a script doing all these steps.
>
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Nelson Lameiras
Hello,

Not sure it is the same issue, but we have had a "major" issue recently in our 
production system when removing a ISCSI volume from oVirt, and then removing it 
from SAN. The issue being that each host was still trying to access regularly 
to the SAN volume in spite of not being completely removed from oVirt. This led 
to an massive increase of error logs, which filled completely /var/log 
partition, which snowballed into crashing vdsm and other nasty consequences.

Anyway, the solution was to manually logout from SAN (in each host) with 
iscsiadm and manually remove iscsi targets (again in each host). It was not 
difficult once the problem was found because currently we only have 3 hosts in 
this cluster, but I'm wondering what would happen if we had hundreds of hosts ?

Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE still 
waiting to be included on this subject or should I write one ?

cordialement, regards, 


Nelson LAMEIRAS 
Ingénieur Systèmes et Réseaux / Systems and Networks engineer 
Tel: +33 5 32 09 09 70 
nelson.lamei...@lyra-network.com 

www.lyra-network.com | www.payzen.eu 





Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE

- Original Message -
From: "Nir Soffer" <nsof...@redhat.com>
To: "Gianluca Cecchi" <gianluca.cec...@gmail.com>, "Adam Litke" 
<ali...@redhat.com>
Cc: "users" <users@ovirt.org>
Sent: Tuesday, February 21, 2017 6:32:18 PM
Subject: Re: [ovirt-users] best way to remove SAN lun

On Tue, Feb 21, 2017 at 7:25 PM, Gianluca Cecchi
<gianluca.cec...@gmail.com> wrote:
> On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer <nsof...@redhat.com> wrote:
>>
>> This is caused by active lvs on the remove storage domains that were not
>> deactivated during the removal. This is a very old known issue.
>>
>> You have remove the remove device mapper entries - you can see the devices
>> using:
>>
>> dmsetup status
>>
>> Then you can remove the mapping using:
>>
>> dmsetup remove device-name
>>
>> Once you removed the stale lvs, you will be able to remove the multipath
>> device and the underlying paths, and lvm will not complain about read
>> errors.
>>
>> Nir
>
>
> OK Nir, thanks for advising.
>
> So what I run with success on the 2 hosts
>
> [root@ovmsrv05 vdsm]# for dev in $(dmsetup status | grep
> 900b1853--e192--4661--a0f9--7c7c396f6f49 | cut -d ":" -f 1)
> do
>dmsetup remove $dev
> done
> [root@ovmsrv05 vdsm]#
>
> and now I can run
>
> [root@ovmsrv05 vdsm]# multipath -f 3600a0b8000299902cd3c5501458f
> [root@ovmsrv05 vdsm]#
>
> Also, with related names depending on host,
>
> previous maps to single devices were for example in ovmsrv05:
>
> 3600a0b8000299902cd3c5501458f dm-4 IBM ,1814  FAStT
> size=2.0T features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | |- 0:0:0:2 sdb8:16  failed undef running
> | `- 1:0:0:2 sdh8:112 failed undef running
> `-+- policy='service-time 0' prio=0 status=enabled
>   |- 0:0:1:2 sdg8:96  failed undef running
>   `- 1:0:1:2 sdn8:208 failed undef running
>
> And removal of single path devices:
>
> [root@ovmsrv05 root]# for dev in sdb sdh sdg sdn
> do
>   echo 1 > /sys/block/${dev}/device/delete
> done
> [root@ovmsrv05 vdsm]#
>
> All clean now... ;-)

Great!

I think we should have a script doing all these steps.

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Nir Soffer
On Tue, Feb 21, 2017 at 7:25 PM, Gianluca Cecchi
 wrote:
> On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer  wrote:
>>
>> This is caused by active lvs on the remove storage domains that were not
>> deactivated during the removal. This is a very old known issue.
>>
>> You have remove the remove device mapper entries - you can see the devices
>> using:
>>
>> dmsetup status
>>
>> Then you can remove the mapping using:
>>
>> dmsetup remove device-name
>>
>> Once you removed the stale lvs, you will be able to remove the multipath
>> device and the underlying paths, and lvm will not complain about read
>> errors.
>>
>> Nir
>
>
> OK Nir, thanks for advising.
>
> So what I run with success on the 2 hosts
>
> [root@ovmsrv05 vdsm]# for dev in $(dmsetup status | grep
> 900b1853--e192--4661--a0f9--7c7c396f6f49 | cut -d ":" -f 1)
> do
>dmsetup remove $dev
> done
> [root@ovmsrv05 vdsm]#
>
> and now I can run
>
> [root@ovmsrv05 vdsm]# multipath -f 3600a0b8000299902cd3c5501458f
> [root@ovmsrv05 vdsm]#
>
> Also, with related names depending on host,
>
> previous maps to single devices were for example in ovmsrv05:
>
> 3600a0b8000299902cd3c5501458f dm-4 IBM ,1814  FAStT
> size=2.0T features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw
> |-+- policy='service-time 0' prio=0 status=enabled
> | |- 0:0:0:2 sdb8:16  failed undef running
> | `- 1:0:0:2 sdh8:112 failed undef running
> `-+- policy='service-time 0' prio=0 status=enabled
>   |- 0:0:1:2 sdg8:96  failed undef running
>   `- 1:0:1:2 sdn8:208 failed undef running
>
> And removal of single path devices:
>
> [root@ovmsrv05 root]# for dev in sdb sdh sdg sdn
> do
>   echo 1 > /sys/block/${dev}/device/delete
> done
> [root@ovmsrv05 vdsm]#
>
> All clean now... ;-)

Great!

I think we should have a script doing all these steps.

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Gianluca Cecchi
On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer  wrote:

> This is caused by active lvs on the remove storage domains that were not
> deactivated during the removal. This is a very old known issue.
>
> You have remove the remove device mapper entries - you can see the devices
> using:
>
> dmsetup status
>
> Then you can remove the mapping using:
>
> dmsetup remove device-name
>
> Once you removed the stale lvs, you will be able to remove the multipath
> device and the underlying paths, and lvm will not complain about read
> errors.
>
> Nir
>

OK Nir, thanks for advising.

So what I run with success on the 2 hosts

[root@ovmsrv05 vdsm]# for dev in $(dmsetup status | grep
900b1853--e192--4661--a0f9--7c7c396f6f49 | cut -d ":" -f 1)
do
   dmsetup remove $dev
done
[root@ovmsrv05 vdsm]#

and now I can run

[root@ovmsrv05 vdsm]# multipath -f 3600a0b8000299902cd3c5501458f
[root@ovmsrv05 vdsm]#

Also, with related names depending on host,

previous maps to single devices were for example in ovmsrv05:

3600a0b8000299902cd3c5501458f dm-4 IBM ,1814  FAStT
size=2.0T features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 0:0:0:2 sdb8:16  failed undef running
| `- 1:0:0:2 sdh8:112 failed undef running
`-+- policy='service-time 0' prio=0 status=enabled
  |- 0:0:1:2 sdg8:96  failed undef running
  `- 1:0:1:2 sdn8:208 failed undef running

And removal of single path devices:

[root@ovmsrv05 root]# for dev in sdb sdh sdg sdn
do
  echo 1 > /sys/block/${dev}/device/delete
done
[root@ovmsrv05 vdsm]#

All clean now... ;-)

Thanks again,

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Nir Soffer
On Tue, Feb 21, 2017 at 7:06 PM, Gianluca Cecchi
 wrote:
> The id of the removed storage domain was
> 900b1853-e192-4661-a0f9-7c7c396f6f49 and on the not yet reboted hosts I get
> this
> The multipath device was 3600a0b8000299902cd3c5501458f
>
> With pvs command I get this errors related to them:
>
> [root@ovmsrv06 ~]# pvs
>   /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of 4096
> at 0: Input/output error
>   /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of 4096
> at 2199023190016: Input/output error
>   /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of 4096
> at 2199023247360: Input/output error
>   /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of 4096
> at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0 of
> 4096 at 536805376: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0 of
> 4096 at 536862720: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0 of
> 4096 at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
> 4096 at 134152192: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
> 4096 at 134209536: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
> 4096 at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
> 4096 at 1073676288: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
> 4096 at 1073733632: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
> 4096 at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
> 4096 at 2147418112: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
> 4096 at 2147475456: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
> 4096 at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of 4096
> at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of 4096
> at 134152192: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of 4096
> at 134209536: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of 4096
> at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
> 4096 at 134152192: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
> 4096 at 134209536: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
> 4096 at 4096: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
> 4096 at 0: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
> 4096 at 1073676288: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
> 4096 at 1073733632: Input/output error
>   /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
> 4096 at 4096: Input/output error
>   PVVG
> Fmt  Attr PSize  PFree
>   /dev/cciss/c0d0p2 cl
> lvm2 a--  67.33g0
>   /dev/mapper/3600a0b8000299aa8d08b55014119
> 922b5269-ab56-4c4d-838f-49d33427e2ab lvm2 a--   4.00t 3.49t
> [root@ovmsrv06 ~]#
>
> How to clean?

This is caused by active lvs on the remove storage domains that were not
deactivated during the removal. This is a very old known issue.

You have remove the remove device mapper entries - you can see the devices
using:

dmsetup status

Then you can remove the mapping using:

dmsetup remove device-name

Once you removed the stale lvs, you will be able to remove the multipath
device and the underlying paths, and lvm will not complain about read errors.

Nir

>
> Gianluca
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list

Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Gianluca Cecchi
The id of the removed storage domain was
900b1853-e192-4661-a0f9-7c7c396f6f49 and on the not yet reboted hosts I get
this
The multipath device was 3600a0b8000299902cd3c5501458f

With pvs command I get this errors related to them:

[root@ovmsrv06 ~]# pvs
  /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of
4096 at 0: Input/output error
  /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of
4096 at 2199023190016: Input/output error
  /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of
4096 at 2199023247360: Input/output error
  /dev/mapper/3600a0b8000299902cd3c5501458f: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0
of 4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0
of 4096 at 536805376: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0
of 4096 at 536862720: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/metadata: read failed after 0
of 4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
4096 at 134152192: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
4096 at 134209536: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/outbox: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
4096 at 1073676288: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
4096 at 1073733632: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/xleases: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
4096 at 2147418112: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
4096 at 2147475456: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/leases: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of
4096 at 134152192: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of
4096 at 134209536: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/ids: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
4096 at 134152192: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
4096 at 134209536: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/inbox: read failed after 0 of
4096 at 4096: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
4096 at 0: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
4096 at 1073676288: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
4096 at 1073733632: Input/output error
  /dev/900b1853-e192-4661-a0f9-7c7c396f6f49/master: read failed after 0 of
4096 at 4096: Input/output error
  PVVG
  Fmt  Attr PSize  PFree
  /dev/cciss/c0d0p2 cl
  lvm2 a--  67.33g0
  /dev/mapper/3600a0b8000299aa8d08b55014119
922b5269-ab56-4c4d-838f-49d33427e2ab lvm2 a--   4.00t 3.49t
[root@ovmsrv06 ~]#

How to clean?

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Gianluca Cecchi
On Tue, Feb 21, 2017 at 5:42 PM, Andrea Ghelardi 
wrote:

> Hello Gianluca,
>
> I have ISCSI SAN here and not FC but process should be very similar.
>
> Correct operation is
>
> Storage maintenance -> detach -> remove
>
>
>
> Once you have removed the storage, Ovirt will release the sanlock.
>
> Note that your server will still “see” the device and multipath will still
> list it as active.
>
>
>
> You can now unmap LUN using your SAN manager.
>
> If your multipath is correctly configured, device will “fail” but nothing
> else (multipath and server won’t hang).
>
> You will be able to perform basic list operations via server shell without
> experiencing freeze or locks.*
>

Yes, but /var/log/messages fills up with multipath errors.
This is not desirable


> Ovirt will continue to scan new devices so you won’t be able to manually
> remove them until you unmap devices from SAN manager.
>
>
>
> If you need to remove devices manually, it is advised to follow this guide.
>
> https://access.redhat.com/documentation/en-US/Red_Hat_
> Enterprise_Linux/6/html/Storage_Administration_Guide/removing_devices.html
>
>
>
> * If you do face command freeze etc, re-enable LUN mappings and check your
> multipath.conf
>

 Yes, but the problem is that I receive error for the LUN previously
configured as storage domain.
I presume sum stale devices of kind "dm-??" have remained somewhere.

I could reboot one ot the 3 hosts and verified that now the layout is
correct on it.
But I can't easily reboot just now the remaining 2 hosts, so it would be
better to find what forces the multipath device in use

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Andrea Ghelardi
Hello Gianluca,
I have ISCSI SAN here and not FC but process should be very similar.
Correct operation is
Storage maintenance -> detach -> remove

Once you have removed the storage, Ovirt will release the sanlock.
Note that your server will still “see” the device and multipath will still list 
it as active.

You can now unmap LUN using your SAN manager.
If your multipath is correctly configured, device will “fail” but nothing else 
(multipath and server won’t hang).
You will be able to perform basic list operations via server shell without 
experiencing freeze or locks.*
Ovirt will continue to scan new devices so you won’t be able to manually remove 
them until you unmap devices from SAN manager.

If you need to remove devices manually, it is advised to follow this guide.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/removing_devices.html

* If you do face command freeze etc, re-enable LUN mappings and check your 
multipath.conf

Cheers
AG

From: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] On Behalf Of 
Gianluca Cecchi
Sent: Tuesday, February 21, 2017 4:19 PM
To: users 
Subject: [ovirt-users] best way to remove SAN lun

Hello,
currently I have a cluster of 3 hosts where each one has FC SAN connectivity to 
4 LUNs: 3 are already configured as storage domains (1TB, 2TB, 4TB), one is 
free, not allocated.
See here for screenshot:
https://drive.google.com/file/d/0BwoPbcrMv8mvRVZZMTlNcTQ5MGs/view?usp=sharing

At the moment the command "multipath -l" run on hosts shows all the 4 LUNs.

Now I want to do 2 things at storage array level:

- remove the 2TB storage domain LUN
- remove the 20Gb LUN not yet allocated

What is the correct workflow, supposing I have already emptied the 2TB from VM 
disks ad such?
Select 2Tb SD, then Datacenter subtab, then "maintenance", detach" and at the 
end "remove"?

I think I continue to see 4 LUNs at this point, correct?

Now I proceed with removal of lun at storage array level?

Should I select an SD line and then "Scan disks" to see refresh the SAN and see 
in multipath only 2 of them at the end?
Or any manual command at host level before removal from array?

Thanks in advance

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Nir Soffer
On Tue, Feb 21, 2017 at 6:12 PM, Adam Litke  wrote:
>
>
> On Tue, Feb 21, 2017 at 10:19 AM, Gianluca Cecchi
>  wrote:
>>
>> Hello,
>> currently I have a cluster of 3 hosts where each one has FC SAN
>> connectivity to 4 LUNs: 3 are already configured as storage domains (1TB,
>> 2TB, 4TB), one is free, not allocated.
>> See here for screenshot:
>>
>> https://drive.google.com/file/d/0BwoPbcrMv8mvRVZZMTlNcTQ5MGs/view?usp=sharing
>>
>> At the moment the command "multipath -l" run on hosts shows all the 4
>> LUNs.
>>
>> Now I want to do 2 things at storage array level:
>>
>> - remove the 2TB storage domain LUN
>> - remove the 20Gb LUN not yet allocated
>>
>> What is the correct workflow, supposing I have already emptied the 2TB
>> from VM disks ad such?
>> Select 2Tb SD, then Datacenter subtab, then "maintenance", detach" and at
>> the end "remove"?
>
>
> Yes, these should be your first steps.
>
>>
>> I think I continue to see 4 LUNs at this point, correct?

We do not manage devices, so this must be performed manually by
the system administrator.

To remove devices, you have to:

1. On the storage server, unzone the devices so the host(s) will not see them
This must be done before you remove the multipath device and
paths, otherwise
vdsm will discover the device again during the periodic scsi/fc rescans.

2. On all hosts ,remove the multipath device and the underlying scsi devices,
as explained here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing-path-to-storage-device.html

Nir

>
>
> Yes.
>
>>
>> Now I proceed with removal of lun at storage array level?
>>
>> Should I select an SD line and then "Scan disks" to see refresh the SAN
>> and see in multipath only 2 of them at the end?
>> Or any manual command at host level before removal from array?
>
>
> After removing the storage domains you should be able to remove the luns.  I
> am not extremely familiar with the multipath and low-level scsi commands but
> I would try the scan disks button and if the luns are not gone from your
> host you can manually remove them.  I think that involves removing the
> device from multipath (multipath -d) and deleting it from the scsi
> subsystem.
>
>> Thanks in advance
>
>
> Hope this helped you.
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Gianluca Cecchi
On Tue, Feb 21, 2017 at 5:12 PM, Adam Litke  wrote:

>
>
> On Tue, Feb 21, 2017 at 10:19 AM, Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> Hello,
>> currently I have a cluster of 3 hosts where each one has FC SAN
>> connectivity to 4 LUNs: 3 are already configured as storage domains (1TB,
>> 2TB, 4TB), one is free, not allocated.
>> See here for screenshot:
>> https://drive.google.com/file/d/0BwoPbcrMv8mvRVZZMTlNcTQ5MGs
>> /view?usp=sharing
>>
>> At the moment the command "multipath -l" run on hosts shows all the 4
>> LUNs.
>>
>> Now I want to do 2 things at storage array level:
>>
>> - remove the 2TB storage domain LUN
>> - remove the 20Gb LUN not yet allocated
>>
>> What is the correct workflow, supposing I have already emptied the 2TB
>> from VM disks ad such?
>> Select 2Tb SD, then Datacenter subtab, then "maintenance", detach" and at
>> the end "remove"?
>>
>
> Yes, these should be your first steps.
>
>
>> I think I continue to see 4 LUNs at this point, correct?
>>
>
> Yes.
>
>
>> Now I proceed with removal of lun at storage array level?
>>
>> Should I select an SD line and then "Scan disks" to see refresh the SAN
>> and see in multipath only 2 of them at the end?
>> Or any manual command at host level before removal from array?
>>
>
> After removing the storage domains you should be able to remove the luns.
> I am not extremely familiar with the multipath and low-level scsi commands
> but I would try the scan disks button and if the luns are not gone from
> your host you can manually remove them.  I think that involves removing the
> device from multipath (multipath -d) and deleting it from the scsi
> subsystem.
>
> Thanks in advance
>>
>
> Hope this helped you.
>
>
Hello,
the "Scan Disks" seems related to the particular storage domain selected in
storage tab, not overall FC SAN connectivity...
If I then select "manage domain", it still shows the now missing disks with
an exclamation mark aside
I try to follow standard RH EL 7 way for removal:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/removing_devices.html

I can remove at os level the 20Gb lun that was never used in oVirt, but for
the previous 2Tb storage domain related LUN I get the error that is in
use

[root@ovmsrv06 ~]# multipath -f 3600a0b8000299902cd3c5501458f
Feb 21 17:25:58 | 3600a0b8000299902cd3c5501458f: map in use
Feb 21 17:25:58 | failed to remove multipath map
3600a0b8000299902cd3c5501458f
[root@ovmsrv06 ~]#

[root@ovmsrv06 ~]# fuser /dev/mapper/3600a0b8000299902cd3c5501458f
[root@ovmsrv06 ~]#


[root@ovmsrv06 ~]# ll /dev/mapper/3600a0b8000299902cd3c5501458f
lrwxrwxrwx. 1 root root 7 Feb 21 17:25
/dev/mapper/3600a0b8000299902cd3c5501458f -> ../dm-4

[root@ovmsrv06 ~]# fuser /dev/dm-4
[root@ovmsrv06 ~]#

Strange thing is that vgs command returns differrent value on th three hosts

[root@ovmsrv05 vdsm]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  922b5269-ab56-4c4d-838f-49d33427e2ab   1  22   0 wz--n-  4.00t 3.49t
  cl_ovmsrv051   3   0 wz--n- 67.33g0
[root@ovmsrv05 vdsm]#


[root@ovmsrv06 ~]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  922b5269-ab56-4c4d-838f-49d33427e2ab   1  22   0 wz--n-  4.00t 3.49t
  cl 1   3   0 wz--n- 67.33g0
[root@ovmsrv06 ~]#

[root@ovmsrv07 vdsm]# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  900b1853-e192-4661-a0f9-7c7c396f6f49   1  10   0 wz--n-  2.00t 1.76t
  922b5269-ab56-4c4d-838f-49d33427e2ab   1  27   0 wz--n-  4.00t 3.34t
  cl 1   3   0 wz--n- 67.33g0
[root@ovmsrv07 vdsm]#

So no host as a VG the 1TB storage domain and In particular ovmsrv07 has a
VG of 2TB that I suspect was the previosu storage domain

[root@ovmsrv07 vdsm]# lvs 900b1853-e192-4661-a0f9-7c7c396f6f49
  LV   VG
Attr   LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  35b8834e-a429-4223-b293-51d562b6def4 900b1853-e192-4661-a0f9-7c7c396f6f49
-wi--- 128.00m
  7ed43974-1039-4a68-a8b3-321e7594fe4c 900b1853-e192-4661-a0f9-7c7c396f6f49
-wi--- 240.00g
  d7f6be37-0f6c-43e3-b0af-a511fc59c842 900b1853-e192-4661-a0f9-7c7c396f6f49
-wi--- 128.00m
  ids  900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a- 128.00m
  inbox900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a- 128.00m
  leases   900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a-   2.00g
  master   900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a-   1.00g
  metadata 900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a- 512.00m
  outbox   900b1853-e192-4661-a0f9-7c7c396f6f49
-wi-a- 128.00m
  xleases  

Re: [ovirt-users] best way to remove SAN lun

2017-02-21 Thread Adam Litke
On Tue, Feb 21, 2017 at 10:19 AM, Gianluca Cecchi  wrote:

> Hello,
> currently I have a cluster of 3 hosts where each one has FC SAN
> connectivity to 4 LUNs: 3 are already configured as storage domains (1TB,
> 2TB, 4TB), one is free, not allocated.
> See here for screenshot:
> https://drive.google.com/file/d/0BwoPbcrMv8mvRVZZMTlNcTQ5MGs/
> view?usp=sharing
>
> At the moment the command "multipath -l" run on hosts shows all the 4 LUNs.
>
> Now I want to do 2 things at storage array level:
>
> - remove the 2TB storage domain LUN
> - remove the 20Gb LUN not yet allocated
>
> What is the correct workflow, supposing I have already emptied the 2TB
> from VM disks ad such?
> Select 2Tb SD, then Datacenter subtab, then "maintenance", detach" and at
> the end "remove"?
>

Yes, these should be your first steps.


> I think I continue to see 4 LUNs at this point, correct?
>

Yes.


> Now I proceed with removal of lun at storage array level?
>
> Should I select an SD line and then "Scan disks" to see refresh the SAN
> and see in multipath only 2 of them at the end?
> Or any manual command at host level before removal from array?
>

After removing the storage domains you should be able to remove the luns.
I am not extremely familiar with the multipath and low-level scsi commands
but I would try the scan disks button and if the luns are not gone from
your host you can manually remove them.  I think that involves removing the
device from multipath (multipath -d) and deleting it from the scsi
subsystem.

Thanks in advance
>

Hope this helped you.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users