Re: [ovirt-users] Storage Problem After Host Update (v3.6)

2017-01-04 Thread Yaniv Kaul 
On Jan 5, 2017 9:05 AM, "Pavel Gashev" <p...@acronis.com> wrote:

Charles,

Please note the warning in logs about lvmetad. It's better to disable this
service if you invoke LVM utilitities manually. It caches some LVM metadata
but VDSM doesn't use it. So technically VDSM has different view on PVs.

+1.
We are going to disable it entirely in coming version.
Y.


--
*From:* Charles Tassell <char...@islandadmin.ca>
*Sent:* Jan 5, 2017 7:33 AM
*To:* Pavel Gashev; users@ovirt.org
*Subject:* Re: [ovirt-users] Storage Problem After Host Update (v3.6)

Hi Pavel,

   Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!


On 2017-01-04 06:13 PM, Pavel Gashev wrote:
> https://bugzilla.redhat.com/1363734
> https://bugzilla.redhat.com/1364339
> https://bugzilla.redhat.com/1366176
>
> Downgrade the lvm2 package.
> Or upgrade vdsm to 4.17.35
>
> -Original Message-
> From: <users-boun...@ovirt.org> on behalf of Charles Tassell <
char...@islandadmin.ca>
> Date: Wednesday 4 January 2017 at 23:22
> To: "users@ovirt.org" <users@ovirt.org>
> Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
>
> Hello,
>
> One of our admins did a package update on one of our 3.6 hosts
> through the web GUI and then rebooted the host.  Since then, it hasn't
> been able to properly mount storage and thus we can't move any VMs to
> it.  I rebooted it myself this afternoon and it took our whole cluster
> down for about a half hour, which was kind of concerning.
>
> We use multipath fibre channel for storage and when I do a vgscan or
> vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
> of PV ..." warning messages.  I also get the log entries below
> constantly repeating in vdsm.log.  Any ideas what's going on and how I
> fix it?
>
> CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
> Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
> of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use
> \\\'--activationmode partial\\\' to
> override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'
metadata\\\']\',)',)
> Thread-1844::ERROR::2017-01-04
> 16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
> {'message': 'Cannot activate Logical Volumes: (\'General Storage
> Exception: (\\\'5 [] [\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing
> activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
> Use \\\'--activationmode partial\\\' to
> override."]n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\
\'metadata\\\']\\\',)\',)',
> 'code': 570}}
> Thread-101::ERROR::2017-01-04
> 16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
> monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
>   self._produceDomain()
> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
> wrapper
>   value = meth(self, *a, **kw)
> File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
>   self.domain = sdCache.produce(self.sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
>   domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>   return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
>   domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
>   dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
>   return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
>   manifest = self.manifestClass(sdUUID)
>File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
>   metadata = selectMetadata(self.sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
>   if len(mdProvider) > 0:
> File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
>   return len(self.keys())
> File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
>   return list(self.__iter__())

Re: [ovirt-users] Storage Problem After Host Update (v3.6)

2017-01-04 Thread Pavel Gashev 
Charles,

Please note the warning in logs about lvmetad. It's better to disable this 
service if you invoke LVM utilitities manually. It caches some LVM metadata but 
VDSM doesn't use it. So technically VDSM has different view on PVs.


From: Charles Tassell <char...@islandadmin.ca>
Sent: Jan 5, 2017 7:33 AM
To: Pavel Gashev; users@ovirt.org
Subject: Re: [ovirt-users] Storage Problem After Host Update (v3.6)

Hi Pavel,

   Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v)
and that got us fixed back up!


On 2017-01-04 06:13 PM, Pavel Gashev wrote:
> https://bugzilla.redhat.com/1363734
> https://bugzilla.redhat.com/1364339
> https://bugzilla.redhat.com/1366176
>
> Downgrade the lvm2 package.
> Or upgrade vdsm to 4.17.35
>
> -Original Message-
> From: <users-boun...@ovirt.org> on behalf of Charles Tassell 
> <char...@islandadmin.ca>
> Date: Wednesday 4 January 2017 at 23:22
> To: "users@ovirt.org" <users@ovirt.org>
> Subject: [ovirt-users] Storage Problem After Host Update (v3.6)
>
> Hello,
>
> One of our admins did a package update on one of our 3.6 hosts
> through the web GUI and then rebooted the host.  Since then, it hasn't
> been able to properly mount storage and thus we can't move any VMs to
> it.  I rebooted it myself this afternoon and it took our whole cluster
> down for about a half hour, which was kind of concerning.
>
> We use multipath fibre channel for storage and when I do a vgscan or
> vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
> of PV ..." warning messages.  I also get the log entries below
> constantly repeating in vdsm.log.  Any ideas what's going on and how I
> fix it?
>
> CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
> Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
> of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use
> \\\'--activationmode partial\\\' to
> override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
> Thread-1844::ERROR::2017-01-04
> 16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
> {'message': 'Cannot activate Logical Volumes: (\'General Storage
> Exception: (\\\'5 [] [\\\'  WARNING: lvmetad is running but
> disabled. Restart lvmetad before enabling it!\\\', " Refusing
> activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
> Use \\\'--activationmode partial\\\' to
> override."]n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\\\',)\',)',
> 'code': 570}}
> Thread-101::ERROR::2017-01-04
> 16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
> monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
> Traceback (most recent call last):
> File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
>   self._produceDomain()
> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
> wrapper
>   value = meth(self, *a, **kw)
> File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
>   self.domain = sdCache.produce(self.sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
>   domain.getRealDomain()
> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>   return self._cache._realProduce(self._sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
>   domain = self._findDomain(sdUUID)
> File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
>   dom = findMethod(sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
>   return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
> File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
>   manifest = self.manifestClass(sdUUID)
>File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
>   metadata = selectMetadata(self.sdUUID)
> File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
>   if len(mdProvider) > 0:
> File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
>   return len(self.keys())
> File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
>   return list(self.__iter__())
> File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
>   self._dict.__iter__())
> File &quo

Re: [ovirt-users] Storage Problem After Host Update (v3.6)

2017-01-04 Thread Charles Tassell 

Hi Pavel,

  Thanks, I added the 3.6-snapshot repo to upgrade to vdsm 4.17.36 and 
then diagnosed a bunch of LVM problems (PV's set as missing in pvs -v) 
and that got us fixed back up!



On 2017-01-04 06:13 PM, Pavel Gashev wrote:

https://bugzilla.redhat.com/1363734
https://bugzilla.redhat.com/1364339
https://bugzilla.redhat.com/1366176

Downgrade the lvm2 package.
Or upgrade vdsm to 4.17.35

-Original Message-
From: <users-boun...@ovirt.org> on behalf of Charles Tassell 
<char...@islandadmin.ca>
Date: Wednesday 4 January 2017 at 23:22
To: "users@ovirt.org" <users@ovirt.org>
Subject: [ovirt-users] Storage Problem After Host Update (v3.6)

Hello,

One of our admins did a package update on one of our 3.6 hosts
through the web GUI and then rebooted the host.  Since then, it hasn't
been able to properly mount storage and thus we can't move any VMs to
it.  I rebooted it myself this afternoon and it took our whole cluster
down for about a half hour, which was kind of concerning.

We use multipath fibre channel for storage and when I do a vgscan or
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate
of PV ..." warning messages.  I also get the log entries below
constantly repeating in vdsm.log.  Any ideas what's going on and how I
fix it?

CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General
Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use
\\\'--activationmode partial\\\' to
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
{'message': 'Cannot activate Logical Volumes: (\'General Storage
Exception: (\\\'5 [] [\\\'  WARNING: lvmetad is running but
disabled. Restart lvmetad before enabling it!\\\', " Refusing
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.
Use \\\'--activationmode partial\\\' to
override."]n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\\\',)\',)',
'code': 570}}
Thread-101::ERROR::2017-01-04
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
  self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in
wrapper
  value = meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
  self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
  domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
  return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
  domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
  dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
  return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
  manifest = self.manifestClass(sdUUID)
   File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
  metadata = selectMetadata(self.sdUUID)
File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
  if len(mdProvider) > 0:
File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
  return len(self.keys())
File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
  return list(self.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
  self._dict.__iter__())
File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
  with self._accessWrapper():
File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  return self.gen.next()
File "/usr/share/vdsm/storage/persistentDict.py", line 152, in
_accessWrapper
  self.refresh()
File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
  lines = self._metaRW.readlines()
File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
  lvm.activateLVs(self._vgName, self._lvName)
File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
  _setLVAvailability(vgName, toActivate, "y")
File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
  raise error(str(e

Re: [ovirt-users] Storage Problem After Host Update (v3.6)

2017-01-04 Thread Pavel Gashev 
https://bugzilla.redhat.com/1363734
https://bugzilla.redhat.com/1364339
https://bugzilla.redhat.com/1366176

Downgrade the lvm2 package.
Or upgrade vdsm to 4.17.35

-Original Message-
From: <users-boun...@ovirt.org> on behalf of Charles Tassell 
<char...@islandadmin.ca>
Date: Wednesday 4 January 2017 at 23:22
To: "users@ovirt.org" <users@ovirt.org>
Subject: [ovirt-users] Storage Problem After Host Update (v3.6)

Hello,

   One of our admins did a package update on one of our 3.6 hosts 
through the web GUI and then rebooted the host.  Since then, it hasn't 
been able to properly mount storage and thus we can't move any VMs to 
it.  I rebooted it myself this afternoon and it took our whole cluster 
down for about a half hour, which was kind of concerning.

   We use multipath fibre channel for storage and when I do a vgscan or 
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate 
of PV ..." warning messages.  I also get the log entries below 
constantly repeating in vdsm.log.  Any ideas what's going on and how I 
fix it?

CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General 
Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation 
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use 
\\\'--activationmode partial\\\' to 
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04 
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': 
{'message': 'Cannot activate Logical Volumes: (\'General Storage 
Exception: (\\\'5 [] [\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\', " Refusing 
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  
Use \\\'--activationmode partial\\\' to 
override."]n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\\\',)\',)',
 
'code': 570}}
Thread-101::ERROR::2017-01-04 
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error 
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b
Traceback (most recent call last):
   File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
 self._produceDomain()
   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in 
wrapper
 value = meth(self, *a, **kw)
   File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
 self.domain = sdCache.produce(self.sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
 domain.getRealDomain()
   File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
 return self._cache._realProduce(self._sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
 domain = self._findDomain(sdUUID)
   File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
 dom = findMethod(sdUUID)
   File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
 return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
   File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
 manifest = self.manifestClass(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
 metadata = selectMetadata(self.sdUUID)
   File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
 if len(mdProvider) > 0:
   File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
 return len(self.keys())
   File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
 return list(self.__iter__())
   File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
 self._dict.__iter__())
   File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
 with self._accessWrapper():
   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
 return self.gen.next()
   File "/usr/share/vdsm/storage/persistentDict.py", line 152, in 
_accessWrapper
 self.refresh()
   File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
 lines = self._metaRW.readlines()
   File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
 lvm.activateLVs(self._vgName, self._lvName)
   File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
 _setLVAvailability(vgName, toActivate, "y")
   File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
 raise error(str(e))

LVM output:
[root@ovirt-01 mapper]# vgchange -a y
   WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used 
from both devices /dev/sdac and /dev/sdk
   Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk 
not /dev/

[ovirt-users] Storage Problem After Host Update (v3.6)

2017-01-04 Thread Charles Tassell 

Hello,

  One of our admins did a package update on one of our 3.6 hosts 
through the web GUI and then rebooted the host.  Since then, it hasn't 
been able to properly mount storage and thus we can't move any VMs to 
it.  I rebooted it myself this afternoon and it took our whole cluster 
down for about a half hour, which was kind of concerning.


  We use multipath fibre channel for storage and when I do a vgscan or 
vgdisplay -s everything shows up correctly but I get a lot of "Duplicate 
of PV ..." warning messages.  I also get the log entries below 
constantly repeating in vdsm.log.  Any ideas what's going on and how I 
fix it?


CannotActivateLogicalVolumes: Cannot activate Logical Volumes: ('General 
Storage Exception: (\'5 [] [\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\', " Refusing activation 
of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  Use 
\\\'--activationmode partial\\\' to 
override."]\\n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\',)',)
Thread-1844::ERROR::2017-01-04 
16:03:43,649::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': 
{'message': 'Cannot activate Logical Volumes: (\'General Storage 
Exception: (\\\'5 [] [\\\'  WARNING: lvmetad is running but 
disabled. Restart lvmetad before enabling it!\\\', " Refusing 
activation of partial LV 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/metadata.  
Use \\\'--activationmode partial\\\' to 
override."]n8dfcc83b-64d7-4ff7-9800-02ba3430ea7b/[\\\'metadata\\\']\\\',)\',)', 
'code': 570}}
Thread-101::ERROR::2017-01-04 
16:03:44,303::monitor::276::Storage.Monitor::(_monitorDomain) Error 
monitoring domain 8dfcc83b-64d7-4ff7-9800-02ba3430ea7b

Traceback (most recent call last):
  File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 769, in 
wrapper

value = meth(self, *a, **kw)
  File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain = sdCache.produce(self.sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain = self._findDomain(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 1402, in findDomain
return BlockStorageDomain(BlockStorageDomain.findDomainPath(sdUUID))
  File "/usr/share/vdsm/storage/blockSD.py", line 455, in __init__
manifest = self.manifestClass(sdUUID)
 File "/usr/share/vdsm/storage/blockSD.py", line 407, in __init__
metadata = selectMetadata(self.sdUUID)
  File "/usr/share/vdsm/storage/blockSD.py", line 375, in selectMetadata
if len(mdProvider) > 0:
  File "/usr/share/vdsm/storage/persistentDict.py", line 65, in __len__
return len(self.keys())
  File "/usr/share/vdsm/storage/persistentDict.py", line 110, in keys
return list(self.__iter__())
  File "/usr/share/vdsm/storage/persistentDict.py", line 107, in __iter__
self._dict.__iter__())
  File "/usr/share/vdsm/storage/persistentDict.py", line 225, in __iter__
with self._accessWrapper():
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 152, in 
_accessWrapper

self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 230, in refresh
lines = self._metaRW.readlines()
  File "/usr/share/vdsm/storage/blockSD.py", line 335, in readlines
lvm.activateLVs(self._vgName, self._lvName)
  File "/usr/share/vdsm/storage/lvm.py", line 1191, in activateLVs
_setLVAvailability(vgName, toActivate, "y")
  File "/usr/share/vdsm/storage/lvm.py", line 826, in _setLVAvailability
raise error(str(e))

LVM output:
[root@ovirt-01 mapper]# vgchange -a y
  WARNING: duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt is being used 
from both devices /dev/sdac and /dev/sdk
  Found duplicate PV 1oDZGSqoQ4ueh0XEMzILNleRoNY8ZDAt: using /dev/sdk 
not /dev/sdac

  Using duplicate PV /dev/sdk which is last seen, replacing /dev/sdac
  Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev 
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 
65:177
  Duplicate of PV EMfxl6-8coB-iUdZ-Dx4P-e5LZ-UcIF-kvIfXx dev 
/dev/mapper/360060160ec732300fa5fdde41c66e111p1 exists on unknown device 
8:145
  Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev 
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 
65:225
  Duplicate of PV i8z5fn-iYa0-ElZg-nL8h-Abul-2HKU-Oh3F8w dev 
/dev/mapper/360060160ec7323000c61c43b3a66e111p1 exists on unknown device 
8:193
  Duplicate of PV f1IMBI-f2kl-vr1b-BbO1-Ydfw-n5Ok-p1zPDd dev