Re: [ovirt-users] Does my Storage Domain crashed or is this iSCSI LUN's a problem?
On 26/01/15 13:05 +0100, shimano wrote: Hi guys, I'm trying to run one of my storage domains, which experienced failure. Unfortunately, I meet a very nasty error (Storage domain does not exist). Could someone tell me, how to try to restore this domain? Could you try moving the host to Maintenance mode and then Activate it again please. I've encountered situations where vdsm restarts and engine does not reconnect storage until an Activate action happens. Let's see if this is your issue. P.S. It's an oVirt 3.4.2-1.el6 ** /var/log/messages: Jan 26 12:48:49 node002 vdsm TaskManager.Task ERROR Task=`10d02993-b585-448f-9a50-bd3e8cda7082`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 873, in _run#012return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2959, in getVGInfo#012return dict(info=self.__getVGsInfo([vgUUID])[0])#012 File "/usr/share/vdsm/storage/hsm.py", line 2892, in __getVGsInfo#012vgList = [lvm.getVGbyUUID(vgUUID) for vgUUID in vgUUIDs]#012 File "/usr/share/vdsm/storage/lvm.py", line 894, in getVGbyUUID#012raise se.VolumeGroupDoesNotExist("vg_uuid: %s" % vgUUID)#012VolumeGroupDoesNotExist: Volume Group does not exist: ('vg_uuid: gyaCWf-6VKi-lI9W-JT6H-IZdy-rIsB-hTvZ4O',) Jan 26 12:48:49 node002 kernel: device-mapper: table: 253:26: multipath: error getting device Jan 26 12:48:49 node002 kernel: device-mapper: ioctl: error adding target to table ** /var/log/vdsm.log: Thread-22::ERROR::2015-01-26 12:43:03,376::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::ERROR::2015-01-26 12:43:03,377::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::DEBUG::2015-01-26 12:43:03,377::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-22::DEBUG::2015-01-26 12:43:03,378::lvm::296::Storage.Misc.excCmd::(cmd) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/mpathb|/dev/mapper/mpathc|/dev/mapper/mpathd|/dev/mapper/mpathe|/dev/mapper/mpathf|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name db52e9cb-7306-43fd-aff3-20831bc2bcaf' (cwd None) Thread-22::DEBUG::2015-01-26 12:43:03,462::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: = ' /dev/mapper/mpathc: Checksum error\n /dev/mapper/mpathc: Checksum error\n Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found\n Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf\n'; = 5 Thread-22::WARNING::2015-01-26 12:43:03,466::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' /dev/mapper/mpathc: Checksum error', ' /dev/mapper/mpathc: Checksum error', ' Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found', ' Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf'] Thread-22::DEBUG::2015-01-26 12:43:03,466::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-22::ERROR::2015-01-26 12:43:03,477::sdc::143::Storage.StorageDomainCache::(_findDomain) domain db52e9cb-7306-43fd-aff3-20831bc2bcaf not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'db52e9cb-7306-43fd-aff3-20831bc2bcaf',) Thread-22::ERROR::2015-01-26 12:43:03,478::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain db52e9cb-7306-43fd-aff3-20831bc2bcaf monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/us
[ovirt-users] Does my Storage Domain crashed or is this iSCSI LUN's a problem?
Hi guys, I'm trying to run one of my storage domains, which experienced failure. Unfortunately, I meet a very nasty error (Storage domain does not exist). Could someone tell me, how to try to restore this domain? P.S. It's an oVirt 3.4.2-1.el6 ** /var/log/messages: Jan 26 12:48:49 node002 vdsm TaskManager.Task ERROR Task=`10d02993-b585-448f-9a50-bd3e8cda7082`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 873, in _run#012return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2959, in getVGInfo#012return dict(info=self.__getVGsInfo([vgUUID])[0])#012 File "/usr/share/vdsm/storage/hsm.py", line 2892, in __getVGsInfo#012vgList = [lvm.getVGbyUUID(vgUUID) for vgUUID in vgUUIDs]#012 File "/usr/share/vdsm/storage/lvm.py", line 894, in getVGbyUUID#012raise se.VolumeGroupDoesNotExist("vg_uuid: %s" % vgUUID)#012VolumeGroupDoesNotExist: Volume Group does not exist: ('vg_uuid: gyaCWf-6VKi-lI9W-JT6H-IZdy-rIsB-hTvZ4O',) Jan 26 12:48:49 node002 kernel: device-mapper: table: 253:26: multipath: error getting device Jan 26 12:48:49 node002 kernel: device-mapper: ioctl: error adding target to table ** /var/log/vdsm.log: Thread-22::ERROR::2015-01-26 12:43:03,376::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::ERROR::2015-01-26 12:43:03,377::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::DEBUG::2015-01-26 12:43:03,377::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-22::DEBUG::2015-01-26 12:43:03,378::lvm::296::Storage.Misc.excCmd::(cmd) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/mpathb|/dev/mapper/mpathc|/dev/mapper/mpathd|/dev/mapper/mpathe|/dev/mapper/mpathf|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name db52e9cb-7306-43fd-aff3-20831bc2bcaf' (cwd None) Thread-22::DEBUG::2015-01-26 12:43:03,462::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: = ' /dev/mapper/mpathc: Checksum error\n /dev/mapper/mpathc: Checksum error\n Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found\n Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf\n'; = 5 Thread-22::WARNING::2015-01-26 12:43:03,466::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' /dev/mapper/mpathc: Checksum error', ' /dev/mapper/mpathc: Checksum error', ' Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found', ' Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf'] Thread-22::DEBUG::2015-01-26 12:43:03,466::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-22::ERROR::2015-01-26 12:43:03,477::sdc::143::Storage.StorageDomainCache::(_findDomain) domain db52e9cb-7306-43fd-aff3-20831bc2bcaf not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'db52e9cb-7306-43fd-aff3-20831bc2bcaf',) Thread-22::ERROR::2015-01-26 12:43:03,478::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain db52e9cb-7306-43fd-aff3-20831bc2bcaf monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'db52e9cb-7306-43fd-aff3-20831bc2bcaf',) Thread-13::DEBUG::2015-01-26 12:43:05,102::task::595::Tas
[ovirt-users] Does my Storage Domain crashed or is this iSCSI LUN's a problem?
Hi guys, I'm trying to run one of my storage domains, which experienced failure. Unfortunately, I meet a very nasty error (Storage domain does not exist). Could someone tell me, how to try to restore this domain? P.S. It's an oVirt 3.4.2-1.el6 ** /var/log/messages: Jan 26 12:48:49 node002 vdsm TaskManager.Task ERROR Task=`10d02993-b585-448f-9a50-bd3e8cda7082`::Unexpected error#012Traceback (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line 873, in _run#012return fn(*args, **kargs)#012 File "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012res = f(*args, **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2959, in getVGInfo#012return dict(info=self.__getVGsInfo([vgUUID])[0])#012 File "/usr/share/vdsm/storage/hsm.py", line 2892, in __getVGsInfo#012vgList = [lvm.getVGbyUUID(vgUUID) for vgUUID in vgUUIDs]#012 File "/usr/share/vdsm/storage/lvm.py", line 894, in getVGbyUUID#012raise se.VolumeGroupDoesNotExist("vg_uuid: %s" % vgUUID)#012VolumeGroupDoesNotExist: Volume Group does not exist: ('vg_uuid: gyaCWf-6VKi-lI9W-JT6H-IZdy-rIsB-hTvZ4O',) Jan 26 12:48:49 node002 kernel: device-mapper: table: 253:26: multipath: error getting device Jan 26 12:48:49 node002 kernel: device-mapper: ioctl: error adding target to table ** /var/log/vdsm.log: Thread-22::ERROR::2015-01-26 12:43:03,376::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::ERROR::2015-01-26 12:43:03,377::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain db52e9cb-7306-43fd-aff3-20831bc2bcaf Thread-22::DEBUG::2015-01-26 12:43:03,377::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-22::DEBUG::2015-01-26 12:43:03,378::lvm::296::Storage.Misc.excCmd::(cmd) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ \'a|/dev/mapper/mpathb|/dev/mapper/mpathc|/dev/mapper/mpathd|/dev/mapper/mpathe|/dev/mapper/mpathf|\', \'r|.*|\' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name db52e9cb-7306-43fd-aff3-20831bc2bcaf' (cwd None) Thread-22::DEBUG::2015-01-26 12:43:03,462::lvm::296::Storage.Misc.excCmd::(cmd) FAILED: = ' /dev/mapper/mpathc: Checksum error\n /dev/mapper/mpathc: Checksum error\n Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found\n Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf\n'; = 5 Thread-22::WARNING::2015-01-26 12:43:03,466::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' /dev/mapper/mpathc: Checksum error', ' /dev/mapper/mpathc: Checksum error', ' Volume group "db52e9cb-7306-43fd-aff3-20831bc2bcaf" not found', ' Skipping volume group db52e9cb-7306-43fd-aff3-20831bc2bcaf'] Thread-22::DEBUG::2015-01-26 12:43:03,466::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-22::ERROR::2015-01-26 12:43:03,477::sdc::143::Storage.StorageDomainCache::(_findDomain) domain db52e9cb-7306-43fd-aff3-20831bc2bcaf not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'db52e9cb-7306-43fd-aff3-20831bc2bcaf',) Thread-22::ERROR::2015-01-26 12:43:03,478::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain db52e9cb-7306-43fd-aff3-20831bc2bcaf monitoring information Traceback (most recent call last): File "/usr/share/vdsm/storage/domainMonitor.py", line 204, in _monitorDomain self.domain = sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'db52e9cb-7306-43fd-aff3-20831bc2bcaf',) Thread-13::DEBUG::2015-01-26 12:43:05,102::task::595::Task