Hello All,
I am having an issue on my test setup, the master storage goes inactive. It
is set up as to use Gluster and the volume is up and running without issues.
Here is a snippet of /var/log/ovirt-engine/engine.log (these are just the
ERRORs):
2014-08-18 01:47:04,173 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Failed in SpmStatusVDS method
2014-08-18 01:47:04,174 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-98) [40e8ae30] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:47:14,286 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Failed in SpmStatusVDS method
2014-08-18 01:47:14,287 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-68) [357af14e] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Failed in SpmStatusVDS method
2014-08-18 01:47:24,400 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-51) [11269c17] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:47:34,514 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Failed in SpmStatusVDS method
2014-08-18 01:47:34,515 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-59) [40622d60] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:47:44,651 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Failed in SpmStatusVDS method
2014-08-18 01:47:44,652 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-97) Command SpmStatusVDSCommand(HostName =
u5.veerotech.net, HostId = 57dac47a-8db9-4d64-86c2-6f9b2a89b3f5,
storagePoolId = 83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed.
Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed
to SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:47:54,765 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Failed in SpmStatusVDS method
2014-08-18 01:47:54,766 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-73) [44b5af81] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:04,881 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Failed in SpmStatusVDS method
2014-08-18 01:48:04,883 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-19) [7bf1ea79] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:14,994 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Failed in SpmStatusVDS method
2014-08-18 01:48:14,995 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-20) [4fed0eac] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:25,117 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Failed in SpmStatusVDS method
2014-08-18 01:48:25,118 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-35) [48ff78af] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:35,231 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Failed in SpmStatusVDS method
2014-08-18 01:48:35,232 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-16) [76fe9857] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:45,344 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Failed in SpmStatusVDS method
2014-08-18 01:48:45,345 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-69) [61356e24] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:48:55,460 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Failed in SpmStatusVDS method
2014-08-18 01:48:55,461 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-28) [38c2326b] Command
SpmStatusVDSCommand(HostName = u5.veerotech.net, HostId =
57dac47a-8db9-4d64-86c2-6f9b2a89b3f5, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
2014-08-18 01:49:05,574 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Failed in SpmStatusVDS method
2014-08-18 01:49:05,575 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand]
(DefaultQuartzScheduler_Worker-87) [3c65466] Command
SpmStatusVDSCommand(HostName = u6.veerotech.net, HostId =
4105003a-d04b-4838-89d9-fcdd8953c98e, storagePoolId =
83ff2f2d-3c8e-4865-8656-c01d867cff06) execution failed. Exception:
VDSErrorException: VDSGenericException: VDSErrorException: Failed to
SpmStatusVDS, error = [Errno 5] Input/output error, code = 100
Here is the vdsm.log on one of the hosts:
[root@u5 ~]# tail -f /var/log/vdsm/vdsm.log
validateFileSystemFeatures(sdUUID, self.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch
ioproc.touch(path, flags, mode)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch
self.timeout)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 5] Input/output error
Thread-36::DEBUG::2014-08-17
19:39:16,037::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:16,054::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230303 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:16,055::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:18,057::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:18,075::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000259145 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:18,075::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-1489::DEBUG::2014-08-17
19:39:18,938::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]
Thread-1489::DEBUG::2014-08-17
19:39:18,939::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state init -> state
preparing
Thread-1489::INFO::2014-08-17
19:39:18,939::logUtils::44::dispatcher::(wrapper) Run and protect:
getSpmStatus(spUUID='83ff2f2d-3c8e-4865-8656-c01d867cff06', options=None)
Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...
Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Message size is 118
Thread-15::DEBUG::2014-08-17
19:39:18,941::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) DEBUG|Queuing
request...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Finding
callback 'glob'...
Thread-15::DEBUG::2014-08-17
19:39:18,942::__init__::225::IOProcess::(_processLogs) (3640) Got request
for method 'glob'
Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|(3640) Building
response
Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) (3640) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:18,944::__init__::225::IOProcess::(_processLogs) DEBUG|Generating
json...
Thread-15::DEBUG::2014-08-17
19:39:18,945::__init__::225::IOProcess::(_processLogs) Sending response
sized 168
Thread-1489::DEBUG::2014-08-17
19:39:18,950::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989
Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) DEBUG|Receiving
request...
Thread-15::DEBUG::2014-08-17
19:39:18,950::__init__::225::IOProcess::(_processLogs) Message size is 147
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:18,951::__init__::225::IOProcess::(_processLogs) (3641) Finding
callback 'touch'...
Thread-15::DEBUG::2014-08-17
19:39:18,952::__init__::225::IOProcess::(_processLogs) (3641) Got request
for method 'touch'
Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Message size is
147DEBUG|(3641) Building response
Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) (3641) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-15::DEBUG::2014-08-17
19:39:18,954::__init__::225::IOProcess::(_processLogs) Sending response
sized 86
Thread-1489::ERROR::2014-08-17
19:39:18,955::task::866::Storage.TaskManager.Task::(_setError)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus
status = self._getSpmStatusInfo(pool)
File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo
(pool.spmRole,) + pool.getSpmStatus()))
File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus
return self._backend.getSpmStatus()
File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus
lVer, spmId = self.masterDomain.inquireClusterLock()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain
return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__
validateFileSystemFeatures(sdUUID, self.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch
ioproc.touch(path, flags, mode)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch
self.timeout)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 5] Input/output error
Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::885::Storage.TaskManager.Task::(_run)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._run:
ada614f8-75c5-4728-9083-340f918a4417
('83ff2f2d-3c8e-4865-8656-c01d867cff06',) {} failed - stopping task
Thread-1489::DEBUG::2014-08-17
19:39:18,955::task::1217::Storage.TaskManager.Task::(stop)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::stopping in state preparing
(force False)
Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 1 aborting True
Thread-1489::INFO::2014-08-17
19:39:18,956::task::1171::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::aborting: Task is aborted:
u'[Errno 5] Input/output error' - code 100
Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::1176::Storage.TaskManager.Task::(prepare)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Prepare: aborted: [Errno 5]
Input/output error
Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::993::Storage.TaskManager.Task::(_decref)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::ref 0 aborting True
Thread-1489::DEBUG::2014-08-17
19:39:18,956::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::Task._doAbort: force False
Thread-1489::DEBUG::2014-08-17
19:39:18,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}
Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state preparing ->
state aborting
Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::_aborting: recover policy none
Thread-1489::DEBUG::2014-08-17
19:39:18,957::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ada614f8-75c5-4728-9083-340f918a4417`::moving from state aborting ->
state failed
Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}
Thread-1489::DEBUG::2014-08-17
19:39:18,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}
Thread-1489::ERROR::2014-08-17
19:39:18,957::dispatcher::79::Storage.Dispatcher::(wrapper) [Errno 5]
Input/output error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper
result = ctask.prepare(func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1179, in prepare
raise self.error
OSError: [Errno 5] Input/output error
Thread-36::DEBUG::2014-08-17
19:39:20,078::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:20,096::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245377 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:20,097::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-1489::DEBUG::2014-08-17
19:39:20,588::BindingXMLRPC::318::vds::(wrapper) client [74.63.225.147]
flowID [fedc88e]
Thread-1489::DEBUG::2014-08-17
19:39:20,589::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state init -> state
preparing
Thread-1489::INFO::2014-08-17
19:39:20,589::logUtils::44::dispatcher::(wrapper) Run and protect:
connectStorageServer(domType=7,
spUUID='00000000-0000-0000-0000-000000000000', conList=[{'port': '',
'connection': 'u6:ralgv01', 'iqn': '', 'user': '', 'tpgt': '1', 'vfs_type':
'glusterfs', 'password': '******', 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}], options=None)
Thread-1489::DEBUG::2014-08-17
19:39:20,594::hsm::2378::Storage.HSM::(__prefetchDomains) glusterDomPath:
glusterSD/*
Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-17
19:39:20,595::__init__::225::IOProcess::(_processLogs) Message size is 118
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:20,596::__init__::225::IOProcess::(_processLogs) (3642) Finding
callback 'glob'...
Thread-15::DEBUG::2014-08-17
19:39:20,597::__init__::225::IOProcess::(_processLogs) (3642) Got request
for method 'glob'
Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUG|(3642) Building response
Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3642) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:20,598::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Generating json...
Thread-15::DEBUG::2014-08-17
19:39:20,599::__init__::225::IOProcess::(_processLogs) Sending response
sized 168
Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2390::Storage.HSM::(__prefetchDomains) Found SD uuids:
(u'2ed32e34-2483-4262-a57b-72bf70f3e989',)
Thread-1489::DEBUG::2014-08-17
19:39:20,600::hsm::2446::Storage.HSM::(connectStorageServer) knownSDs:
{2ed32e34-2483-4262-a57b-72bf70f3e989: storage.glusterSD.findDomain}
Thread-1489::INFO::2014-08-17
19:39:20,600::logUtils::47::dispatcher::(wrapper) Run and protect:
connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id':
'08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-1489::DEBUG::2014-08-17
19:39:20,600::task::1191::Storage.TaskManager.Task::(prepare)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::finished: {'statuslist':
[{'status': 0, 'id': '08128803-0141-46aa-835a-2f9734ef0d3b'}]}
Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::595::Storage.TaskManager.Task::(_updateState)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::moving from state preparing ->
state finished
Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}
Thread-1489::DEBUG::2014-08-17
19:39:20,601::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}
Thread-1489::DEBUG::2014-08-17
19:39:20,601::task::993::Storage.TaskManager.Task::(_decref)
Task=`2316ebfe-4752-4a5a-ae2e-795d6064b732`::ref 0 aborting False
Thread-1489::DEBUG::2014-08-17
19:39:21,020::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state init -> state
preparing
Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::44::dispatcher::(wrapper) Run and protect:
repoStats(options=None)
Thread-1489::INFO::2014-08-17
19:39:21,021::logUtils::47::dispatcher::(wrapper) Run and protect:
repoStats, Return response: {'2ed32e34-2483-4262-a57b-72bf70f3e989':
{'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck':
'6.3', 'valid': False}}
Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::1191::Storage.TaskManager.Task::(prepare)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::finished:
{'2ed32e34-2483-4262-a57b-72bf70f3e989': {'code': 200, 'version': -1,
'acquired': False, 'delay': '0', 'lastCheck': '6.3', 'valid': False}}
Thread-1489::DEBUG::2014-08-17
19:39:21,021::task::595::Storage.TaskManager.Task::(_updateState)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::moving from state preparing ->
state finished
Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseA
ll) Owner.releaseAll requests {} resources {}
Thread-1489::DEBUG::2014-08-17
19:39:21,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAl
l) Owner.cancelAll requests {}
Thread-1489::DEBUG::2014-08-17
19:39:21,022::task::993::Storage.TaskManager.Task::(_decref)
Task=`81e444df-0de1-41fb-808d-84ae2c6669c5`::ref 0 aborting False
Thread-36::DEBUG::2014-08-17
19:39:22,099::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:22,117::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000253763 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:22,117::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:24,120::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:24,139::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000237548 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:24,139::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.sdc.refreshStorage)
Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method
Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::741::Storage.SamplingMethod::(__call__) Trying to enter
sampling method (storage.iscsi.rescan)
Thread-29::DEBUG::2014-08-17
19:39:24,741::misc::743::Storage.SamplingMethod::(__call__) Got in to
sampling method
Thread-29::DEBUG::2014-08-17
19:39:24,741::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this
will take up to 30 seconds
Thread-29::DEBUG::2014-08-17
19:39:24,742::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n
/sbin/iscsiadm -m session -R (cwd None)
Thread-29::DEBUG::2014-08-17
19:39:24,761::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-29::DEBUG::2014-08-17
19:39:24,763::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n
/sbin/multipath (cwd None)
Thread-29::DEBUG::2014-08-17
19:39:24,797::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> =
''; <rc> = 0
Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::489::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' got the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,797::lvm::491::Storage.OperationMutex::(_invalidateAllPvs)
Operation 'lvm invalidate operation' released the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::500::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' got the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::502::Storage.OperationMutex::(_invalidateAllVgs)
Operation 'lvm invalidate operation' released the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::520::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' got the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,798::lvm::522::Storage.OperationMutex::(_invalidateAllLvs)
Operation 'lvm invalidate operation' released the operation mutex
Thread-29::DEBUG::2014-08-17
19:39:24,799::misc::751::Storage.SamplingMethod::(__call__) Returning last
result
Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-17
19:39:24,800::__init__::225::IOProcess::(_processLogs) Message size is 118
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:24,801::__init__::225::IOProcess::(_processLogs) (3643) Finding
callback 'glob'...
Thread-15::DEBUG::2014-08-17
19:39:24,802::__init__::225::IOProcess::(_processLogs) (3643) Got request
for method 'glob'
Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3643) Building response
Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) (3643) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-15::DEBUG::2014-08-17
19:39:24,804::__init__::225::IOProcess::(_processLogs) Sending response
sized 168
Thread-29::DEBUG::2014-08-17
19:39:24,809::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989
Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Message size is 147
Thread-15::DEBUG::2014-08-17
19:39:24,810::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Finding
callback 'touch'...
Thread-15::DEBUG::2014-08-17
19:39:24,811::__init__::225::IOProcess::(_processLogs) (3644) Got request
for method 'touch'
Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|(3644) Building response
Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3644) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-15::DEBUG::2014-08-17
19:39:24,814::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Sending response sized 86
Thread-29::ERROR::2014-08-17
19:39:24,815::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain
return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__
validateFileSystemFeatures(sdUUID, self.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch
ioproc.touch(path, flags, mode)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch
self.timeout)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 5] Input/output error
Thread-36::DEBUG::2014-08-17
19:39:26,142::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:26,160::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000228935 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:26,161::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:28,164::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:28,186::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000187527 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:28,186::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:30,189::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:30,207::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000238901 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:30,207::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:32,209::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:32,227::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000230531 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:32,227::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-36::DEBUG::2014-08-17
19:39:34,230::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
/bin/dd
if=/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/ou
tbox iflag=direct,fullblock bs=512 count=8 skip=8 (cwd None)
Thread-36::DEBUG::2014-08-17
19:39:34,247::storage_mailbox::84::Storage.Misc.excCmd::(_mboxExecCmd)
FAILED: <err> = "/bin/dd: reading
`/rhev/data-center/83ff2f2d-3c8e-4865-8656-c01d867cff06/mastersd/dom_md/outb
ox': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B)
copied, 0.000245347 s, 0.0 kB/s\n"; <rc> = 1
Thread-36::WARNING::2014-08-17
19:39:34,248::storage_mailbox::281::Storage.MailBox.HsmMailMonitor::(_initMa
ilbox) HSM_MailboxMonitor - Could not initialize mailbox, will not accept
requests until init succeeds
Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) (3641) Queuing
responseDEBUGDEBUG|Receiving request...
Thread-15::DEBUG::2014-08-17
19:39:34,817::__init__::225::IOProcess::(_processLogs) Message size is 118
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) Extracting request
information...
Thread-15::DEBUG::2014-08-17
19:39:34,818::__init__::225::IOProcess::(_processLogs) (3645) Finding
callback 'glob'...
Thread-15::DEBUG::2014-08-17
19:39:34,819::__init__::225::IOProcess::(_processLogs) (3645) Got request
for method 'glob'
Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) (3645) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:34,821::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-29::DEBUG::2014-08-17
19:39:34,826::fileSD::150::Storage.StorageDomain::(__init__) Reading domain
in path
/rhev/data-center/mnt/glusterSD/u6:ralgv01/2ed32e34-2483-4262-a57b-72bf70f3e
989
Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Message size is 147
Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Marshaling message...
Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Queuing request...
Thread-15::DEBUG::2014-08-17
19:39:34,827::__init__::225::IOProcess::(_processLogs) Waiting for next
request...
Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) Queuing request in
the thread pool...
Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Finding
callback 'touch'...
Thread-15::DEBUG::2014-08-17
19:39:34,828::__init__::225::IOProcess::(_processLogs) (3646) Got request
for method 'touch'
Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) (3646) Queuing
response
Thread-15::DEBUG::2014-08-17
19:39:34,830::__init__::225::IOProcess::(_processLogs) Generating json...
Thread-29::ERROR::2014-08-17
19:39:34,831::domainMonitor::239::Storage.DomainMonitorThread::(_monitorDoma
in) Error while collecting domain 2ed32e34-2483-4262-a57b-72bf70f3e989
monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 215, in
_monitorDomain
self.domain.selftest()
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/glusterSD.py", line 32, in findDomain
return GlusterStorageDomain(GlusterStorageDomain.findDomainPath(sdUUID))
File "/usr/share/vdsm/storage/fileSD.py", line 157, in __init__
validateFileSystemFeatures(sdUUID, self.mountpoint)
File "/usr/share/vdsm/storage/fileSD.py", line 86, in
validateFileSystemFeatures
oop.getProcessPool(sdUUID).directTouch(testFilePath)
File "/usr/share/vdsm/storage/outOfProcess.py", line 320, in directTouch
ioproc.touch(path, flags, mode)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 460,
in touch
self.timeout)
File "/usr/lib/python2.6/site-packages/ioprocess/__init__.py", line 344,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 5] Input/output error
Any help is getting this sorted is appreciated.
Chris
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
