Hi,
I have a very weird issue with the last version of oVirt 3.4.1, I've upgraded
because I already had this issue.
It came when I wanted to re-create a new export domain on my first hypervisor
larger than the first one.
After this point, the datacenter came in a non responsive state, so I've
upgraded oVirt to 3.4.1, but the issue is still there.
I join the tail of the engine.log and the vdsm.log of one of the two
hypervisors.
Tell me if you want more logs.
Thank you in advance for your help :)
Engine.log :
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not
SPM: ()]]
2014-05-14 12:26:34,195 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) HostName = unc-srv-hyp2
2014-05-14 12:26:34,199 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-82) Command
HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp2, HostId =
2d8722cc-5041-427d-964d-8980f40c5aa6) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:34,248 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp2,
spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:34,269 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: aaf1e60
2014-05-14 12:26:35,998 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log
id: aaf1e60
2014-05-14 12:26:35,998 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due
to: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:36,082 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) Irs placed on server
2d8722cc-5041-427d-964d-8980f40c5aa6 failed. Proceed Failover
2014-05-14 12:26:36,126 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) hostFromVds::selectedVds - unc-srv-hyp1,
spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:36,147 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) START, ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp1, HostId = 4987bc7d-82b2-444f-ad6c-8289da5e4fb9, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 1, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 3b569aaf
2014-05-14 12:26:37,177 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-82) FINISH, ConnectStoragePoolVDSCommand, log
id: 3b569aaf
2014-05-14 12:26:37,182 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-82) IrsBroker::Failed::GetStoragePoolInfoVDS due
to: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
2014-05-14 12:26:47,305 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand
return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not
SPM: ()]]
2014-05-14 12:26:47,321 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) HostName = unc-srv-hyp1
2014-05-14 12:26:47,325 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-13) Command
HSMGetAllTasksStatusesVDSCommand(HostName = unc-srv-hyp1, HostId =
4987bc7d-82b2-444f-ad6c-8289da5e4fb9) execution failed. Exception:
IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM: ()
2014-05-14 12:26:47,375 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) hostFromVds::selectedVds - unc-srv-hyp2,
spmStatus Unknown_Pool, storage pool UNICEF
2014-05-14 12:26:47,396 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) START, ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp2, HostId = 2d8722cc-5041-427d-964d-8980f40c5aa6, storagePoolId =
4eeccf64-715d-4ebe-a44c-eeca94a09a05, vds_spm_id = 2, masterDomainId =
9bf9ed01-43ab-4372-acdd-3500645f3bd0, masterVersion = 1), log id: 2fce851e
2014-05-14 12:26:49,042 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-13) FINISH, ConnectStoragePoolVDSCommand, log
id: 2fce851e
2014-05-14 12:26:49,042 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-13) IrsBroker::Failed::GetStoragePoolInfoVDS due
to: IRSNonOperationalException: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Could not connect host to Data Center(Storage issue)
Vdsm.log :
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1009, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1080, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 638, in connect
self.__createMailboxMonitor()
File "/usr/share/vdsm/storage/sp.py", line 459, in __createMailboxMonitor
self.masterDomain.supportsMailbox):
File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
return getattr(self.getRealDomain(), attrName)
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14
12:27:52,895::task::885::TaskManager.Task::(_run)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._run:
2096f0da-3c8a-4837-9d0d-2e4b8abed77a ('4eeccf64-715d-4ebe-a44c-eeca94a09a05',
1, '9bf9ed01-43ab-4372-acdd-3500645f3bd0', 1, None) {} failed - stopping task
Thread-4535::DEBUG::2014-05-14
12:27:52,896::task::1211::TaskManager.Task::(stop)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::stopping in state preparing (force
False)
Thread-4535::DEBUG::2014-05-14
12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 1 aborting True
Thread-4535::INFO::2014-05-14
12:27:52,896::task::1168::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::aborting: Task is aborted:
u"BlockStorageDomain instance has no attribute 'supportsMailbox'" - code 100
Thread-4535::DEBUG::2014-05-14
12:27:52,896::task::1173::TaskManager.Task::(prepare)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Prepare: aborted:
BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-4535::DEBUG::2014-05-14
12:27:52,896::task::990::TaskManager.Task::(_decref)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::ref 0 aborting True
Thread-4535::DEBUG::2014-05-14
12:27:52,896::task::925::TaskManager.Task::(_doAbort)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::Task._doAbort: force False
Thread-4535::DEBUG::2014-05-14
12:27:52,896::resourceManager::977::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-4535::DEBUG::2014-05-14
12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state preparing ->
state aborting
Thread-4535::DEBUG::2014-05-14
12:27:52,897::task::550::TaskManager.Task::(__state_aborting)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::_aborting: recover policy none
Thread-4535::DEBUG::2014-05-14
12:27:52,897::task::595::TaskManager.Task::(_updateState)
Task=`2096f0da-3c8a-4837-9d0d-2e4b8abed77a`::moving from state aborting ->
state failed
Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::940::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-4535::DEBUG::2014-05-14
12:27:52,897::resourceManager::977::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-4535::ERROR::2014-05-14
12:27:52,897::dispatcher::68::Storage.Dispatcher.Protect::(run)
BlockStorageDomain instance has no attribute 'supportsMailbox'
Traceback (most recent call last):
File "/usr/share/vdsm/storage/dispatcher.py", line 60, in run
result = ctask.prepare(self.func, *args, **kwargs)
File "/usr/share/vdsm/storage/task.py", line 103, in wrapper
return m(self, *a, **kw)
File "/usr/share/vdsm/storage/task.py", line 1176, in prepare
raise self.error
AttributeError: BlockStorageDomain instance has no attribute 'supportsMailbox'
Thread-16::DEBUG::2014-05-14 12:27:55,639::lvm::295::Storage.Misc.excCmd::(cmd)
'/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
\'a|/dev/mapper/3690b11c0005592a9000004855344bf5c|\', \'r|.*|\' ] } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup {
retain_min = 50 retain_days = 0 } " 9bf9ed01-43ab-4372-acdd-3500645f3bd0' (cwd
None)
Thread-16::DEBUG::2014-05-14 12:27:55,765::lvm::295::Storage.Misc.excCmd::(cmd)
SUCCESS: <err> = ''; <rc> = 0
Thread-16::DEBUG::2014-05-14
12:27:55,768::blockSD::605::Storage.Misc.excCmd::(getReadDelay) '/bin/dd
iflag=direct if=/dev/9bf9ed01-43ab-4372-acdd-3500645f3bd0/metadata bs=4096
count=1' (cwd None)
Thread-16::DEBUG::2014-05-14
12:27:55,773::blockSD::605::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err>
= '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0003455 s,
11.9 MB/s\n'; <rc> = 0
Thread-4535::DEBUG::2014-05-14
12:27:57,995::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state init -> state
preparing
Thread-4535::INFO::2014-05-14 12:27:57,995::logUtils::44::dispatcher::(wrapper)
Run and protect: repoStats(options=None)
Thread-4535::INFO::2014-05-14 12:27:57,996::logUtils::47::dispatcher::(wrapper)
Run and protect: repoStats, Return response:
{'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired':
True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14
12:27:57,996::task::1185::TaskManager.Task::(prepare)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::finished:
{'9bf9ed01-43ab-4372-acdd-3500645f3bd0': {'code': 0, 'version': 3, 'acquired':
True, 'delay': '0.0003455', 'lastCheck': '2.2', 'valid': True}}
Thread-4535::DEBUG::2014-05-14
12:27:57,996::task::595::TaskManager.Task::(_updateState)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::moving from state preparing ->
state finished
Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::940::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
Thread-4535::DEBUG::2014-05-14
12:27:57,996::resourceManager::977::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-4535::DEBUG::2014-05-14
12:27:57,996::task::990::TaskManager.Task::(_decref)
Task=`89ef3bd0-f222-470b-b678-6228d5254831`::ref 0 aborting False
________________________________
Alain VONDRA
Charg? d'exploitation des Syst?mes d'Information
Direction Administrative et Financi?re
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006 PARIS
www.unicef.fr<http://www.unicef.fr/>
[cid:signature_email_consultation223f6c]<http://www.jeparledemesdroits.fr>
________________________________
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
