On 29-04-2015 11:56, Liron Aravot wrote:
----- Original Message -----
From: "Christopher Pereira" <[email protected]>
To: "Nir Soffer" <[email protected]>
Cc: [email protected], "Liron Aravot" <[email protected]>
Sent: Wednesday, April 29, 2015 6:14:32 AM
Subject: Re: [ovirt-devel] "Please activate the master Storage Domain first"
On 28-04-2015 18:14, Nir Soffer wrote:
The DC storage master domain is on a (unrecoverable) storage on a remote
dead
host.
Engine is automatically setting another storage as the "Data (Master)".
Seconds later, the unrecoverable storage is marked as "Data (Master)"
again.
There is no way to start the Datacenter.
Both storages are gluster. The old (unrecoverable) one worked fine as a
master.
This may be related to this bug:
https://bugzilla.redhat.com/1183977.
Ok. I added a comment and explained more in detail the issue on BZ.
[...]
Hi Chris,
Can you please attach the engine/vdsm logs from the time the issue occurred?
thanks.
Hi, relevant VDSM logs were published here:
https://bugzilla.redhat.com/show_bug.cgi?id=1183977
And engine says "Failed in 'ReconstructMasterVDS' method" because of
"Cannot acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error'))":
2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654,
mMessage=Not SPM: ()]]
'
2015-04-28 16:34:01,915 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H4
2015-04-28 16:34:01,915 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = H4,
VdsIdVDSCommandParametersBase:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26'})' execution failed:
IRSGenericException: IRSErrorException: IRSNonOperationalException:
Not SPM: ()
2015-04-28 16:34:01,951 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
(DefaultQuartzScheduler_Worker-56) [] hostFromVds::selectedVds -
'H6', spmStatus 'Unknown_Pool', storage pool 'Default', storage pool
version '3.6'
2015-04-28 16:34:01,953 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] START,
ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625), log id:
15842e4f
2015-04-28 16:34:02,035 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=304, mMessage=Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f']]'
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] HostName = H6
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] Command
'ConnectStoragePoolVDSCommand(HostName = H6,
ConnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsId =
75e32383-90dd-42c5-9bc9-73b6e5333fe1, storagePoolId =
00000001-0001-0001-0001-00000000007e, masterVersion = 2625)'
execution failed: IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Cannot find master domain:
u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
2015-04-28 16:34:02,036 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(DefaultQuartzScheduler_Worker-56) [] FINISH,
ConnectStoragePoolVDSCommand, log id: 15842e4f
2015-04-28 16:34:02,036 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(DefaultQuartzScheduler_Worker-56) []
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master
domain: u'spUUID=00000001-0001-0001-0001-00000000007e,
msdUUID=3233144b-7be1-445f-9ea6-6aebbacbb93f'
2015-04-28 16:34:02,137 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Running command:
ReconstructMasterDomainCommand internal: true. Entities affected :
ID: 3233144b-7be1-445f-9ea6-6aebbacbb93f Type: Storage
2015-04-28 16:34:02,235 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
DisconnectStoragePoolVDSCommand(HostName = H6,
DisconnectStoragePoolVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, storagePoolId =
00000001-0001-0001-0001-00000000007e, vds_spm_id = 3), log id: 782490bb
2015-04-28 16:34:02,239 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
DisconnectStoragePoolVDSCommand, log id: 782490bb
2015-04-28 16:34:02,240 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] START,
ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
status: Inactive };]), log id: 626fc611
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Failed in
'ReconstructMasterVDS' method
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand'
return value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=661, mMessage=Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error'))]]'
2015-04-28 16:34:03,318 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] HostName = H6
2015-04-28 16:34:03,318 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'ReconstructMasterVDSCommand(HostName = H6,
ReconstructMasterVDSCommandParameters:{runAsync='true',
hostId='75e32383-90dd-42c5-9bc9-73b6e5333fe1'}, vdsSpmId = 3,
storagePoolId = 00000001-0001-0001-0001-00000000007e,
storagePoolName = Default, masterDomainId =
ba7be27f-aee5-4436-ae9a-0764f551f9a7, masterVersion = 2626,
domainsList = [{ domainId: ad08adbc-b3a7-4198-a1cb-4ad70c724454,
status: Unknown };{ domainId: 15674257-e14a-4693-9b95-cc4bb92aa765,
status: Unknown };{ domainId: ba7be27f-aee5-4436-ae9a-0764f551f9a7,
status: Unknown };{ domainId: 3233144b-7be1-445f-9ea6-6aebbacbb93f,
status: Inactive };])' execution failed: VDSGenericException:
VDSErrorException: Failed to ReconstructMasterVDS, error = Cannot
acquire host id: (u'ba7be27f-aee5-4436-ae9a-0764f551f9a7',
SanlockException(5, 'Sanlock lockspace add failure', 'Input/output
error')), code = 661
2015-04-28 16:34:03,319 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ReconstructMasterVDSCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] FINISH,
ReconstructMasterVDSCommand, log id: 626fc611
2015-04-28 16:34:03,319 ERROR
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
'org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand'
failed: VdcBLLException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to
ReconstructMasterVDS, error = Cannot acquire host id:
(u'ba7be27f-aee5-4436-ae9a-0764f551f9a7', SanlockException(5,
'Sanlock lockspace add failure', 'Input/output error')), code = 661
(Failed with error AcquireHostIdFailure and code 661)
2015-04-28 16:34:03,320 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot: EntityStatusSnapshot:{id='storagePoolId =
00000001-0001-0001-0001-00000000007e, storageId =
3233144b-7be1-445f-9ea6-6aebbacbb93f', status='Unknown'}.
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks-ssd,
3233144b-7be1-445f-9ea6-6aebbacbb93f].
2015-04-28 16:34:03,321 INFO
[org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand]
(org.ovirt.thread.pool-8-thread-25) [1339ee5] Command
[id=449f2035-96bb-4b70-867c-05a7179dc75b]: Compensating
UPDATED_ONLY_ENTITY of
org.ovirt.engine.core.common.businessentities.StorageDomainStatic;
snapshot: StorageDomainStatic[vdisks,
ba7be27f-aee5-4436-ae9a-0764f551f9a7].
2015-04-28 16:34:03,370 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(org.ovirt.thread.pool-8-thread-6) [] Finished reconstruct for pool
'00000001-0001-0001-0001-00000000007e'. Clearing event queue
2015-04-28 16:34:04,154 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerObjectsBuilder]
(DefaultQuartzScheduler_Worker-65) [c1df557] Can't set graphics data
from XML.
2015-04-28 16:34:04,155 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] START,
DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''}), log id: 238de0d4
2015-04-28 16:34:04,158 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Unexpected return
value: StatusForXmlRpc [mCode=-32603,
mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Failed in 'DestroyVDS'
method
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return
value 'StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc
[mCode=-32603, mMessage=u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe']]'
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] HostName = H4
2015-04-28 16:34:04,158 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] Command
'DestroyVDSCommand(HostName = H4,
DestroyVmVDSCommandParameters:{runAsync='true',
hostId='0146bec9-9e8a-4611-a2ca-e4f3c8ccdb26',
vmId='6ec9c0a0-2879-4bfe-9a79-92471881ebfe', force='false',
secondsToWait='0', gracefully='false', reason=''})' execution
failed: VDSGenericException: VDSErrorException: Failed to
DestroyVDS, error = u'6ec9c0a0-2879-4bfe-9a79-92471881ebfe', code =
-32603
2015-04-28 16:34:04,158 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-65) [c1df557] FINISH,
DestroyVDSCommand, log id: 238de0d4
DC is now up, because I attached the old MSD, but I could try
reproducing the failing command if you want.
How do I specify the dataDict in the vdsClient command?
I guess it should be something like:
vdsClient -s 0 reconstructMaster
'00000001-0001-0001-0001-00000000007e' 'Default'
'ba7be27f-aee5-4436-ae9a-0764f551f9a7' '[{ domainId:
ad08adbc-b3a7-4198-a1cb-4ad70c724454, status: Unknown };{ domainId:
15674257-e14a-4693-9b95-cc4bb92aa765, status: Unknown };{ domainId:
ba7be27f-aee5-4436-ae9a-0764f551f9a7, status: Unknown };{ domainId:
3233144b-7be1-445f-9ea6-6aebbacbb93f, status: Inactive };]'
Maybe related with sanlock giving me "add_lockspace fail result -5" all
the time.
Best regards,
Chris
_______________________________________________
Devel mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/devel
