Thanks Shani - yes we plan to upgrade to 4.4 in the future, but we're on
4.3 right now due to only running CentOS 7 at the moment.
I was able to clear the job from the SPM:
[root@daccs01 ~]# vdsm-client Host getAllTasksStatuses
{
"5fa9edf0-56c3-40e4-9327-47bf7764d28d": {
"message": "1 jobs completed successfully",
"code": 0,
"taskID": "5fa9edf0-56c3-40e4-9327-47bf7764d28d",
"taskResult": "success",
"taskState": "finished"
}
}
[root@daccs01 ~]# vdsm-client Task clear
taskID=5fa9edf0-56c3-40e4-9327-47bf7764d28d
true
[root@daccs01 ~]# vdsm-client Host getAllTasksStatuses
{}
And confirm there were no async_tasks:
engine=# select * from async_tasks;
task_id | action_type | status | result | step_id | command_id |
started_at | storage_pool_id | task_type | vdsm_task_id |
root_command_id | user_id
---------+-------------+--------+--------+---------+------------+------------+-----------------+-----------+--------------+-----------------+---------
(0 rows)
However, when putting the vm-storage-ssd domain into maintenance mode,
it failed again:
Here are some the logs entries - anything else I can look at?
2021-07-29 10:30:37,848-07 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802),
VDSM compute7.pcic.uvic.ca command ConnectStoragePoolVDS failed: Wrong
Master domain or its version: u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8,
pool=f72ec125-69a1-4c1b-a5e1-313fcb70b6ff'
2021-07-29 10:30:37,848-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] Command
'org.ovirt.engine.core.vdsbroker.vd
sbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn
[status=Status [code=324, message=Wrong Master domain or its version:
u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8, pool=f72ec125-69a1-4c1b-a5e1-
313fcb70b6ff']]'
...
2021-07-29 10:30:37,848-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] HostName =
compute7.pcic.uvic.ca
2021-07-29 10:30:37,849-07 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] Command
'ConnectStoragePoolVDSCommand(HostN
ame = compute7.pcic.uvic.ca,
ConnectStoragePoolVDSCommandParameters:{hostId='51769733-0cf6-4270-8288-ec96474b7609',
vdsId='51769733-0cf6-4270-8288-ec96474b7609',
storagePoolId='f72ec125-69a1-4c1b-a5e1-313fcb70b6
ff', masterVersion='288'})' execution failed: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or
its version: u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8, pool=f72ec125-69a1
-4c1b-a5e1-313fcb70b6ff'
...
2021-07-29 10:30:37,849-07 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47]
IrsBroker::Failed::DeactivateStorageDomainVDS: IRSGener
icException: IRSErrorException: IRSNoMasterDomainException: Wrong Master
domain or its version: u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8,
pool=f72ec125-69a1-4c1b-a5e1-313fcb70b6ff'
2021-07-29 10:30:37,855-07 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] FINISH,
DeactivateStorageDomainVDSComm
and, return: , log id: 1c215ca4
2021-07-29 10:30:37,855-07 ERROR
[org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47]
DeactivateStorageDomainVDS failed 'a5a83df
1-47e2-4927-9add-079199ca7ef8':
org.ovirt.engine.core.common.errors.EngineException: EngineException:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException:
IRSGenericException: IRSErrorException:
IRSNoMasterDomainException: Wrong Master domain or its version:
u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8,
pool=f72ec125-69a1-4c1b-a5e1-313fcb70b6ff' (Failed with error
StoragePoolWrongMaster and code 324)
at
org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:118)
[bll.jar:]
at
org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33)
[bll.jar:]
at
org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2112)
[bll.jar:]
at
org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand.dectivateStorageDomain(DeactivateStorageDomainCommand.java:340)
[bll.jar:]
...
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_292]
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250)
[javax.enterprise.concurrent-1.0.jar:]
Caused by:
org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException:
IRSGenericException: IRSErrorException: IRSNoMasterDomainException:
Wrong Master domain or its version:
u'SD=a5a83df1-47e2-4927-9add-079199ca7ef8,
pool=f72ec125-69a1-4c1b-a5e1-313fcb70b6ff'
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:50)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedConnectProxyReturnValue(ConnectStoragePoolVDSCommand.java:48)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand.proceedProxyReturnValue(ConnectStoragePoolVDSCommand.java:36)
[vdsbroker.jar:]
...
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
[weld-core-impl-3.1.1.Final.jar:3.1.1.Final]
at
org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
[weld-core-impl-3.1.1.Final.jar:3.1.1.Final]
at
org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown
Source) [vdsbroker.jar:]
... 94 more
2021-07-29 10:30:37,861-07 ERROR
[org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] Failed to
deactivate storage domain 'a5a83df1-47e2-4927-9add-079199ca7ef8'
2021-07-29 10:30:37,868-07 INFO
[org.ovirt.engine.core.bll.CommandCompensator]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] Command
[id=c63199f8-a720-4053-8e5c-92c8d21e0ce2]: Compensating
CHANGED_STATUS_ONLY of
org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap;
snapshot:
EntityStatusSnapshot:{id='StoragePoolIsoMapId:{storagePoolId='f72ec125-69a1-4c1b-a5e1-313fcb70b6ff',
storageId='a5a83df1-47e2-4927-9add-079199ca7ef8'}', status='Unknown'}.
2021-07-29 10:30:37,882-07 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-25) [35c5b47] EVENT_ID:
USER_DEACTIVATE_STORAGE_DOMAIN_FAILED(969), Failed to deactivate Storage
Domain vm-storage-ssd (Data Center EDC2).
2021-07-29 10:30:37,884-07 WARN
[org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand]
(EE-ManagedThreadFactory-engine-Thread-25) [60d33d] Validation of action
'ReconstructMasterDomain' failed for user SYSTEM. Reasons:
VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status
Locked
2021-07-29 10:30:37,888-07 INFO
[org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor]
(EE-ManagedThreadFactory-engine-Thread-48) [35c5b47] Finished
reconstruct for pool 'f72ec125-69a1-4c1b-a5e1-313fcb70b6ff'. Clearing
event queue
2021-07-29 10:30:37,899-07 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'}
Thanks,
-Matthew
On 7/29/21 2:52 AM, Shani Leviim wrote:
> Notice: This message was sent from outside the University of Victoria
> email system. Please be cautious with links and sensitive information.
>
> Hi Matthew,
> Actually, your description is related to 2 features available for
> ovirt 4.4.5 <https://www.ovirt.org/release/4.4.5/>
> 1. The ability to switch the master storage domain while domains are
> up and running [1]
> 2. Clearing the finished tasks from REST API [2] and UI [3].
>
> We recommend you upgrade your engine to enjoy those features.
>
> In the meanwhile, as you've described, moving the Master role from one
> storage to the other is available using putting the domain into
> maintenance.
> In order to clear the finished tasks from SPM:
> vdsm-client Host getAllTasksStatuses
>
> It should be something like that:
> {
> "1dc4d885-577a-4b6a-b01f-e682602a907c": {
> "code": 0,
> "message": "1 jobs completed successfully",
> "taskID": "1dc4d885-577a-4b6a-b01f-e682602a907c",
> "taskResult": "success",
> "taskState": "finished"
> }
> }
>
> Then clear that tasks:
> vdsm-client Task clear taskID=12345
> Once it gets cleared, the reconstruction can be finished.
>
> To verify there are no more finished async tasks, you can run this SQL
> query on the engine:
> engine=# select * from async_tasks WHERE storage_pool_id = '123';
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1910022
> <https://bugzilla.redhat.com/show_bug.cgi?id=1910022>
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1627997
> <https://bugzilla.redhat.com/show_bug.cgi?id=1627997>
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1910302
> <https://bugzilla.redhat.com/show_bug.cgi?id=1910302>
>
> *Regards,
> *
> *Shani Leviim
> *
>
>
> On Thu, Jul 29, 2021 at 8:33 AM Matthew Benstead <matth...@uvic.ca
> <mailto:matth...@uvic.ca>> wrote:
>
> Hello,
>
> I'm trying to decommission the old master storage domain in ovirt,
> and replace it with a new one. All of the VMs have been migrated
> off of the old master, and everything has been running on the new
> storage domain for a couple months. But when I try to put the old
> domain into maintenance mode I get an error.
>
> Old Master: vm-storage-ssd
> New Domain: vm-storage-ssd2
>
> The error is:
>
> Failed to Reconstruct Master Domain for Data Center EDC2
>
> As well as:
>
> Sync Error on Master Domain between Host daccs01 and oVirt Engine.
> Domain: vm-storage-ssd is marked as Master in oVirt Engine
> database but not on the Storage side. Please consult with Support
> on how to fix this issue.
>
> 2021-07-28 11:41:34,870-07 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
> (EE-ManagedThreadFactory-engine-Thread-23) [] Master domain
> version is not in sync between DB and VDSM. Domain vm-storage-ssd
> marked as master, but the version in DB: 283 and in VDSM: 280
>
> And:
>
> Not stopping SPM on vds daccs01, pool id
> f72ec125-69a1-4c1b-a5e1-313fcb70b6ff as there are uncleared tasks
> Task '5fa9edf0-56c3-40e4-9327-47bf7764d28d', status 'finished'
>
>
> After a couple minutes all the domains are marked as active again
> and things continue, but vm-storage-ssd is still listed as the
> master domain. Any thoughts?
>
> This is on 4.3.10.4-1.el7 on CentOS 7.
>
> engine=# SELECT storage_name, storage_pool_id, storage, status
> FROM storage_pool_with_storage_domain ORDER BY storage_name;
> storage_name | storage_pool_id
> | storage | status
>
> -----------------------+--------------------------------------+----------------------------------------+--------
> compute1-iscsi-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> yvUESE-yWUv-VIWL-qX90-aAq7-gK0I-EqppRL | 1
> compute7-iscsi-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> 8ekHdv-u0RJ-B0FO-LUUK-wDWs-iaxb-sh3W3J | 1
> export-domain-storage | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> d3932528-6844-481a-bfed-542872ace9e5 | 1
> iso-storage | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> f800b7a6-6a0c-4560-8476-2f294412d87d | 1
> vm-storage-7200rpm | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> a0bff472-1348-4302-a5c7-f1177efa45a9 | 1
> vm-storage-ssd | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> 95acd9a4-a6fb-4208-80dd-1c53d6aacad0 | 1
> vm-storage-ssd2 | f72ec125-69a1-4c1b-a5e1-313fcb70b6ff |
> 829d0600-c3f7-4dae-a749-d7f05c6a6ca4 | 1
> (7 rows)
>
> Thanks,
> -Matthew
> --
> _______________________________________________
> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org>
> To unsubscribe send an email to users-le...@ovirt.org
> <mailto:users-le...@ovirt.org>
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> <https://www.ovirt.org/privacy-policy.html>
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> <https://www.ovirt.org/community/about/community-guidelines/>
> List Archives:
>
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/OXOXW6B2NWXOUGZV3OKO4OMDXVDJSQLZ/
>
> <https://lists.ovirt.org/archives/list/users@ovirt.org/message/OXOXW6B2NWXOUGZV3OKO4OMDXVDJSQLZ/>
>
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FHC5V7S36PQQXR5FEJA56ZR6LZMQX2UC/
