Can you share the REST API data of the Storage domain and Data Center? Here an example of the URLs, you will need to replace with correct ids.
http://MY-SERVER/ovirt-engine/api/v4/storagedomains/13461356-f6f7-4a58-9897-2fac61ff40af http://MY-SERVER/ovirt-engine/api/v4/datacenters/5a5df553-022d-036d-01e8-000000000071/storagedomains On Wed, May 2, 2018 at 12:53 PM, Callum Smith <cal...@well.ox.ac.uk> wrote: > This is on 4.2.0.2-1, I've linked the main logs to dropbox simply because > they're big, full of noise right now. > https://www.dropbox.com/s/f8q3m5amro2a1b2/engine.log?dl=0 > https://www.dropbox.com/s/uods85jk65halo3/vdsm.log?dl=0 > > Regards, > Callum > > -- > > Callum Smith > Research Computing Core > Wellcome Trust Centre for Human Genetics > University of Oxford > e. cal...@well.ox.ac.uk > > On 2 May 2018, at 10:43, Fred Rolland <froll...@redhat.com> wrote: > > Which version are you using? > Can you provide the whole log? > > For some reason, it looks like the Vdsm thinks that the Storage Domain is > not part of the pool. > > On Wed, May 2, 2018 at 11:20 AM, Callum Smith <cal...@well.ox.ac.uk> > wrote: > >> State is maintenance for the ISOs storage. I've extracted what is >> hopefully the relevant bits of the log. >> >> VDSM.log (SPM) >> >> 2018-05-02 09:16:03,455+0100 INFO (ioprocess communication (179084)) >> [IOProcess] Starting ioprocess (__init__:447) >> 2018-05-02 09:16:03,456+0100 INFO (ioprocess communication (179091)) >> [IOProcess] Starting ioprocess (__init__:447) >> 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [vdsm.api] FINISH >> activateStorageDomain error=Storage domain not in pool: >> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304' from=::ffff:192.168.64.254,58968, >> flow_id=93433989-8e26-48a9-bd3a-2ab95f296c08, >> task_id=7f21f911-348f-45a3-b79c-e3cb11642035 (api:50) >> 2018-05-02 09:16:03,461+0100 ERROR (jsonrpc/0) [storage.TaskManager.Task] >> (Task='7f21f911-348f-45a3-b79c-e3cb11642035') Unexpected error (task:875) >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >> 882, in _run >> return fn(*args, **kargs) >> File "<string>", line 2, in activateStorageDomain >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, >> in method >> ret = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line >> 1256, in activateStorageDomain >> pool.activateSD(sdUUID) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", >> line 79, in wrapper >> return method(self, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1130, >> in activateSD >> self.validateAttachedDomain(dom) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", >> line 79, in wrapper >> return method(self, *args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 557, >> in validateAttachedDomain >> raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID) >> StorageDomainNotInPool: Storage domain not in pool: >> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304' >> 2018-05-02 09:16:03,461+0100 INFO (jsonrpc/0) [storage.TaskManager.Task] >> (Task='7f21f911-348f-45a3-b79c-e3cb11642035') aborting: Task is aborted: >> "Storage domain not in pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304'" - code 353 (task:1181) >> 2018-05-02 09:16:03,462+0100 ERROR (jsonrpc/0) [storage.Dispatcher] >> FINISH activateStorageDomain error=Storage domain not in pool: >> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304' (dispatcher:82) >> >> engine.log >> >> 2018-05-02 09:16:02,326+01 INFO [org.ovirt.engine.core.bll.st >> orage.domain.ActivateStorageDomainCommand] (default task-20) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock Acquired to object >> 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', >> sharedLocks=''}' >> 2018-05-02 09:16:02,376+01 INFO [org.ovirt.engine.core.bll.st >> orage.domain.ActivateStorageDomainCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Running command: >> ActivateStorageDomainCommand internal: false. Entities affected : ID: >> f5914df0-f46c-4cc0-b666-c929aa0225ae Type: StorageAction group >> MANIPULATE_STORAGE_DOMA >> IN with role type ADMIN >> 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st >> orage.domain.ActivateStorageDomainCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Lock freed to object >> 'EngineLock:{exclusiveLocks='[f5914df0-f46c-4cc0-b666-c929aa0225ae=STORAGE]', >> sharedLocks=''}' >> 2018-05-02 09:16:02,385+01 INFO [org.ovirt.engine.core.bll.st >> orage.domain.ActivateStorageDomainCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] ActivateStorage Domain. Before >> Connect all hosts to pool. Time: Wed May 02 09:16:02 BST 2018 >> 2018-05-02 09:16:02,407+01 INFO [org.ovirt.engine.core.bll.st >> orage.connection.ConnectStorageToVdsCommand] >> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] Running >> command: ConnectStorageToVdsCommand internal: true. Entities affected : >> ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group >> CREATE_STORAGE_DOMAIN with role type ADMIN >> 2018-05-02 09:16:02,421+01 INFO [org.ovirt.engine.core.vdsbro >> ker.vdsbroker.ConnectStorageServerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] START, >> ConnectStorageServerVDSCommand(HostName = virtA003, >> StorageServerConnectionManagementVDSParameters:{hostId='fe28 >> 61fc-2b47-4807-b054-470198eda473', storagePoolId='00000000-0000-0 >> 000-0000-000000 >> 000000', storageType='NFS', connectionList='[StorageServer >> Connections:{id='da392861-aedc-4f1e-97f4-6919fb01f1e9', >> connection='backoffice01.cluster:/vm-iso', iqn='null', vfsType='null', >> mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', >> iface='null', netIfaceName='null'}]'}), log id: 23ce648f >> 2018-05-02 09:16:02,446+01 INFO [org.ovirt.engine.core.vdsbro >> ker.vdsbroker.ConnectStorageServerVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-33456) [40a82b47] FINISH, >> ConnectStorageServerVDSCommand, return: >> {da392861-aedc-4f1e-97f4-6919fb01f1e9=0}, >> log id: 23ce648f >> 2018-05-02 09:16:02,450+01 INFO [org.ovirt.engine.core.vdsbro >> ker.irsbroker.ActivateStorageDomainVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] START, >> ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman >> dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', >> ignoreFailoverLimit='false', stor >> ageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}), log id: 5c864594 >> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: >> IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command >> ActivateStorageDomainVDS failed: Storage domain not in pool: >> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, pool=5a5 >> 4bf81-0228-02bc-0358-000000000304' >> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.vdsbrok >> er.irsbroker.ActivateStorageDomainVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command >> 'ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSComman >> dParameters:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', >> ignoreFailoverLimit='false', st >> orageDomainId='f5914df0-f46c-4cc0-b666-c929aa0225ae'})' execution >> failed: IRSGenericException: IRSErrorException: Storage domain not in pool: >> u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304' >> 2018-05-02 09:16:02,635+01 INFO [org.ovirt.engine.core.vdsbro >> ker.irsbroker.ActivateStorageDomainVDSCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] FINISH, >> ActivateStorageDomainVDSCommand, log id: 5c864594 >> 2018-05-02 09:16:02,635+01 ERROR [org.ovirt.engine.core.bll.sto >> rage.domain.ActivateStorageDomainCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command >> 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' >> failed: EngineException: org.ovirt.engine.core.vdsbroke >> r.irsbroker.IrsOperationFailedNoFailove >> rException: IRSGenericException: IRSErrorException: Storage domain not in >> pool: u'domain=f5914df0-f46c-4cc0-b666-c929aa0225ae, >> pool=5a54bf81-0228-02bc-0358-000000000304' (Failed with error >> StorageDomainNotInPool and code 353) >> 2018-05-02 09:16:02,636+01 INFO [org.ovirt.engine.core.bll.st >> orage.domain.ActivateStorageDomainCommand] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] Command >> [id=22b0f3c1-9a09-4e26-8096-d83465c8f4ee]: Compensating >> CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.b >> usinessentities.StoragePoolIsoMap; snapshot: EntityStatus >> Snapshot:{id='StoragePoolIsoMapId:{storagePoolId='5a54bf81-0228-02bc-0358-000000000304', >> storageId='f5914df0-f46c-4cc0-b666-c929aa0225ae'}', >> status='Maintenance'}. >> 2018-05-02 09:16:02,660+01 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] >> (EE-ManagedThreadFactory-engine-Thread-33455) >> [93433989-8e26-48a9-bd3a-2ab95f296c08] EVENT_ID: >> USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Failed to activate Storage >> Domain VMISOs (Data Center Default) by admin@internal-authz >> >> Regards, >> Callum >> >> -- >> >> Callum Smith >> Research Computing Core >> Wellcome Trust Centre for Human Genetics >> University of Oxford >> e. cal...@well.ox.ac.uk >> >> On 2 May 2018, at 08:44, Fred Rolland <froll...@redhat.com> wrote: >> >> Hi, >> >> Can you provide logs from engine and Vdsm(SPM)? >> What is the state now? >> >> Thanks, >> Fred >> >> On Tue, May 1, 2018 at 4:11 PM, Callum Smith <cal...@well.ox.ac.uk> >> wrote: >> >>> Dear All, >>> >>> It appears that clicking "detach" on the ISO storage domain is a really >>> bad idea. This has gotten half way through the procedure and now can't be >>> recovered from. Is there any advice for re-attaching the ISO storage domain >>> manually? An NFS mount didn't add it back to the "pool" unfortunately. >>> >>> On a separate note, is it possible to migrate this storage to a new >>> location? And if so how. >>> >>> Regards, >>> Callum >>> >>> -- >>> >>> Callum Smith >>> Research Computing Core >>> Wellcome Trust Centre for Human Genetics >>> University of Oxford >>> e. cal...@well.ox.ac.uk >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
