Dear Sir;
Attached are engine and vdsm logs.
Please; check and advise.
Best Regards
Fawzy
On 02/11/2016 12:05 AM, Fawzy Ibrahim wrote:
> Dear Sir;
>
> I have did the same and the same error message still exists
>
> Did you check the logs attached in my previous message?
>
> Fawzy Ibrahim
> Redhat Certified Engineer
> Palestine
>
>
> On 02/10/2016 11:47 PM, Raz Tamir wrote:
>> Hi Fawzy,
>> After you rebooted your 3 hosts you need to make sure you are
>> confirming the hosts are actually rebooted.
>> You can do it by right clicking on each host and select "confirm host
>> has been rebooted"
>>
>>
>>
>> Thanks,
>> Raz Tamir
>> Red Hat Israel
>>
>> On Wed, Feb 10, 2016 at 8:50 PM, Fawzy Ibrahim <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Dear All;
>>
>> I have oVirt Engine Version: 3.5.6.2-1.el6 with three nodesrunning
>> standard Centos 6.7 64bit.
>>
>> The main data storage storage is on OpenFiler iScsi volume.
>>
>> After rebooting the three nodes; the main data storage domain is down
>> and all the three hosts are marked as "Non Operational"
>>
>> When I try to activate any host; I get the below error message:-
>>
>> "Host XXXXX cannot access the Storage Domain(s) <UNKNOWN> attached
>> to the Data Center Default. Setting Host state to
>> Non-Operational."
>>
>>
>> I have attached the engine and vdsm log.
>>
>> Best Regards
>> Fawzy Ibrahim
>>
>> _______________________________________________
>> Users mailing list
>> [email protected] <mailto:[email protected]>
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
2016-02-07 03:56:40,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf.
2016-02-07 04:56:40,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
2016-02-07 05:56:40,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
2016-02-07 06:56:40,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
2016-02-07 07:56:40,454 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf.
2016-02-07 08:56:40,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf.
2016-02-07 09:56:40,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf.
2016-02-07 10:56:40,459 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf.
2016-02-07 11:56:40,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf.
2016-02-07 12:56:40,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf.
2016-02-07 13:56:40,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf.
2016-02-07 14:56:40,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf.
2016-02-07 15:56:40,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
2016-02-07 16:56:40,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
2016-02-07 17:56:40,467 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf.
2016-02-07 18:56:40,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
2016-02-07 19:56:40,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf.
2016-02-07 20:56:40,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
2016-02-07 21:56:40,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
2016-02-07 22:56:40,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf.
2016-02-07 23:56:40,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf.
2016-02-08 00:56:40,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
2016-02-08 01:56:40,513 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf.
2016-02-08 02:56:40,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf.
2016-02-08 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-65) Start deleteAgedOutCommandEntities
2016-02-08 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-99) Start deleteAgedOutAuditLogs
2016-02-08 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-65) Finished deleteAgedOutCommandEntities
2016-02-08 03:35:35,018 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-99) Finished deleteAgedOutAuditLogs
2016-02-08 03:56:40,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf.
2016-02-08 04:56:40,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf.
2016-02-08 05:56:40,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf.
2016-02-08 06:56:40,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
2016-02-08 07:56:40,521 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf.
2016-02-08 08:56:40,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf.
2016-02-08 09:56:40,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
2016-02-08 10:56:40,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf.
2016-02-08 11:56:40,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf.
2016-02-08 12:56:40,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf.
2016-02-08 13:56:40,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
2016-02-08 14:56:40,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf.
2016-02-08 15:56:40,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf.
2016-02-08 16:56:40,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf.
2016-02-08 17:56:40,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf.
2016-02-08 18:56:40,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf.
2016-02-08 19:56:40,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf.
2016-02-08 20:56:40,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf.
2016-02-08 21:56:40,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf.
2016-02-08 22:56:40,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf.
2016-02-08 23:56:40,597 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf.
2016-02-09 00:56:40,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf.
2016-02-09 01:56:40,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf.
2016-02-09 02:56:40,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf.
2016-02-09 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-98) Start deleteAgedOutAuditLogs
2016-02-09 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-2) Start deleteAgedOutCommandEntities
2016-02-09 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-98) Finished deleteAgedOutAuditLogs
2016-02-09 03:35:35,014 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-2) Finished deleteAgedOutCommandEntities
2016-02-09 03:56:40,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf.
2016-02-09 04:56:40,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf.
2016-02-09 05:56:40,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf.
2016-02-09 06:56:40,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf.
2016-02-09 07:56:40,632 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf.
2016-02-09 08:56:40,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf.
2016-02-09 09:56:40,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
2016-02-09 10:56:40,662 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
2016-02-09 11:56:40,663 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf.
2016-02-09 12:56:40,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf.
2016-02-09 13:56:40,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf.
2016-02-09 14:56:40,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
2016-02-09 15:37:22,531 INFO [org.ovirt.engine.docs.utils.servlet.ContextSensitiveHelpMappingServlet] (ajp--127.0.0.1-8702-1) Context-sensitive help is not installed. Manual directory doesn't exist: /usr/share/ovirt-engine/manual
2016-02-09 15:37:28,063 INFO [org.ovirt.engine.core.bll.aaa.LoginBaseCommand] (ajp--127.0.0.1-8702-2) Cant login user "admin" with authentication profile "internal" because the authentication failed.
2016-02-09 15:37:28,082 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin cannot login, please verify the username and password.
2016-02-09 15:37:28,090 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal failed to log in.
2016-02-09 15:37:28,092 WARN [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-2) CanDoAction of action LoginAdminUser failed for user admin@internal. Reasons: USER_FAILED_TO_AUTHENTICATE_WRONG_USERNAME_OR_PASSWORD
2016-02-09 15:37:31,122 INFO [org.ovirt.engine.core.bll.aaa.LoginAdminUserCommand] (ajp--127.0.0.1-8702-5) Running command: LoginAdminUserCommand internal: false.
2016-02-09 15:37:31,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in.
2016-02-09 15:37:42,241 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-3) [3e19cc30] Lock Acquired to object EngineLock [exclusiveLocks= key: b8945bff-8fdd-497c-9c8c-3a642294a7a0 value: VDS
, sharedLocks= ]
2016-02-09 15:37:42,335 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b8945bff-8fdd-497c-9c8c-3a642294a7a0 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2016-02-09 15:37:42,336 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Before acquiring lock in order to prevent monitoring for host KavLab from data-center Default
2016-02-09 15:37:42,337 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Lock acquired, from now a monitoring of host will be skipped for host KavLab from data-center Default
2016-02-09 15:37:42,345 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] START, SetVdsStatusVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3a7208be
2016-02-09 15:37:42,346 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] VDS KavLab is spm and moved from up calling resetIrs.
2016-02-09 15:37:42,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] START, ResetIrsVDSCommand( storagePoolId = 00000002-0002-0002-0002-000000000184, ignoreFailoverLimit = false, vdsId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, ignoreStopFailed = false), log id: 714600d4
2016-02-09 15:37:42,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] START, SpmStopVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, storagePoolId = 00000002-0002-0002-0002-000000000184), log id: 4511274a
2016-02-09 15:37:42,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value
TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]]
2016-02-09 15:37:42,372 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] HostName = KavLab
2016-02-09 15:37:42,372 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Command HSMGetAllTasksStatusesVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0) execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()
2016-02-09 15:37:42,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] FINISH, SpmStopVDSCommand, log id: 4511274a
2016-02-09 15:37:42,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] FINISH, ResetIrsVDSCommand, log id: 714600d4
2016-02-09 15:37:42,412 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] FINISH, SetVdsStatusVDSCommand, log id: 3a7208be
2016-02-09 15:37:42,446 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Activate finished. Lock released. Monitoring can run now for host KavLab from data-center Default
2016-02-09 15:37:42,466 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Correlation ID: 3e19cc30, Job ID: c3e7a9c0-b878-4ec6-8d51-637b4a608ed3, Call Stack: null, Custom Event ID: -1, Message: Host KavLab was activated by admin@internal.
2016-02-09 15:37:42,479 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-8) [3e19cc30] Lock freed to object EngineLock [exclusiveLocks= key: b8945bff-8fdd-497c-9c8c-3a642294a7a0 value: VDS
, sharedLocks= ]
2016-02-09 15:37:43,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) START, GetHardwareInfoVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, vds=Host[KavLab,b8945bff-8fdd-497c-9c8c-3a642294a7a0]), log id: 297b0ff6
2016-02-09 15:37:43,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) FINISH, GetHardwareInfoVDSCommand, log id: 297b0ff6
2016-02-09 15:37:43,326 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [37454c69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b8945bff-8fdd-497c-9c8c-3a642294a7a0 Type: VDS
2016-02-09 15:37:43,344 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-40) [6be5fbfa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:37:43,359 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [14f84a1f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:37:43,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [14f84a1f] START, ConnectStorageServerVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, storagePoolId = 00000002-0002-0002-0002-000000000184, storageType = ISCSI, connectionList = [{ id: fe3cc82b-6668-4c17-b9ee-7db54557237e, connection: 129.1.2.40, iqn: iqn.2006-01.com.openfiler:tsn.6f138bf1cc59, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6642c004
2016-02-09 15:37:44,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [14f84a1f] FINISH, ConnectStorageServerVDSCommand, return: {fe3cc82b-6668-4c17-b9ee-7db54557237e=0}, log id: 6642c004
2016-02-09 15:37:44,139 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [14f84a1f] Host KavLab storage connection was succeeded
2016-02-09 15:37:44,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) START, ConnectStoragePoolVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, vdsId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5), log id: 774876f9
2016-02-09 15:37:44,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc']]
2016-02-09 15:37:44,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) HostName = KavLab
2016-02-09 15:37:44,540 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) Command ConnectStoragePoolVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, vdsId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'
2016-02-09 15:37:44,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-10) FINISH, ConnectStoragePoolVDSCommand, log id: 774876f9
2016-02-09 15:37:44,542 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-10) Could not connect host KavLab to pool Default, as the master domain is in inactive/unknown status - not failing the operation
2016-02-09 15:37:44,550 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage='NoneType' object has no attribute 'statistics']
2016-02-09 15:37:44,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) Failed in GetStatsVDS method
2016-02-09 15:37:44,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-10) Command GetStatsVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, vds=Host[KavLab,b8945bff-8fdd-497c-9c8c-3a642294a7a0]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603
2016-02-09 15:37:44,552 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-10) Could not get Host statistics for Host KavLab, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2066) [bll.jar:]
at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:852) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_95]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
2016-02-09 15:37:44,579 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host KavLab reports about one of the Active Storage Domains as Problematic.
2016-02-09 15:37:44,635 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-40) [57703869] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b8945bff-8fdd-497c-9c8c-3a642294a7a0 Type: VDS
2016-02-09 15:37:44,637 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [57703869] START, SetVdsStatusVDSCommand(HostName = KavLab, HostId = b8945bff-8fdd-497c-9c8c-3a642294a7a0, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 43b98ab1
2016-02-09 15:37:44,655 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [57703869] FINISH, SetVdsStatusVDSCommand, log id: 43b98ab1
2016-02-09 15:37:44,668 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [57703869] Correlation ID: 57703869, Job ID: ed254bb6-c63b-4b28-af87-015de858c079, Call Stack: null, Custom Event ID: -1, Message: Host KavLab cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
2016-02-09 15:37:44,684 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [57703869] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host KavLab.
2016-02-09 15:37:44,690 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [57703869] Correlation ID: 6be5fbfa, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host KavLab to Storage Pool Default
2016-02-09 15:37:44,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [209c9d50] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b8945bff-8fdd-497c-9c8c-3a642294a7a0 Type: VDS
2016-02-09 15:37:44,727 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) [209c9d50] Host b8945bff-8fdd-497c-9c8c-3a642294a7a0 : KavLab is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.
2016-02-09 15:37:54,134 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [13c7b715] Lock Acquired to object EngineLock [exclusiveLocks= key: 708088a0-a641-4a7e-8732-55dde96c08dc value: STORAGE
, sharedLocks= ]
2016-02-09 15:37:54,143 WARN [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [13c7b715] CanDoAction of action ActivateStorageDomain failed for user admin@internal. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
2016-02-09 15:37:54,144 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-4) [13c7b715] Lock freed to object EngineLock [exclusiveLocks= key: 708088a0-a641-4a7e-8732-55dde96c08dc value: STORAGE
, sharedLocks= ]
2016-02-09 15:37:54,167 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [707fe94f] Lock Acquired to object EngineLock [exclusiveLocks= key: 708088a0-a641-4a7e-8732-55dde96c08dc value: STORAGE
, sharedLocks= ]
2016-02-09 15:37:54,171 WARN [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [707fe94f] CanDoAction of action ActivateStorageDomain failed for user admin@internal. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ACTIVATE,ACTION_TYPE_FAILED_NO_VDS_IN_POOL
2016-02-09 15:37:54,171 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp--127.0.0.1-8702-1) [707fe94f] Lock freed to object EngineLock [exclusiveLocks= key: 708088a0-a641-4a7e-8732-55dde96c08dc value: STORAGE
, sharedLocks= ]
2016-02-09 15:38:05,343 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [5acbff1e] Lock Acquired to object EngineLock [exclusiveLocks= key: b68ef697-cd93-489a-b269-f6b7868415dc value: VDS
, sharedLocks= ]
2016-02-09 15:38:05,371 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8) [18a73b8] Failed to Acquire Lock to object EngineLock [exclusiveLocks= key: b68ef697-cd93-489a-b269-f6b7868415dc value: VDS
, sharedLocks= ]
2016-02-09 15:38:05,372 WARN [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8) [18a73b8] CanDoAction of action ActivateVds failed for user admin@internal. Reasons: VAR__ACTION__ACTIVATE,VAR__TYPE__HOST,ACTION_TYPE_FAILED_OBJECT_LOCKED
2016-02-09 15:38:05,412 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2016-02-09 15:38:05,413 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Before acquiring lock in order to prevent monitoring for host KhoLab from data-center Default
2016-02-09 15:38:05,413 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Lock acquired, from now a monitoring of host will be skipped for host KhoLab from data-center Default
2016-02-09 15:38:05,416 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] START, SetVdsStatusVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 52c03ae9
2016-02-09 15:38:05,423 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] FINISH, SetVdsStatusVDSCommand, log id: 52c03ae9
2016-02-09 15:38:05,434 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Activate finished. Lock released. Monitoring can run now for host KhoLab from data-center Default
2016-02-09 15:38:05,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Correlation ID: 5acbff1e, Job ID: c9db7d39-1c82-44be-866a-e4d641916ee6, Call Stack: null, Custom Event ID: -1, Message: Host KhoLab was activated by admin@internal.
2016-02-09 15:38:05,456 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-15) [5acbff1e] Lock freed to object EngineLock [exclusiveLocks= key: b68ef697-cd93-489a-b269-f6b7868415dc value: VDS
, sharedLocks= ]
2016-02-09 15:38:07,188 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to kholab.gmc.com.sa/129.1.2.41
2016-02-09 15:38:07,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) START, GetHardwareInfoVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vds=Host[KhoLab,b68ef697-cd93-489a-b269-f6b7868415dc]), log id: 31668f3b
2016-02-09 15:38:07,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, GetHardwareInfoVDSCommand, log id: 31668f3b
2016-02-09 15:38:07,891 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [770d7b20] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:07,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [770d7b20] START, GetHardwareInfoVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vds=Host[KhoLab,b68ef697-cd93-489a-b269-f6b7868415dc]), log id: 6b674c97
2016-02-09 15:38:07,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [770d7b20] FINISH, GetHardwareInfoVDSCommand, log id: 6b674c97
2016-02-09 15:38:07,972 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [36b7ced4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:08,072 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [34c4d270] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:08,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [34c4d270] START, ConnectStorageServerVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, storageType = ISCSI, connectionList = [{ id: fe3cc82b-6668-4c17-b9ee-7db54557237e, connection: 129.1.2.40, iqn: iqn.2006-01.com.openfiler:tsn.6f138bf1cc59, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3d7cfc1a
2016-02-09 15:38:09,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [34c4d270] FINISH, ConnectStorageServerVDSCommand, return: {fe3cc82b-6668-4c17-b9ee-7db54557237e=0}, log id: 3d7cfc1a
2016-02-09 15:38:09,108 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [34c4d270] Host KhoLab storage connection was succeeded
2016-02-09 15:38:09,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) START, ConnectStoragePoolVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vdsId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5), log id: 5fed283c
2016-02-09 15:38:09,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc']]
2016-02-09 15:38:09,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) HostName = KhoLab
2016-02-09 15:38:09,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) Command ConnectStoragePoolVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vdsId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'
2016-02-09 15:38:09,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-18) FINISH, ConnectStoragePoolVDSCommand, log id: 5fed283c
2016-02-09 15:38:09,574 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-18) Could not connect host KhoLab to pool Default, as the master domain is in inactive/unknown status - not failing the operation
2016-02-09 15:38:09,607 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage='NoneType' object has no attribute 'statistics']
2016-02-09 15:38:09,608 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) Failed in GetStatsVDS method
2016-02-09 15:38:09,608 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-18) Command GetStatsVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vds=Host[KhoLab,b68ef697-cd93-489a-b269-f6b7868415dc]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603
2016-02-09 15:38:09,609 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-18) Could not get Host statistics for Host KhoLab, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2066) [bll.jar:]
at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:852) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_95]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
2016-02-09 15:38:09,623 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-18) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host KhoLab reports about one of the Active Storage Domains as Problematic.
2016-02-09 15:38:09,667 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-73) [47575eb8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:09,669 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [47575eb8] START, SetVdsStatusVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 8676698
2016-02-09 15:38:09,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [47575eb8] FINISH, SetVdsStatusVDSCommand, log id: 8676698
2016-02-09 15:38:09,723 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [47575eb8] Correlation ID: 47575eb8, Job ID: 5388d397-eee1-4779-80c8-ff23ff33edf5, Call Stack: null, Custom Event ID: -1, Message: Host KhoLab cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
2016-02-09 15:38:09,737 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [47575eb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host KhoLab.
2016-02-09 15:38:09,745 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [47575eb8] Correlation ID: 36b7ced4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host KhoLab to Storage Pool Default
2016-02-09 15:38:09,770 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [1893a380] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:09,774 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [1893a380] Host b68ef697-cd93-489a-b269-f6b7868415dc : KhoLab is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.
2016-02-09 15:38:20,193 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-4) [43a0ba16] Lock Acquired to object EngineLock [exclusiveLocks= key: b68ef697-cd93-489a-b269-f6b7868415dc value: VDS
, sharedLocks= ]
2016-02-09 15:38:20,265 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Running command: ActivateVdsCommand internal: false. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2016-02-09 15:38:20,265 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Before acquiring lock in order to prevent monitoring for host KhoLab from data-center Default
2016-02-09 15:38:20,266 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Lock acquired, from now a monitoring of host will be skipped for host KhoLab from data-center Default
2016-02-09 15:38:20,269 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] START, SetVdsStatusVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 541fc8f7
2016-02-09 15:38:20,278 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] FINISH, SetVdsStatusVDSCommand, log id: 541fc8f7
2016-02-09 15:38:20,312 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Activate finished. Lock released. Monitoring can run now for host KhoLab from data-center Default
2016-02-09 15:38:20,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Correlation ID: 43a0ba16, Job ID: 277f42d1-fc64-41d7-974f-faa72dac0ca0, Call Stack: null, Custom Event ID: -1, Message: Host KhoLab was activated by admin@internal.
2016-02-09 15:38:20,334 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-21) [43a0ba16] Lock freed to object EngineLock [exclusiveLocks= key: b68ef697-cd93-489a-b269-f6b7868415dc value: VDS
, sharedLocks= ]
2016-02-09 15:38:21,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) START, GetHardwareInfoVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vds=Host[KhoLab,b68ef697-cd93-489a-b269-f6b7868415dc]), log id: 7a8af93d
2016-02-09 15:38:21,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, GetHardwareInfoVDSCommand, log id: 7a8af93d
2016-02-09 15:38:21,903 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [3da74316] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:21,910 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [57564649] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:21,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [5f378577] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:21,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [5f378577] START, ConnectStorageServerVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, storageType = ISCSI, connectionList = [{ id: fe3cc82b-6668-4c17-b9ee-7db54557237e, connection: 129.1.2.40, iqn: iqn.2006-01.com.openfiler:tsn.6f138bf1cc59, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 41cc4b24
2016-02-09 15:38:22,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [5f378577] FINISH, ConnectStorageServerVDSCommand, return: {fe3cc82b-6668-4c17-b9ee-7db54557237e=0}, log id: 41cc4b24
2016-02-09 15:38:22,414 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [5f378577] Host KhoLab storage connection was succeeded
2016-02-09 15:38:22,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-23) START, ConnectStoragePoolVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vdsId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5), log id: 3b0678ac
2016-02-09 15:38:22,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-23) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc']]
2016-02-09 15:38:22,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-23) HostName = KhoLab
2016-02-09 15:38:22,760 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-23) Command ConnectStoragePoolVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vdsId = b68ef697-cd93-489a-b269-f6b7868415dc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'
2016-02-09 15:38:22,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-23) FINISH, ConnectStoragePoolVDSCommand, log id: 3b0678ac
2016-02-09 15:38:22,762 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-23) Could not connect host KhoLab to pool Default, as the master domain is in inactive/unknown status - not failing the operation
2016-02-09 15:38:22,770 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage='NoneType' object has no attribute 'statistics']
2016-02-09 15:38:22,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Failed in GetStatsVDS method
2016-02-09 15:38:22,770 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-23) Command GetStatsVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, vds=Host[KhoLab,b68ef697-cd93-489a-b269-f6b7868415dc]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603
2016-02-09 15:38:22,771 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-23) Could not get Host statistics for Host KhoLab, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2066) [bll.jar:]
at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:852) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_95]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
2016-02-09 15:38:22,789 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host KhoLab reports about one of the Active Storage Domains as Problematic.
2016-02-09 15:38:22,834 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [3bfc972d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:22,836 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [3bfc972d] START, SetVdsStatusVDSCommand(HostName = KhoLab, HostId = b68ef697-cd93-489a-b269-f6b7868415dc, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 8defd9
2016-02-09 15:38:22,845 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [3bfc972d] FINISH, SetVdsStatusVDSCommand, log id: 8defd9
2016-02-09 15:38:22,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [3bfc972d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host KhoLab.
2016-02-09 15:38:22,900 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [3bfc972d] Correlation ID: 57564649, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host KhoLab to Storage Pool Default
2016-02-09 15:38:22,925 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [64e06f3e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: b68ef697-cd93-489a-b269-f6b7868415dc Type: VDS
2016-02-09 15:38:22,926 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [64e06f3e] Host b68ef697-cd93-489a-b269-f6b7868415dc : KhoLab is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.
2016-02-09 15:38:26,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-8) [1f68b2fb] Lock Acquired to object EngineLock [exclusiveLocks= key: 158b0263-0eee-4425-926d-a58346651efc value: VDS
, sharedLocks= ]
2016-02-09 15:38:26,800 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 158b0263-0eee-4425-926d-a58346651efc Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2016-02-09 15:38:26,801 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Before acquiring lock in order to prevent monitoring for host OccLab from data-center Default
2016-02-09 15:38:26,802 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Lock acquired, from now a monitoring of host will be skipped for host OccLab from data-center Default
2016-02-09 15:38:26,804 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] START, SetVdsStatusVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 54712bc9
2016-02-09 15:38:26,812 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] FINISH, SetVdsStatusVDSCommand, log id: 54712bc9
2016-02-09 15:38:26,823 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Activate finished. Lock released. Monitoring can run now for host OccLab from data-center Default
2016-02-09 15:38:26,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Correlation ID: 1f68b2fb, Job ID: 773b0a4e-c404-431c-8180-8eac52d84f22, Call Stack: null, Custom Event ID: -1, Message: Host OccLab was activated by admin@internal.
2016-02-09 15:38:26,845 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-26) [1f68b2fb] Lock freed to object EngineLock [exclusiveLocks= key: 158b0263-0eee-4425-926d-a58346651efc value: VDS
, sharedLocks= ]
2016-02-09 15:38:26,870 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp--127.0.0.1-8702-9) [191b5257] Lock Acquired to object EngineLock [exclusiveLocks= key: 158b0263-0eee-4425-926d-a58346651efc value: VDS
, sharedLocks= ]
2016-02-09 15:38:26,923 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 158b0263-0eee-4425-926d-a58346651efc Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2016-02-09 15:38:26,924 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] Before acquiring lock in order to prevent monitoring for host OccLab from data-center Default
2016-02-09 15:38:26,924 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] Lock acquired, from now a monitoring of host will be skipped for host OccLab from data-center Default
2016-02-09 15:38:26,928 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] START, SetVdsStatusVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4dfdaa60
2016-02-09 15:38:26,943 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] FINISH, SetVdsStatusVDSCommand, log id: 4dfdaa60
2016-02-09 15:38:26,956 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] Activate finished. Lock released. Monitoring can run now for host OccLab from data-center Default
2016-02-09 15:38:26,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-27) [191b5257] Correlation ID: 191b5257, Job ID: 367585bb-5082-46be-a561-079f9ffee4f4, Call Stack: null, Custom Event ID: -1, Message: Host OccLab was activated by admin@internal.
2016-02-09 15:38:26,978 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-8-thread-27) [191b5257] Lock freed to object EngineLock [exclusiveLocks= key: 158b0263-0eee-4425-926d-a58346651efc value: VDS
, sharedLocks= ]
2016-02-09 15:38:29,305 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) Connecting to occlab.gmc.com.sa/129.1.2.38
2016-02-09 15:38:30,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) START, GetHardwareInfoVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, vds=Host[OccLab,158b0263-0eee-4425-926d-a58346651efc]), log id: 2c0099ae
2016-02-09 15:38:30,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) FINISH, GetHardwareInfoVDSCommand, log id: 2c0099ae
2016-02-09 15:38:30,440 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [4e22acc1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 158b0263-0eee-4425-926d-a58346651efc Type: VDS
2016-02-09 15:38:30,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4e22acc1] START, GetHardwareInfoVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, vds=Host[OccLab,158b0263-0eee-4425-926d-a58346651efc]), log id: 12f5067b
2016-02-09 15:38:30,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4e22acc1] FINISH, GetHardwareInfoVDSCommand, log id: 12f5067b
2016-02-09 15:38:30,518 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [842640f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:30,532 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [45a7c27f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000184 Type: StoragePool
2016-02-09 15:38:30,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [45a7c27f] START, ConnectStorageServerVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, storagePoolId = 00000002-0002-0002-0002-000000000184, storageType = ISCSI, connectionList = [{ id: fe3cc82b-6668-4c17-b9ee-7db54557237e, connection: 129.1.2.40, iqn: iqn.2006-01.com.openfiler:tsn.6f138bf1cc59, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1d30708f
2016-02-09 15:38:31,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [45a7c27f] FINISH, ConnectStorageServerVDSCommand, return: {fe3cc82b-6668-4c17-b9ee-7db54557237e=0}, log id: 1d30708f
2016-02-09 15:38:31,628 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [45a7c27f] Host OccLab storage connection was succeeded
2016-02-09 15:38:31,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) START, ConnectStoragePoolVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, vdsId = 158b0263-0eee-4425-926d-a58346651efc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5), log id: 656ccf95
2016-02-09 15:38:32,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc']]
2016-02-09 15:38:32,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) HostName = OccLab
2016-02-09 15:38:32,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) Command ConnectStoragePoolVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, vdsId = 158b0263-0eee-4425-926d-a58346651efc, storagePoolId = 00000002-0002-0002-0002-000000000184, masterVersion = 5) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'
2016-02-09 15:38:32,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-29) FINISH, ConnectStoragePoolVDSCommand, log id: 656ccf95
2016-02-09 15:38:32,115 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-29) Could not connect host OccLab to pool Default, as the master domain is in inactive/unknown status - not failing the operation
2016-02-09 15:38:32,163 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) Unexpected return value: StatusForXmlRpc [mCode=-32603, mMessage='NoneType' object has no attribute 'statistics']
2016-02-09 15:38:32,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) Failed in GetStatsVDS method
2016-02-09 15:38:32,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (org.ovirt.thread.pool-8-thread-29) Command GetStatsVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, vds=Host[OccLab,158b0263-0eee-4425-926d-a58346651efc]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603
2016-02-09 15:38:32,165 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-29) Could not get Host statistics for Host OccLab, Error is org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16): org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetStatsVDS, error = 'NoneType' object has no attribute 'statistics', code = -32603 (Failed with error unexpected and code 16)
at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:117) [bll.jar:]
at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:2066) [bll.jar:]
at org.ovirt.engine.core.bll.storage.StorageHandlingCommandBase.runVdsCommand(StorageHandlingCommandBase.java:852) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.proceedVdsStats(InitVdsOnUpCommand.java:306) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.runConnectHostToPoolEvent(InitVdsOnUpCommand.java:276) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand.access$000(InitVdsOnUpCommand.java:79) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:237) [bll.jar:]
at org.ovirt.engine.core.bll.InitVdsOnUpCommand$1.call(InitVdsOnUpCommand.java:234) [bll.jar:]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_95]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
2016-02-09 15:38:32,198 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host OccLab reports about one of the Active Storage Domains as Problematic.
2016-02-09 15:38:32,256 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [350e733d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 158b0263-0eee-4425-926d-a58346651efc Type: VDS
2016-02-09 15:38:32,258 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [350e733d] START, SetVdsStatusVDSCommand(HostName = OccLab, HostId = 158b0263-0eee-4425-926d-a58346651efc, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1bf5bd5c
2016-02-09 15:38:32,267 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [350e733d] FINISH, SetVdsStatusVDSCommand, log id: 1bf5bd5c
2016-02-09 15:38:32,278 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [350e733d] Correlation ID: 350e733d, Job ID: 05251e4d-955a-417f-954b-7c19eabc48b6, Call Stack: null, Custom Event ID: -1, Message: Host OccLab cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center Default. Setting Host state to Non-Operational.
2016-02-09 15:38:32,293 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [350e733d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host OccLab.
2016-02-09 15:38:32,300 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [350e733d] Correlation ID: 842640f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host OccLab to Storage Pool Default
2016-02-09 15:38:32,326 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [13b54b01] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 158b0263-0eee-4425-926d-a58346651efc Type: VDS
2016-02-09 15:38:32,329 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [13b54b01] Host 158b0263-0eee-4425-926d-a58346651efc : OccLab is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.
2016-02-09 15:56:40,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf.
2016-02-09 16:56:40,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf.
2016-02-09 17:56:40,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf.
2016-02-09 18:56:40,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf.
2016-02-09 19:56:40,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf.
2016-02-09 20:56:40,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf.
2016-02-09 21:56:40,679 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
2016-02-09 22:56:40,683 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf.
2016-02-09 23:56:40,684 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf.
2016-02-10 00:56:40,685 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf.
2016-02-10 01:56:40,686 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf.
2016-02-10 02:56:40,688 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf.
2016-02-10 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-65) Start deleteAgedOutAuditLogs
2016-02-10 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Start deleteAgedOutCommandEntities
2016-02-10 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-65) Finished deleteAgedOutAuditLogs
2016-02-10 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-15) Finished deleteAgedOutCommandEntities
2016-02-10 03:56:40,690 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf.
2016-02-10 04:56:40,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf.
2016-02-10 05:56:40,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf.
2016-02-10 06:56:40,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
2016-02-10 07:56:40,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf.
2016-02-10 08:56:40,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf.
2016-02-10 09:56:40,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf.
2016-02-10 10:56:40,699 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
2016-02-10 11:56:40,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf.
2016-02-10 12:56:40,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf.
2016-02-10 13:56:40,703 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf.
2016-02-10 14:56:40,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf.
2016-02-10 15:56:40,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf.
2016-02-10 16:56:40,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf.
2016-02-10 17:56:40,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf.
2016-02-10 18:56:40,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf.
2016-02-10 19:56:40,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf.
2016-02-10 20:56:40,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf.
Thread-27::DEBUG::2016-02-09 15:37:42,356::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getAllTasksStatuses' in bridge with {}
Thread-27::DEBUG::2016-02-09 15:37:42,360::task::595::Storage.TaskManager.Task::(_updateState) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::moving from state init -> state preparing
Thread-27::INFO::2016-02-09 15:37:42,361::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None)
Thread-27::ERROR::2016-02-09 15:37:42,361::task::866::Storage.TaskManager.Task::(_setError) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 2188, in getAllTasksStatuses
raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-27::DEBUG::2016-02-09 15:37:42,362::task::885::Storage.TaskManager.Task::(_run) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::Task._run: 9368cd0e-86df-475d-a2bc-8dc589e32844 () {} failed - stopping task
Thread-27::DEBUG::2016-02-09 15:37:42,362::task::1217::Storage.TaskManager.Task::(stop) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::stopping in state preparing (force False)
Thread-27::DEBUG::2016-02-09 15:37:42,362::task::993::Storage.TaskManager.Task::(_decref) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::ref 1 aborting True
Thread-27::INFO::2016-02-09 15:37:42,363::task::1171::Storage.TaskManager.Task::(prepare) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::aborting: Task is aborted: 'Not SPM' - code 654
Thread-27::DEBUG::2016-02-09 15:37:42,363::task::1176::Storage.TaskManager.Task::(prepare) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::Prepare: aborted: Not SPM
Thread-27::DEBUG::2016-02-09 15:37:42,363::task::993::Storage.TaskManager.Task::(_decref) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::ref 0 aborting True
Thread-27::DEBUG::2016-02-09 15:37:42,363::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::Task._doAbort: force False
Thread-27::DEBUG::2016-02-09 15:37:42,363::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::DEBUG::2016-02-09 15:37:42,363::task::595::Storage.TaskManager.Task::(_updateState) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::moving from state preparing -> state aborting
Thread-27::DEBUG::2016-02-09 15:37:42,363::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::_aborting: recover policy none
Thread-27::DEBUG::2016-02-09 15:37:42,364::task::595::Storage.TaskManager.Task::(_updateState) Task=`9368cd0e-86df-475d-a2bc-8dc589e32844`::moving from state aborting -> state failed
Thread-27::DEBUG::2016-02-09 15:37:42,364::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-27::DEBUG::2016-02-09 15:37:42,364::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-27::ERROR::2016-02-09 15:37:42,364::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-28::DEBUG::2016-02-09 15:37:43,214::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getCapabilities' in bridge with {}
Thread-28::DEBUG::2016-02-09 15:37:43,221::utils::755::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None)
Thread-28::DEBUG::2016-02-09 15:37:43,236::utils::775::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-28::DEBUG::2016-02-09 15:37:43,264::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,269::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-object',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,270::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-plugin',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,276::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-account',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,276::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-proxy',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,276::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-doc',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,277::caps::718::root::(_getKeyPackages) rpm package ('gluster-swift-container',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,277::caps::718::root::(_getKeyPackages) rpm package ('glusterfs-geo-replication',) not found
Thread-28::DEBUG::2016-02-09 15:37:43,282::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getCapabilities' in bridge with {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:7eb1a0bf77d3'}], 'FC': []}, 'packages2': {'kernel': {'release': '358.2.1.el6.centos.plus.x86_64', 'buildtime': 1363129747.0, 'version': '2.6.32'}, 'glusterfs-rdma': {'release': '1.el6', 'buildtime': 1447082454L, 'version': '3.7.6'}, 'glusterfs-fuse': {'release': '1.el6', 'buildtime': 1447082454L, 'version': '3.7.6'}, 'spice-server': {'release': '12.el6_7.3', 'buildtime': 1444677692L, 'version': '0.12.4'}, 'vdsm': {'release': '0.el6', 'buildtime': 1448292377L, 'version': '4.16.30'}, 'qemu-kvm': {'release': '2.479.el6_7.2', 'buildtime': 1445728098L, 'version': '0.12.1.2'}, 'qemu-img': {'release': '2.479.el6_7.2', 'buildtime': 1445728098L, 'version': '0.12.1.2'}, 'libvirt': {'release': '54.el6_7.3', 'buildtime': 1450193614L, 'version': '0.10.2'}, 'glusterfs': {'release': '1.el6', 'buildtime': 1447082454L, 'version': '3.7.6'}, 'mom': {'release': '1.el6', 'buildtime': 1431706644L, 'version': '0.4.4'}, 'glusterfs-server': {'release': '1.el6', 'buildtime': 1447082454L, 'version': '3.7.6'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5540 @ 2.53GHz', 'liveMerge': 'false', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '0'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '129.1.2.42', 'cfg': {'DEFROUTE': 'yes', 'DNS1': '129.1.111.253', 'IPADDR': '129.1.2.42', 'HOTPLUG': 'no', 'GATEWAY': '129.1.111.249', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['fe80::4261:86ff:fe29:a52c/64'], 'gateway': '129.1.111.249', 'bootproto4': 'none', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv4addrs': ['129.1.2.42/16'], 'mtu': '1500', 'ipv6gateway': '::', 'ports': ['eth0']}}, 'bridges': {'ovirtmgmt': {'addr': '129.1.2.42', 'cfg': {'DEFROUTE': 'yes', 'DNS1': '129.1.111.253', 'IPADDR': '129.1.2.42', 'HOTPLUG': 'no', 'GATEWAY': '129.1.111.249', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.0.0', 'BOOTPROTO': 'none', 'STP': 'off', 'DEVICE': 'ovirtmgmt', 'MTU': '1500', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::4261:86ff:fe29:a52c/64'], 'mtu': '1500', 'netmask': '255.255.0.0', 'stp': 'off', 'ipv4addrs': ['129.1.2.42/16'], 'ipv6gateway': '::', 'gateway': '129.1.111.249', 'opts': {'topology_change_detected': '0', 'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '999', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3124', 'hello_timer': '136', 'multicast_querier_interval': '25496', 'max_age': '1999', 'hash_max': '512', 'stp_state': '0', 'root_id': '8000.40618629a52c', 'priority': '32768', 'multicast_membership_interval': '25996', 'root_path_cost': '0', 'root_port': '0', 'multicast_startup_query_count': '2', 'hello_time': '199', 'topology_change': '0', 'bridge_id': '8000.40618629a52c', 'topology_change_timer': '0', 'ageing_time': '29995', 'gc_timer': '36', 'group_addr': '1:80:c2:0:0:0', 'tcn_timer': '0', 'multicast_query_interval': '12498', 'multicast_last_member_interval': '99', 'multicast_router': '1', 'forward_delay': '0'}, 'ports': ['eth0']}}, 'uuid': '4FDDBE80-D656-11E2-8001-40618629A6A1', 'onlineCpus': u'0,1,2,3,4,5,6,7', 'nics': {'eth1': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '40:61:86:29:a5:2d', 'speed': 0}, 'eth0': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '40:61:86:29:a5:2c', 'DEVICE': 'eth0', 'ONBOOT': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '40:61:86:29:a5:2c', 'speed': 1000}}, 'software_revision': '0', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,dtes64,monitor,ds_cpl,vmx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,popcnt,lahf_lm,ida,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:7eb1a0bf77d3', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'autoNumaBalancing': 2, 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '00:00:00:00:00:00', 'slaves': [], 'opts': {}}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.16', 'memSize': '15944', 'cpuSpeed': '2660.000', 'numaNodes': {u'0': {'totalMemory': '15944', 'cpus': [0, 1, 2, 3, 4, 5, 6, 7]}}, 'version_name': 'Snow Man', 'vlans': {}, 'lastClientIface': 'ovirtmgmt', 'cpuCores': '4', 'kvmEnabled': 'true', 'guestOverhead': '65', 'cpuThreads': '8', 'emulatedMachines': [u'rhel6.6.0', u'pc', u'rhel6.5.0', u'rhel6.4.0', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'rngSources': ['random'], 'operatingSystem': {'release': '7.el6.centos.12.3', 'version': '6', 'name': 'RHEL'}}
Thread-29::DEBUG::2016-02-09 15:37:43,293::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getHardwareInfo' in bridge with {}
Thread-29::DEBUG::2016-02-09 15:37:43,295::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'Express5800/R120a-2 [N8100-1501F]', 'systemUUID': '4fddbe80-d656-11e2-8001-40618629a6a1', 'systemSerialNumber': '1000234811', 'systemVersion': 'FR1.0', 'systemManufacturer': 'NEC'}
Thread-31::DEBUG::2016-02-09 15:37:43,409::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {'connectionParams': [{'port': '3260', 'connection': '129.1.2.40', 'iqn': 'iqn.2006-01.com.openfiler:tsn.6f138bf1cc59', 'user': 'ovirt', 'tpgt': '1', 'password': '********', 'id': 'fe3cc82b-6668-4c17-b9ee-7db54557237e'}], 'storagepoolID': '00000002-0002-0002-0002-000000000184', 'domainType': 3}
Thread-31::DEBUG::2016-02-09 15:37:43,409::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f74c7e3-eef2-4c3d-a1b9-b20c0432f3c0`::moving from state init -> state preparing
Thread-31::INFO::2016-02-09 15:37:43,410::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID='00000002-0002-0002-0002-000000000184', conList=[{'connection': '129.1.2.40', 'iqn': 'iqn.2006-01.com.openfiler:tsn.6f138bf1cc59', 'user': 'ovirt', 'tpgt': '1', 'password': '******', 'id': 'fe3cc82b-6668-4c17-b9ee-7db54557237e', 'port': '3260'}], options=None)
Thread-31::DEBUG::2016-02-09 15:37:43,410::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 --op=new (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,447::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,448::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 -n node.session.auth.authmethod -v '****' --op=update (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,466::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,468::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 -n node.session.auth.username -v '****' --op=update (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,549::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,549::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 -n node.session.auth.password -v '****' --op=update (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,574::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,575::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m iface -I default (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,593::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::INFO::2016-02-09 15:37:43,593::iscsi::557::Storage.ISCSI::(setRpFilterIfNeeded) iSCSI iface.net_ifacename not provided. Skipping.
Thread-31::DEBUG::2016-02-09 15:37:43,594::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 -l (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,627::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,628::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.6f138bf1cc59 -I default -p 129.1.2.40:3260,1 -n node.startup -v manual --op=update (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,643::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,644::utils::755::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:43,689::utils::775::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:43,690::lvm::371::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-31::DEBUG::2016-02-09 15:37:43,690::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name (cwd None)
Thread-31::DEBUG::2016-02-09 15:37:44,126::lvm::291::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0
Thread-31::DEBUG::2016-02-09 15:37:44,127::lvm::416::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-31::DEBUG::2016-02-09 15:37:44,127::hsm::2416::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
Thread-31::DEBUG::2016-02-09 15:37:44,127::hsm::2472::Storage.HSM::(connectStorageServer) knownSDs: {}
Thread-31::INFO::2016-02-09 15:37:44,128::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'fe3cc82b-6668-4c17-b9ee-7db54557237e'}]}
Thread-31::DEBUG::2016-02-09 15:37:44,128::task::1191::Storage.TaskManager.Task::(prepare) Task=`3f74c7e3-eef2-4c3d-a1b9-b20c0432f3c0`::finished: {'statuslist': [{'status': 0, 'id': 'fe3cc82b-6668-4c17-b9ee-7db54557237e'}]}
Thread-31::DEBUG::2016-02-09 15:37:44,128::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f74c7e3-eef2-4c3d-a1b9-b20c0432f3c0`::moving from state preparing -> state finished
Thread-31::DEBUG::2016-02-09 15:37:44,128::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-31::DEBUG::2016-02-09 15:37:44,128::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-31::DEBUG::2016-02-09 15:37:44,129::task::993::Storage.TaskManager.Task::(_decref) Task=`3f74c7e3-eef2-4c3d-a1b9-b20c0432f3c0`::ref 0 aborting False
Thread-31::DEBUG::2016-02-09 15:37:44,129::__init__::514::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': 'fe3cc82b-6668-4c17-b9ee-7db54557237e'}]
Thread-32::DEBUG::2016-02-09 15:37:44,149::__init__::481::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connect' in bridge with {'masterVersion': 5, 'domainDict': {'708088a0-a641-4a7e-8732-55dde96c08dc': 'active'}, 'storagepoolID': '00000002-0002-0002-0002-000000000184', 'scsiKey': '00000002-0002-0002-0002-000000000184', 'masterSdUUID': '708088a0-a641-4a7e-8732-55dde96c08dc', 'hostID': 1}
Thread-32::DEBUG::2016-02-09 15:37:44,151::task::595::Storage.TaskManager.Task::(_updateState) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::moving from state init -> state preparing
Thread-32::INFO::2016-02-09 15:37:44,151::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='00000002-0002-0002-0002-000000000184', hostID=1, msdUUID='708088a0-a641-4a7e-8732-55dde96c08dc', masterVersion=5, domainsMap={'708088a0-a641-4a7e-8732-55dde96c08dc': 'active'}, options=None)
Thread-32::DEBUG::2016-02-09 15:37:44,152::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4bc724bb-f2d7-4828-9148-bcd93915b093`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1037' at 'connectStoragePool'
Thread-32::DEBUG::2016-02-09 15:37:44,152::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive'
Thread-32::DEBUG::2016-02-09 15:37:44,152::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user)
Thread-32::DEBUG::2016-02-09 15:37:44,153::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`4bc724bb-f2d7-4828-9148-bcd93915b093`::Granted request
Thread-32::DEBUG::2016-02-09 15:37:44,153::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-000000000184`ReqID=`34a3a8d9-5c94-4c35-a0a0-23d69c68fe92`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1085' at '_connectStoragePool'
Thread-32::DEBUG::2016-02-09 15:37:44,154::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-000000000184' for lock type 'exclusive'
Thread-32::DEBUG::2016-02-09 15:37:44,154::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-000000000184' is free. Now locking as 'exclusive' (1 active user)
Thread-32::DEBUG::2016-02-09 15:37:44,154::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-000000000184`ReqID=`34a3a8d9-5c94-4c35-a0a0-23d69c68fe92`::Granted request
Thread-32::DEBUG::2016-02-09 15:37:44,155::spbackends::485::Storage.StoragePoolMemoryBackend::(updateVersionAndDomains) updating domain version to 5 and domains map to {'708088a0-a641-4a7e-8732-55dde96c08dc': 'active'}
Thread-32::INFO::2016-02-09 15:37:44,155::spbackends::433::Storage.StoragePoolMemoryBackend::(setDomainsMap) new storage pool master version 5 and domains map {'708088a0-a641-4a7e-8732-55dde96c08dc': 'Active'}
Thread-32::INFO::2016-02-09 15:37:44,155::sp::133::Storage.StoragePool::(setBackend) updating pool 00000002-0002-0002-0002-000000000184 backend from type NoneType instance 0x32d5b721f0 to type StoragePoolMemoryBackend instance 0x7f1f1843f100
Thread-32::INFO::2016-02-09 15:37:44,156::sp::630::Storage.StoragePool::(connect) Connect host #1 to the storage pool 00000002-0002-0002-0002-000000000184 with master domain: 708088a0-a641-4a7e-8732-55dde96c08dc (ver = 5)
Thread-32::DEBUG::2016-02-09 15:37:44,156::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,156::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,157::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,157::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,157::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,158::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,158::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
Thread-32::DEBUG::2016-02-09 15:37:44,158::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-32::DEBUG::2016-02-09 15:37:44,158::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
Thread-32::DEBUG::2016-02-09 15:37:44,159::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-32::DEBUG::2016-02-09 15:37:44,159::iscsi::424::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-32::DEBUG::2016-02-09 15:37:44,160::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-32::DEBUG::2016-02-09 15:37:44,179::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-32::DEBUG::2016-02-09 15:37:44,179::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.hba.rescan)
Thread-32::DEBUG::2016-02-09 15:37:44,179::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-32::DEBUG::2016-02-09 15:37:44,180::hba::53::Storage.HBA::(rescan) Starting scan
Thread-32::DEBUG::2016-02-09 15:37:44,180::utils::755::Storage.HBA::(execCmd) /usr/bin/sudo -n /usr/libexec/vdsm/fc-scan (cwd None)
Thread-32::DEBUG::2016-02-09 15:37:44,212::hba::66::Storage.HBA::(rescan) Scan finished
Thread-32::DEBUG::2016-02-09 15:37:44,226::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-32::DEBUG::2016-02-09 15:37:44,226::multipath::131::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None)
Thread-32::DEBUG::2016-02-09 15:37:44,249::multipath::131::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-32::DEBUG::2016-02-09 15:37:44,250::utils::755::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None)
Thread-32::DEBUG::2016-02-09 15:37:44,272::utils::775::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-32::DEBUG::2016-02-09 15:37:44,272::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,273::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,273::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,273::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,273::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,273::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,274::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-32::ERROR::2016-02-09 15:37:44,274::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 708088a0-a641-4a7e-8732-55dde96c08dc
Thread-32::ERROR::2016-02-09 15:37:44,274::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 708088a0-a641-4a7e-8732-55dde96c08dc
Thread-32::DEBUG::2016-02-09 15:37:44,274::lvm::371::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-32::DEBUG::2016-02-09 15:37:44,276::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 708088a0-a641-4a7e-8732-55dde96c08dc (cwd None)
Thread-32::DEBUG::2016-02-09 15:37:44,476::lvm::291::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "708088a0-a641-4a7e-8732-55dde96c08dc" not found\n Cannot process volume group 708088a0-a641-4a7e-8732-55dde96c08dc\n'; <rc> = 5
Thread-32::WARNING::2016-02-09 15:37:44,477::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "708088a0-a641-4a7e-8732-55dde96c08dc" not found', ' Cannot process volume group 708088a0-a641-4a7e-8732-55dde96c08dc']
Thread-32::DEBUG::2016-02-09 15:37:44,477::lvm::416::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-34::DEBUG::2016-02-09 15:37:44,488::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Thread-35::DEBUG::2016-02-09 15:37:44,497::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Thread-36::DEBUG::2016-02-09 15:37:44,506::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Thread-38::DEBUG::2016-02-09 15:37:44,518::__init__::298::IOProcessClient::(_run) Starting IOProcess...
Thread-32::ERROR::2016-02-09 15:37:44,533::sdc::143::Storage.StorageDomainCache::(_findDomain) domain 708088a0-a641-4a7e-8732-55dde96c08dc not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: ('708088a0-a641-4a7e-8732-55dde96c08dc',)
Thread-32::DEBUG::2016-02-09 15:37:44,534::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-000000000184'
Thread-32::DEBUG::2016-02-09 15:37:44,534::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-000000000184' (0 active users)
Thread-32::DEBUG::2016-02-09 15:37:44,534::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-000000000184' is free, finding out if anyone is waiting for it.
Thread-32::DEBUG::2016-02-09 15:37:44,534::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-000000000184', Clearing records.
Thread-32::DEBUG::2016-02-09 15:37:44,535::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-32::DEBUG::2016-02-09 15:37:44,535::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-32::DEBUG::2016-02-09 15:37:44,535::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-32::DEBUG::2016-02-09 15:37:44,535::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-32::ERROR::2016-02-09 15:37:44,535::task::866::Storage.TaskManager.Task::(_setError) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1039, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1104, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 637, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1179, in __rebuild
self.setMasterDomain(msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1390, in setMasterDomain
raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'
Thread-32::DEBUG::2016-02-09 15:37:44,536::task::885::Storage.TaskManager.Task::(_run) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::Task._run: 667f53ec-857e-49a2-b3a6-da9f575262bc ('00000002-0002-0002-0002-000000000184', 1, '708088a0-a641-4a7e-8732-55dde96c08dc', 5, {'708088a0-a641-4a7e-8732-55dde96c08dc': 'active'}) {} failed - stopping task
Thread-32::DEBUG::2016-02-09 15:37:44,536::task::1217::Storage.TaskManager.Task::(stop) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::stopping in state preparing (force False)
Thread-32::DEBUG::2016-02-09 15:37:44,536::task::993::Storage.TaskManager.Task::(_decref) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::ref 1 aborting True
Thread-32::INFO::2016-02-09 15:37:44,536::task::1171::Storage.TaskManager.Task::(prepare) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-32::DEBUG::2016-02-09 15:37:44,536::task::1176::Storage.TaskManager.Task::(prepare) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::Prepare: aborted: Cannot find master domain
Thread-32::DEBUG::2016-02-09 15:37:44,536::task::993::Storage.TaskManager.Task::(_decref) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::ref 0 aborting True
Thread-32::DEBUG::2016-02-09 15:37:44,537::task::928::Storage.TaskManager.Task::(_doAbort) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::Task._doAbort: force False
Thread-32::DEBUG::2016-02-09 15:37:44,537::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::DEBUG::2016-02-09 15:37:44,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::moving from state preparing -> state aborting
Thread-32::DEBUG::2016-02-09 15:37:44,537::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::_aborting: recover policy none
Thread-32::DEBUG::2016-02-09 15:37:44,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`667f53ec-857e-49a2-b3a6-da9f575262bc`::moving from state aborting -> state failed
Thread-32::DEBUG::2016-02-09 15:37:44,537::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-32::DEBUG::2016-02-09 15:37:44,537::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-32::ERROR::2016-02-09 15:37:44,538::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: 'spUUID=00000002-0002-0002-0002-000000000184, msdUUID=708088a0-a641-4a7e-8732-55dde96c08dc'", 'code': 304}}
Thread-39::DEBUG::2016-02-09 15:37:44,545::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5fb0ad-3542-4d66-809d-f312d6552ef2`::moving from state init -> state preparing
Thread-39::INFO::2016-02-09 15:37:44,545::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)
Thread-39::INFO::2016-02-09 15:37:44,546::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}
Thread-39::DEBUG::2016-02-09 15:37:44,546::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d5fb0ad-3542-4d66-809d-f312d6552ef2`::finished: {}
Thread-39::DEBUG::2016-02-09 15:37:44,546::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5fb0ad-3542-4d66-809d-f312d6552ef2`::moving from state preparing -> state finished
Thread-39::DEBUG::2016-02-09 15:37:44,546::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-39::DEBUG::2016-02-09 15:37:44,546::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-39::DEBUG::2016-02-09 15:37:44,546::task::993::Storage.TaskManager.Task::(_decref) Task=`6d5fb0ad-3542-4d66-809d-f312d6552ef2`::ref 0 aborting False
Thread-39::ERROR::2016-02-09 15:37:44,548::__init__::506::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/yajsonrpc/__init__.py", line 501, in _serveRequest
res = method(**params)
File "/usr/share/vdsm/rpc/Bridge.py", line 267, in _dynamicMethod
result = fn(*methodArgs)
File "/usr/share/vdsm/API.py", line 1330, in getStats
stats.update(self._cif.mom.getKsmStats())
File "/usr/share/vdsm/momIF.py", line 60, in getKsmStats
stats = self._mom.getStatistics()['host']
File "/usr/lib/python2.6/site-packages/mom/MOMFuncs.py", line 75, in getStatistics
host_stats = self.threads['host_monitor'].interrogate().statistics[-1]
AttributeError: 'NoneType' object has no attribute 'statistics'
_______________________________________________
Users mailing list
[email protected]
http://lists.ovirt.org/mailman/listinfo/users
