> On Apr 21, 2017, at 6:38 AM, knarra <kna...@redhat.com> wrote:
>
> On 04/21/2017 06:34 PM, Jamie Lawrence wrote:
>>> On Apr 20, 2017, at 10:36 PM, knarra <kna...@redhat.com> wrote:
>>>> The installer claimed it did, but I believe it didn’t. Below the error
>>>> from my original email, there’s the below (apologies for not including it
>>>> earlier; I missed it). Note: 04ff4cf1-135a-4918-9a1f-8023322f89a3 is the
>>>> HE - I’m pretty sure it is complaining about itself. (In any case, I
>>>> verified that there are no other VMs running with both virsh and
>>>> vdsClient.)
>> ^^^
>>
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:128 Stage
>>>> late_setup METHOD otopi.plugins.gr_he_setup.vm.runvm.Plugin._late_setup
>>>> 2017-04-19 12:27:02 DEBUG otopi.plugins.gr_he_setup.vm.runvm
>>>> runvm._late_setup:83 {'status': {'message': 'Done', 'code': 0}, 'items':
>>>> [u'04ff4cf1-135a-4918-9a1f-8023322f89a3']}
>>>> 2017-04-19 12:27:02 ERROR otopi.plugins.gr_he_setup.vm.runvm
>>>> runvm._late_setup:91 The following VMs have been found:
>>>> 04ff4cf1-135a-4918-9a1f-8023322f89a3
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context._executeMethod:142 method
>>>> exception
>>>> Traceback (most recent call last):
>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in
>>>> _executeMethod
>>>> method['method']()
>>>> File
>>>> "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/vm/runvm.py",
>>>> line 95, in _late_setup
>>>> _('Cannot setup Hosted Engine with other VMs running')
>>>> RuntimeError: Cannot setup Hosted Engine with other VMs running
>>>> 2017-04-19 12:27:02 ERROR otopi.context context._executeMethod:151 Failed
>>>> to execute stage 'Environment setup': Cannot setup Hosted Engine with
>>>> other VMs running
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:760
>>>> ENVIRONMENT DUMP - BEGIN
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV
>>>> BASE/error=bool:'True'
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:770 ENV
>>>> BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>,
>>>> RuntimeError('Cannot setup Hosted Engine with other VMs running',),
>>>> <traceback object at 0x4358098>)]'
>>>> 2017-04-19 12:27:02 DEBUG otopi.context context.dumpEnvironment:774
>>>> ENVIRONMENT DUMP - END
>>> James, generally this issue happens when the setup failed once and you
>>> tried re running it again. Can you clean it and deploy it again? HE
>>> should come up successfully. Below are the steps for cleaning it up.
>> Knarra,
>>
>> I realize that. However, that is not the situation in my case. See above, at
>> the mark - the UUID it is complaining about is the UUID of the hosted-engine
>> it just installed. From the answers file generated from the run (whole thing
>> below):
>>
>>>>>> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
>> Also see the WARNs I mentioned previously, quoted below. Excerpt:
>>
>>>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
>>>>>> File:
>>>>>> /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm
>>>>>> already removed
>>>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
>>>>>> File:
>>>>>> /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0
>>>>>> already removed
>>>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
>>>>>> to broker, the number of errors has exceeded the limit (1)
>> I’m not clear on what it is attempting to do there, but it seems relevant.
> I remember that you said HE vm was not started when the installation was
> successful. Is Local Maintenance enabled on that host?
>
> can you please check if the services 'ovirt-ha-agent' and 'ovirt-ha-broker'
> running fine and try to restart them once ?
Agent and broker logs from before are down in the original message quoting.
They’re running, but not fine.
[root@sc5-ovirt-2 jlawrence]# ps ax|grep ha-
130599 ? Ssl 3:52 /usr/bin/python
/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker --no-daemon
132869 ? Ss 0:13 /usr/bin/python
/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon
133501 pts/0 S+ 0:00 grep --color=auto ha-
[root@sc5-ovirt-2 jlawrence]# systemctl restart ovirt-ha-agent ovirt-ha-broker
[root@sc5-ovirt-2 jlawrence]# tail -40
/var/log/ovirt-hosted-engine-ha/broker.log
Thread-46::INFO::2017-04-21
10:52:57,058::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
Cleaning up stale LV link
'/rhev/data-center/mnt/glusterSD/sc5-gluster-1.squaretrade.com:_ovirt__engine/a1155699-0bcf-44c5-aa55-a574ca3ad313/ha_agent/hosted-engine.metadata'
Thread-53::INFO::2017-04-21
10:52:57,070::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-50::INFO::2017-04-21
10:52:57,118::mem_free::50::mem_free.MemFree::(action) memFree: 255148
Thread-54::INFO::2017-04-21
10:53:00,408::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-54::INFO::2017-04-21
10:53:00,421::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-55::INFO::2017-04-21
10:53:02,158::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-55::INFO::2017-04-21
10:53:02,171::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-38::INFO::2017-04-21
10:53:02,179::mem_free::50::mem_free.MemFree::(action) memFree: 255153
Thread-35::INFO::2017-04-21
10:53:04,354::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-35::INFO::2017-04-21
10:53:04,439::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopping submonitor ping, id 35329808
Thread-35::INFO::2017-04-21
10:53:04,441::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopped submonitor ping, id 35329808
Thread-35::INFO::2017-04-21
10:53:04,452::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopping submonitor mgmt-bridge, id 35328400
Thread-35::INFO::2017-04-21
10:53:04,485::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopped submonitor mgmt-bridge, id 35328400
Thread-35::INFO::2017-04-21
10:53:04,507::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopping submonitor mem-free, id 35328592
Thread-35::INFO::2017-04-21
10:53:04,555::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopped submonitor mem-free, id 35328592
Thread-35::INFO::2017-04-21
10:53:04,659::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopping submonitor cpu-load-no-engine, id 35330000
Thread-35::INFO::2017-04-21
10:53:04,659::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopped submonitor cpu-load-no-engine, id 35330000
Thread-35::INFO::2017-04-21
10:53:04,676::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopping submonitor engine-health, id 35330640
Thread-35::INFO::2017-04-21
10:53:04,746::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
Stopped submonitor engine-health, id 35330640
Thread-56::INFO::2017-04-21
10:53:05,322::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-56::INFO::2017-04-21
10:53:05,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-50::INFO::2017-04-21
10:53:05,342::mem_free::50::mem_free.MemFree::(action) memFree: 255154
Thread-51::INFO::2017-04-21
10:53:09,325::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load)
System load total=0.0112, engine=0.0000, non-engine=0.0112
Thread-57::INFO::2017-04-21
10:53:14,958::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-57::INFO::2017-04-21
10:53:14,971::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
Connection closed
Thread-50::INFO::2017-04-21
10:53:14,978::mem_free::50::mem_free.MemFree::(action) memFree: 255156
Thread-58::INFO::2017-04-21
10:53:15,243::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-58::INFO::2017-04-21
10:53:15,243::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Starting submonitor ping
Thread-58::INFO::2017-04-21
10:53:15,244::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Started submonitor ping, id 140605556704272
Thread-58::INFO::2017-04-21
10:53:15,255::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Starting submonitor mgmt-bridge
Thread-58::INFO::2017-04-21
10:53:15,257::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Started submonitor mgmt-bridge, id 140605556704400
Thread-58::INFO::2017-04-21
10:53:15,259::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Starting submonitor mem-free
Thread-58::INFO::2017-04-21
10:53:15,260::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Started submonitor mem-free, id 140605556704336
Thread-58::INFO::2017-04-21
10:53:15,261::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Starting submonitor cpu-load-no-engine
Thread-58::INFO::2017-04-21
10:53:15,262::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Started submonitor cpu-load-no-engine, id 140605556705872
Thread-58::INFO::2017-04-21
10:53:15,264::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Starting submonitor engine-health
Thread-58::INFO::2017-04-21
10:53:15,265::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
Started submonitor engine-health, id 140605556706640
Thread-64::INFO::2017-04-21
10:53:15,413::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-65::INFO::2017-04-21
10:53:19,843::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
Connection established
Thread-58::INFO::2017-04-21
10:53:19,913::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
Cleaning up stale LV link
'/rhev/data-center/mnt/glusterSD/sc5-gluster-1.squaretrade.com:_ovirt__engine/a1155699-0bcf-44c5-aa55-a574ca3ad313/ha_agent/hosted-engine.lockspace'
[root@sc5-ovirt-2 jlawrence]# tail -40 /var/log/ovirt-hosted-engine-ha/agent.log
MainThread::INFO::2017-04-21
10:53:15,242::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
Initializing ha-broker connection
MainThread::INFO::2017-04-21
10:53:15,242::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor ping, options {'addr': '10.181.26.1'}
MainThread::INFO::2017-04-21
10:53:15,254::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605556704272
MainThread::INFO::2017-04-21
10:53:15,254::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name':
'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-04-21
10:53:15,258::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605556704400
MainThread::INFO::2017-04-21
10:53:15,258::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-04-21
10:53:15,260::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605556704336
MainThread::INFO::2017-04-21
10:53:15,260::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid':
'583a3cfa-a4f8-4920-88d8-a5fb12f21519', 'address': '0'}
MainThread::INFO::2017-04-21
10:53:15,263::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605556705872
MainThread::INFO::2017-04-21
10:53:15,263::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid':
'583a3cfa-a4f8-4920-88d8-a5fb12f21519', 'address': '0'}
MainThread::INFO::2017-04-21
10:53:15,265::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605556706640
MainThread::INFO::2017-04-21
10:53:19,990::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain)
Success, id 140605556914448
MainThread::INFO::2017-04-21
10:53:19,991::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
Broker initialized, all submonitors started
MainThread::ERROR::2017-04-21
10:53:20,010::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Service sanlock is not running and the admin is responsible for starting it.
Waiting...
MainThread::WARNING::2017-04-21
10:53:25,016::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Restarting agent, attempt '5'
MainThread::INFO::2017-04-21
10:53:25,045::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
Found certificate common name: sc5-ovirt-2.squaretrade.com
MainThread::INFO::2017-04-21
10:53:25,047::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
MainThread::INFO::2017-04-21
10:53:27,516::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
MainThread::INFO::2017-04-21
10:53:27,517::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2017-04-21
10:53:32,280::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2017-04-21
10:53:32,296::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain
MainThread::INFO::2017-04-21
10:53:32,513::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images
MainThread::INFO::2017-04-21
10:53:32,513::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images
MainThread::INFO::2017-04-21
10:53:35,304::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Refreshing vm.conf
MainThread::INFO::2017-04-21
10:53:35,305::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-04-21
10:53:35,305::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::WARNING::2017-04-21
10:55:10,904::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Unable to find OVF_STORE
MainThread::ERROR::2017-04-21
10:55:10,906::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
Unable to identify the OVF_STORE volume, falling back to initial vm.conf.
Please ensure you already added your first data domain for regular VMs
MainThread::INFO::2017-04-21
10:55:10,930::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
Initializing ha-broker connection
MainThread::INFO::2017-04-21
10:55:10,931::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor ping, options {'addr': '10.181.26.1'}
MainThread::INFO::2017-04-21
10:55:10,942::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605093437904
MainThread::INFO::2017-04-21
10:55:10,943::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name':
'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-04-21
10:55:10,945::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605093438608
MainThread::INFO::2017-04-21
10:55:10,945::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-04-21
10:55:10,947::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605093438544
MainThread::INFO::2017-04-21
10:55:10,947::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid':
'583a3cfa-a4f8-4920-88d8-a5fb12f21519', 'address': '0'}
MainThread::INFO::2017-04-21
10:55:10,949::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605093439888
MainThread::INFO::2017-04-21
10:55:10,949::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid':
'583a3cfa-a4f8-4920-88d8-a5fb12f21519', 'address': '0'}
MainThread::INFO::2017-04-21
10:55:10,951::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
Success, id 140605093440528
>>
>> I know there is no failed install left on the gluster volume, because when I
>> attempt an install, part of my scripted prep process is deleting and
>> recreating the Gluster volume. The below instructions are more or less what
>> I’m doing already in a script[1]. (the gluster portion of the script process
>> is: stop the volume, delete the volume, remove the mount point directory to
>> avoid Gluster’s xattr problem with recycling directories, recreate the
>> directory, change perms, create the volume, start the volume, set
>> Ovirt-recc’ed volume options.)
>>
>> -j
>>
>> [1] We have a requirement for automated setup of all production resources,
>> so all of this ends up being scripted.
>>
>>> 1) vdsClient -s 0 list table | awk '{print $1}' | xargs vdsClient -s 0
>>> destroy
>>>
>>> 2) stop the volume and delete all the information inside the bricks from
>>> all the hosts
>>>
>>> 3) try to umount storage from /rhev/data-center/mnt/ - umount -f
>>> /rhev/data-center/mnt/ if it is mounted
>>>
>>> 4) remove all dirs from /rhev/data-center/mnt/ - rm -rf
>>> /rhev/data-center/mnt/*
>>>
>>> 5) start volume again and start the deployment.
>>>
>>> Thanks
>>> kasturi
>>>>
>>>>>> If I start it manually, the default DC is down, the default cluster has
>>>>>> the installation host in the cluster, there is no storage, and the VM
>>>>>> doesn’t show up in the GUI. In this install run, I have not yet started
>>>>>> the engine manually.
>>>>> you wont be seeing HE vm until HE storage is imported into the UI. HE
>>>>> storage will be automatically imported into the UI (which will import HE
>>>>> vm too )once a master domain is present .
>>>> Sure; I’m just attempting to provide context.
>>>>
>>>>>> I assume this is related to the errors in ovirt-hosted-engine-setup.log,
>>>>>> below. (The timestamps are confusing; it looks like the Python errors
>>>>>> are logged some time after they’re captured or something.) The HA broker
>>>>>> and agent logs just show them looping in the sequence below.
>>>>>>
>>>>>> Is there a decent way to pick this up and continue? If not, how do I
>>>>>> make this work?
>>>>> Can you please check the following things.
>>>>>
>>>>> 1) is glusterd running on all the nodes ? 'systemctl status glistered’
>>>>> 2) Are you able to connect to your storage server which is ovirt_engine
>>>>> in your case.
>>>>> 3) Can you check if all the brick process in the volume is up ?
>>>> 1) Verified that glusterd is running on all three nodes.
>>>>
>>>> 2)
>>>> [root@sc5-thing-1]# mount -tglusterfs sc5-gluster-1:/ovirt_engine
>>>> /mnt/ovirt_engine
>>>> [root@sc5-thing-1]# df -h
>>>> Filesystem Size Used Avail Use% Mounted on
>>>> […]
>>>> sc5-gluster-1:/ovirt_engine 300G 2.6G 298G 1% /mnt/ovirt_engine
>>>>
>>>>
>>>> 3)
>>>> [root@sc5-gluster-1 jlawrence]# gluster volume status
>>>> Status of volume: ovirt_engine
>>>> Gluster process TCP Port RDMA Port Online
>>>> Pid
>>>> ------------------------------------------------------------------------------
>>>> Brick sc5-gluster-1:/gluster-bricks/ovirt_e
>>>> ngine/ovirt_engine-1 49217 0 Y
>>>> 22102
>>>> Brick sc5-gluster-2:/gluster-bricks/ovirt_e
>>>> ngine/ovirt_engine-1 49157 0 Y
>>>> 37842
>>>> Brick sc5-gluster-3:/gluster-bricks/ovirt_e
>>>> ngine/ovirt_engine-1 49157 0 Y
>>>> 112018
>>>> Self-heal Daemon on localhost N/A N/A Y
>>>> 113364
>>>> Self-heal Daemon on sc5-gluster-2.squaretra
>>>> de.com N/A N/A Y
>>>> 19344
>>>> Self-heal Daemon on sc5-gluster-3.squaretra
>>>> de.com N/A N/A Y
>>>> 37956
>>>>
>>>> Task Status of Volume ovirt_engine
>>>> ------------------------------------------------------------------------------
>>>> There are no active volume tasks
>>>>
>>>> Volume ovirt_images is not started
>>>>
>>>>
>>>>
>>>>
>>>>> Thanks
>>>>> kasturi.
>>>> Thank you!
>>>>
>>>> -j
>>>>
>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> -j
>>>>>>
>>>>>> - - - - ovirt-hosted-engine-setup.log snippet: - - - -
>>>>>>
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.context context._executeMethod:128 Stage
>>>>>> late_setup METHOD
>>>>>> otopi.plugins.gr_he_setup.system.vdsmenv.Plugin._late_setup
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
>>>>>> systemd.status:90 check service vdsmd status
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
>>>>>> plugin.executeRaw:813 execute: ('/bin/systemctl', 'status',
>>>>>> 'vdsmd.service'), executable='None', cwd='None', env=None
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
>>>>>> plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'status',
>>>>>> 'vdsmd.service'), rc=0
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
>>>>>> plugin.execute:921 execute-output: ('/bin/systemctl', 'status',
>>>>>> 'vdsmd.service') stdout:
>>>>>> ● vdsmd.service - Virtual Desktop Server Manager
>>>>>> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled;
>>>>>> vendor preset: enabled)
>>>>>> Active: active (running) since Wed 2017-04-19 12:26:59 PDT; 2min 55s
>>>>>> ago
>>>>>> Process: 67370 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
>>>>>> --post-stop (code=exited, status=0/SUCCESS)
>>>>>> Process: 69995 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
>>>>>> --pre-start (code=exited, status=0/SUCCESS)
>>>>>> Main PID: 70062 (vdsm)
>>>>>> CGroup: /system.slice/vdsmd.service
>>>>>> └─70062 /usr/bin/python2 /usr/share/vdsm/vdsm
>>>>>>
>>>>>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
>>>>>> to broker, the number of errors has exceeded the limit (1)
>>>>>> Apr 19 12:29:00 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
>>>>>> failed to retrieve Hosted Engine HA info
>>>>>> Traceback (most
>>>>>> recent call last):
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
>>>>>> _getHaInfo
>>>>>> stats =
>>>>>> instance.get_all_stats()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>>>>> line 102, in get_all_stats
>>>>>> with
>>>>>> broker.connection(self._retries, self._wait):
>>>>>> File
>>>>>> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>>>>>> return
>>>>>> self.gen.next()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 99, in connection
>>>>>>
>>>>>> self.connect(retries, wait)
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 78, in connect
>>>>>> raise
>>>>>> BrokerConnectionError(error_msg)
>>>>>>
>>>>>> BrokerConnectionError: Failed to connect to broker, the number of errors
>>>>>> has exceeded the limit (1)
>>>>>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
>>>>>> to broker, the number of errors has exceeded the limit (1)
>>>>>> Apr 19 12:29:15 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
>>>>>> failed to retrieve Hosted Engine HA info
>>>>>> Traceback (most
>>>>>> recent call last):
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
>>>>>> _getHaInfo
>>>>>> stats =
>>>>>> instance.get_all_stats()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>>>>> line 102, in get_all_stats
>>>>>> with
>>>>>> broker.connection(self._retries, self._wait):
>>>>>> File
>>>>>> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>>>>>> return
>>>>>> self.gen.next()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 99, in connection
>>>>>>
>>>>>> self.connect(retries, wait)
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 78, in connect
>>>>>> raise
>>>>>> BrokerConnectionError(error_msg)
>>>>>>
>>>>>> BrokerConnectionError: Failed to connect to broker, the number of errors
>>>>>> has exceeded the limit (1)
>>>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
>>>>>> File:
>>>>>> /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.com.redhat.rhevm.vdsm
>>>>>> already removed
>>>>>> Apr 19 12:29:20 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root WARN
>>>>>> File:
>>>>>> /var/lib/libvirt/qemu/channels/04ff4cf1-135a-4918-9a1f-8023322f89a3.org.qemu.guest_agent.0
>>>>>> already removed
>>>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
>>>>>> to broker, the number of errors has exceeded the limit (1)
>>>>>> Apr 19 12:29:30 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
>>>>>> failed to retrieve Hosted Engine HA info
>>>>>> Traceback (most
>>>>>> recent call last):
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
>>>>>> _getHaInfo
>>>>>> stats =
>>>>>> instance.get_all_stats()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>>>>> line 102, in get_all_stats
>>>>>> with
>>>>>> broker.connection(self._retries, self._wait):
>>>>>> File
>>>>>> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>>>>>> return
>>>>>> self.gen.next()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 99, in connection
>>>>>>
>>>>>> self.connect(retries, wait)
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 78, in connect
>>>>>> raise
>>>>>> BrokerConnectionError(error_msg)
>>>>>>
>>>>>> BrokerConnectionError: Failed to connect to broker, the number of errors
>>>>>> has exceeded the limit (1)
>>>>>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm
>>>>>> ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink ERROR Failed to connect
>>>>>> to broker, the number of errors has exceeded the limit (1)
>>>>>> Apr 19 12:29:45 sc5-ovirt-2.squaretrade.com vdsm[70062]: vdsm root ERROR
>>>>>> failed to retrieve Hosted Engine HA info
>>>>>> Traceback (most
>>>>>> recent call last):
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in
>>>>>> _getHaInfo
>>>>>> stats =
>>>>>> instance.get_all_stats()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py",
>>>>>> line 102, in get_all_stats
>>>>>> with
>>>>>> broker.connection(self._retries, self._wait):
>>>>>> File
>>>>>> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
>>>>>> return
>>>>>> self.gen.next()
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 99, in connection
>>>>>>
>>>>>> self.connect(retries, wait)
>>>>>> File
>>>>>> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 78, in connect
>>>>>> raise
>>>>>> BrokerConnectionError(error_msg)
>>>>>>
>>>>>> BrokerConnectionError: Failed to connect to broker, the number of errors
>>>>>> has exceeded the limit (1)
>>>>>>
>>>>>> 2017-04-19 12:29:55 DEBUG otopi.plugins.otopi.services.systemd
>>>>>> plugin.execute:926 execute-output: ('/bin/systemctl', 'status',
>>>>>> 'vdsmd.service') stderr:
>>>>>>
>>>>>>
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.system.vdsmenv
>>>>>> util.connect_vdsm_json_rpc:209 {'status': {'message': 'Done', 'code':
>>>>>> 0}, u'systemProductName': u'PowerEdge R630', u'systemSerialNumber':
>>>>>> u'GTH5942', u'systemFamily': u'PowerEdge', u'systemUUID':
>>>>>> u'4C4C4544-0054-4810-8035-C7C04F393432', u'systemManufacturer': u'Dell
>>>>>> Inc.'}
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:760
>>>>>> ENVIRONMENT DUMP - BEGIN
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:770 ENV
>>>>>> OVEHOSTED_VDSM/vdscli=_Server:'<vdsm.jsonrpcvdscli._Server object at
>>>>>> 0x34a8290>'
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context.dumpEnvironment:774
>>>>>> ENVIRONMENT DUMP - END
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage
>>>>>> late_setup METHOD otopi.plugins.gr_he_common.vm.misc.Plugin._late_setup
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:134
>>>>>> condition False
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.context context._executeMethod:128 Stage
>>>>>> late_setup METHOD
>>>>>> otopi.plugins.gr_he_setup.pki.vdsmpki.Plugin._late_setup
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
>>>>>> plugin.executeRaw:813 execute: ('/bin/openssl', 'x509', '-noout',
>>>>>> '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'),
>>>>>> executable='None', cwd='None', env=None
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
>>>>>> plugin.executeRaw:863 execute-result: ('/bin/openssl', 'x509', '-noout',
>>>>>> '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem'), rc=0
>>>>>> 2017-04-19 12:29:57 DEBUG otopi.plugins.gr_he_setup.pki.vdsmpki
>>>>>> plugin.execute:921 execute-output: ('/bin/openssl', 'x509', '-noout',
>>>>>> '-text', '-in', '/etc/pki/vdsm/libvirt-spice/server-cert.pem') stdout:
>>>>>>
>>>>>> - - - - broker.log: - - - -
>>>>>>
>>>>>> Thread-2970::INFO::2017-04-19
>>>>>> 14:29:51,591::cpu_load_no_engine::113::cpu_load_no_engine.EngineHealth::(calculate_load)
>>>>>> System load total=0.0117, engine=0.0000, non-engine=0.0117
>>>>>> Thread-2976::INFO::2017-04-19
>>>>>> 14:29:56,347::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2976::INFO::2017-04-19
>>>>>> 14:29:56,361::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,628::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,629::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Starting submonitor ping
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,631::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Started submonitor ping, id 140603614328464
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,662::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Starting submonitor mgmt-bridge
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,667::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Started submonitor mgmt-bridge, id 140603614326992
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,688::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Starting submonitor mem-free
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,735::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Started submonitor mem-free, id 140604548742096
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,756::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Starting submonitor cpu-load-no-engine
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,798::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Started submonitor cpu-load-no-engine, id 43463568
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,841::monitor::66::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Starting submonitor engine-health
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:29:57,863::monitor::79::ovirt_hosted_engine_ha.broker.monitor.Monitor::(start_submonitor)
>>>>>> Started submonitor engine-health, id 140604881915728
>>>>>> Thread-2983::INFO::2017-04-19
>>>>>> 14:29:58,153::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2984::INFO::2017-04-19
>>>>>> 14:30:02,179::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:30:02,249::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
>>>>>> Cleaning up stale LV link
>>>>>> '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.lockspace'
>>>>>> Thread-2977::INFO::2017-04-19
>>>>>> 14:30:02,321::storage_backends::119::ovirt_hosted_engine_ha.lib.storage_backends::(_check_symlinks)
>>>>>> Cleaning up stale LV link
>>>>>> '/rhev/data-center/mnt/glusterSD/sc5-gluster-1:_ovirt__engine/a81b229c-2422-4110-af45-1bc4d9a01bab/ha_agent/hosted-engine.metadata'
>>>>>> Thread-2983::INFO::2017-04-19
>>>>>> 14:30:02,334::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2984::INFO::2017-04-19
>>>>>> 14:30:02,340::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2969::INFO::2017-04-19
>>>>>> 14:30:02,343::mem_free::50::mem_free.MemFree::(action) memFree: 254533
>>>>>> Thread-2980::INFO::2017-04-19
>>>>>> 14:30:02,382::mem_free::50::mem_free.MemFree::(action) memFree: 254530
>>>>>> Thread-2985::INFO::2017-04-19
>>>>>> 14:30:07,991::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2985::INFO::2017-04-19
>>>>>> 14:30:08,082::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2969::INFO::2017-04-19
>>>>>> 14:30:08,129::mem_free::50::mem_free.MemFree::(action) memFree: 254534
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,719::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopping submonitor ping, id 140603546975120
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopped submonitor ping, id 140603546975120
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,720::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopping submonitor mgmt-bridge, id 140603546973136
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,720::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopped submonitor mgmt-bridge, id 140603546973136
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopping submonitor mem-free, id 140603546973264
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopped submonitor mem-free, id 140603546973264
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopping submonitor cpu-load-no-engine, id 140603546976144
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopped submonitor cpu-load-no-engine, id 140603546976144
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::90::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopping submonitor engine-health, id 140603546974096
>>>>>> Thread-2966::INFO::2017-04-19
>>>>>> 14:30:09,721::monitor::99::ovirt_hosted_engine_ha.broker.monitor.Monitor::(stop_submonitor)
>>>>>> Stopped submonitor engine-health, id 140603546974096
>>>>>> Thread-2979::INFO::2017-04-19
>>>>>> 14:30:11,145::mgmt_bridge::55::mgmt_bridge.MgmtBridge::(action) Found
>>>>>> bridge ovirtmgmt with ports
>>>>>> Thread-2986::INFO::2017-04-19
>>>>>> 14:30:11,462::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2986::INFO::2017-04-19
>>>>>> 14:30:11,480::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2987::INFO::2017-04-19
>>>>>> 14:30:11,481::listener::134::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(setup)
>>>>>> Connection established
>>>>>> Thread-2987::INFO::2017-04-19
>>>>>> 14:30:11,488::listener::186::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle)
>>>>>> Connection closed
>>>>>> Thread-2980::INFO::2017-04-19
>>>>>> 14:30:11,502::mem_free::50::mem_free.MemFree::(action) memFree: 254535
>>>>>>
>>>>>> - - - - ha agent.log: - - - -
>>>>>>
>>>>>> MainThread::WARNING::2017-04-19
>>>>>> 14:31:16,352::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>>>>>> Restarting agent, attempt '6'
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:16,383::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
>>>>>> Found certificate common name: sc5-ovirt-2.squaretrade.com
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:16,385::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
>>>>>> Initializing VDSM
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:18,863::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>>>>>> Connecting the storage
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:18,864::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>>>>>> Connecting storage server
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:23,672::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>>>>>> Connecting storage server
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:23,689::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>>>>>> Refreshing the storage domain
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:23,956::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>>>>>> Preparing images
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:23,957::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
>>>>>> Preparing images
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:26,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>>>>>> Refreshing vm.conf
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:26,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf)
>>>>>> Reloading vm.conf from the shared storage domain
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:26,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>>>>> Trying to get a fresher copy of vm configuration from the OVF_STORE
>>>>>> MainThread::WARNING::2017-04-19
>>>>>> 14:31:29,548::ovf_store::107::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
>>>>>> Unable to find OVF_STORE
>>>>>> MainThread::ERROR::2017-04-19
>>>>>> 14:31:29,551::config::450::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>>>>>> Unable to identify the OVF_STORE volume, falling back to initial
>>>>>> vm.conf. Please ensure you already added your first data domain for
>>>>>> regular VMs
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,576::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
>>>>>> Initializing ha-broker connection
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,577::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Starting monitor ping, options {'addr': '10.181.26.1'}
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,596::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Success, id 140604881712208
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,596::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Starting monitor mgmt-bridge, options {'use_ssl': 'true',
>>>>>> 'bridge_name': 'ovirtmgmt', 'address': '0'}
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,600::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Success, id 140604881712016
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,600::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Success, id 140604881711568
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Starting monitor cpu-load-no-engine, options {'use_ssl': 'true',
>>>>>> 'vm_uuid': '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,605::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Success, id 140604881711312
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,605::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid':
>>>>>> '04ff4cf1-135a-4918-9a1f-8023322f89a3', 'address': '0'}
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:29,608::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
>>>>>> Success, id 140604881714896
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:34,326::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain)
>>>>>> Success, id 140604881713168
>>>>>> MainThread::INFO::2017-04-19
>>>>>> 14:31:34,327::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
>>>>>> Broker initialized, all submonitors started
>>>>>> MainThread::ERROR::2017-04-19
>>>>>> 14:31:34,372::agent::196::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>>>>>> Service sanlock is not running and the admin is responsible for
>>>>>> starting it. Waiting…
>>>>>>
>>>>>>
>>>>>> - - - - answers.conf - - - -
>>>>>>
>>>>>> [environment:default]
>>>>>> OVEHOSTED_CORE/rollbackProceed=none:None
>>>>>> OVEHOSTED_CORE/screenProceed=none:None
>>>>>> OVEHOSTED_CORE/deployProceed=bool:True
>>>>>> OVEHOSTED_CORE/upgradeProceed=none:None
>>>>>> OVEHOSTED_CORE/confirmSettings=bool:True
>>>>>> OVEHOSTED_NETWORK/fqdn=str:sc5-ovirt-engine-1.squaretrade.com
>>>>>> OVEHOSTED_NETWORK/bridgeName=str:ovirtmgmt
>>>>>> OVEHOSTED_NETWORK/firewallManager=none:None
>>>>>> OVEHOSTED_NETWORK/gateway=str:10.181.26.1
>>>>>> OVEHOSTED_ENGINE/insecureSSL=none:None
>>>>>> OVEHOSTED_ENGINE/clusterName=str:Default
>>>>>> OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
>>>>>> OVEHOSTED_STORAGE/domainType=str:glusterfs
>>>>>> OVEHOSTED_STORAGE/connectionUUID=str:c2dd62cb-79aa-4547-b191-b74e55fe5311
>>>>>> OVEHOSTED_STORAGE/LunID=none:None
>>>>>> OVEHOSTED_STORAGE/imgSizeGB=str:50
>>>>>> OVEHOSTED_STORAGE/mntOptions=none:None
>>>>>> OVEHOSTED_STORAGE/iSCSIPortalIPAddress=none:None
>>>>>> OVEHOSTED_STORAGE/metadataVolumeUUID=str:ac9d7fbf-fb41-4b96-8940-b95698e58b18
>>>>>> OVEHOSTED_STORAGE/sdUUID=str:a81b229c-2422-4110-af45-1bc4d9a01bab
>>>>>> OVEHOSTED_STORAGE/iSCSITargetName=none:None
>>>>>> OVEHOSTED_STORAGE/metadataImageUUID=str:298fef26-3039-4c9e-bee8-39770fb2e821
>>>>>> OVEHOSTED_STORAGE/lockspaceVolumeUUID=str:c90a1959-369d-4109-8a6e-5c81f735e43b
>>>>>> OVEHOSTED_STORAGE/iSCSIPortalPort=none:None
>>>>>> OVEHOSTED_STORAGE/imgUUID=str:284c61f2-a901-46f0-a05a-2e5bf220efc6
>>>>>> OVEHOSTED_STORAGE/confImageUUID=str:8bcc7fae-2fbc-40aa-b75a-6c6adb8ba7dc
>>>>>> OVEHOSTED_STORAGE/spUUID=str:00000000-0000-0000-0000-000000000000
>>>>>> OVEHOSTED_STORAGE/lockspaceImageUUID=str:0ce0f2f9-b9ea-45b3-b091-d4f0e46e150f
>>>>>> OVEHOSTED_ENGINE/enableHcGlusterService=bool:False
>>>>>> OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
>>>>>> OVEHOSTED_STORAGE/iSCSIPortal=none:None
>>>>>> OVEHOSTED_STORAGE/volUUID=str:16304df2-2d95-4d75-932d-21d7c6de000d
>>>>>> OVEHOSTED_STORAGE/vgUUID=none:None
>>>>>> OVEHOSTED_STORAGE/confVolUUID=str:3b42f578-5d38-4820-b977-fa37b49926f7
>>>>>> OVEHOSTED_STORAGE/storageDomainConnection=str:sc5-gluster-1:/ovirt_engine
>>>>>> OVEHOSTED_STORAGE/iSCSIPortalUser=none:None
>>>>>> OVEHOSTED_VDSM/consoleType=str:qxl
>>>>>> OVEHOSTED_VM/vmMemSizeMB=int:16384
>>>>>> OVEHOSTED_VM/vmUUID=str:04ff4cf1-135a-4918-9a1f-8023322f89a3
>>>>>> OVEHOSTED_VM/vmMACAddr=str:00:16:3e:6e:86:b2
>>>>>> OVEHOSTED_VM/emulatedMachine=str:pc
>>>>>> OVEHOSTED_VM/consoleUUID=str:7d26f23e-59b6-4378-a0a4-f50e9653ac78
>>>>>> OVEHOSTED_VM/vmVCpus=str:4
>>>>>> OVEHOSTED_VM/nicUUID=str:a2b7dc56-02f6-4585-bd50-079b17aafe1f
>>>>>> OVEHOSTED_VM/cdromUUID=str:cde48946-a7dc-41a5-87ee-854eb4d4dc22
>>>>>> OVEHOSTED_VM/ovfArchive=str:/usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.1-20170406.1.el7.centos.ova
>>>>>> OVEHOSTED_VM/vmCDRom=none:None
>>>>>> OVEHOSTED_VM/automateVMShutdown=bool:True
>>>>>> OVEHOSTED_VM/cloudInitISO=str:generate
>>>>>> OVEHOSTED_VM/cloudinitInstanceDomainName=str:squaretrade.com
>>>>>> OVEHOSTED_VM/cloudinitInstanceHostName=str:sc5-ovirt-engine-1.squaretrade.com
>>>>>> OVEHOSTED_VM/rootSshPubkey=str:ssh-rsa [snip]
>>>>>> OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:True
>>>>>> OVEHOSTED_VM/cloudinitVMStaticCIDR=[snip]
>>>>>> OVEHOSTED_VM/cloudinitVMTZ=str:America/Los_Angeles
>>>>>> OVEHOSTED_VM/rootSshAccess=str:yes
>>>>>> OVEHOSTED_VM/cloudinitVMETCHOSTS=bool:False
>>>>>> OVEHOSTED_VM/cloudinitVMDNS=[snip]
>>>>>> OVEHOSTED_VDSM/spicePkiSubject=str:O=squaretrade.com,
>>>>>> CN=sc5-ovirt-2.squaretrade.com
>>>>>> OVEHOSTED_VDSM/pkiSubject=str:/C=EN/L=Test/O=Test/CN=Test
>>>>>> OVEHOSTED_VDSM/caSubject=str:/C=EN/L=Test/O=Test/CN=TestCA
>>>>>> OVEHOSTED_VDSM/cpu=str:model_Haswell-noTSX
>>>>>> OVEHOSTED_NOTIF/smtpPort=str:25
>>>>>> OVEHOSTED_NOTIF/smtpServer=[snip]
>>>>>> OVEHOSTED_NOTIF/sourceEmail=[snip]
>>>>>> OVEHOSTED_NOTIF/destEmail=[snip]
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users@ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
