[ovirt-users] ovirt-ha-agent

[Ariez Ahito]

last dec i installed hosted-engine seems to be working, we can migrate the 
engine to different host. but we need to reinstall everything because of 
gluster additional configuration.
so we did installed hosted-engine. but as per checking. we cannot migrate the 
engine to other hosts. and the ovirt-ha-agent and ovirt-ha-broker is status is 
inactive (dead) what are we missing?
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VDC6FBPC2TR3PKLJSGKJELJ65G4AQLUK/

[ovirt-users] ovirt-ha-agent not running

[Stefan Wolf]

hello,

since some days ovirt-ha-agent is not running anymore 
i ve 4 ovirt hosts and only on one host the agent is running.
maybe it was from an update, because i lost one agent after an other.

i ve done a complete fresh install for the host with the latest ovirt node.
I ve got on tree hosts this error

[root@kvm380 ~]# systemctl status ovirt-ha-agent
● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring 
Agent
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; 
vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Sa 2019-12-07 
14:56:21 UTC; 5s ago
  Process: 28002 ExecStart=/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent 
(code=exited, status=157)
 Main PID: 28002 (code=exited, status=157)
Tasks: 0
   CGroup: /system.slice/ovirt-ha-agent.service

Dez 07 14:56:21 kvm380.durchhalten.intern systemd[1]: ovirt-ha-agent.service: 
main process exited, code=exited, status=157/n/a
Dez 07 14:56:21 kvm380.durchhalten.intern systemd[1]: Unit 
ovirt-ha-agent.service entered failed state.
Dez 07 14:56:21 kvm380.durchhalten.intern systemd[1]: ovirt-ha-agent.service 
failed.

and this is in /var/log/ovirt-hosted-engine-ha/agent.log

MainThread::INFO::2019-12-07 
15:01:51,048::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) 
ovirt-hosted-engine-ha agent 2.3.6 started
MainThread::INFO::2019-12-07 
15:01:51,161::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname)
 Found certificate common name: kvm380.durchhalten.intern
MainThread::INFO::2019-12-07 
15:01:51,374::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Initializing ha-broker connection
MainThread::INFO::2019-12-07 
15:01:51,378::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor)
 Starting monitor network, options {'tcp_t_address': None, 'network_test': 
None, 'tcp_t_port': None, 'addr': '192.168.200.1'}
MainThread::ERROR::2019-12-07 
15:01:51,379::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
 Failed to start necessary monitors
MainThread::ERROR::2019-12-07 
15:01:51,381::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
 Traceback (most recent call last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
131, in _run_agent
return action(he)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
55, in action_proper
return he.start_monitoring()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
 line 432, in start_monitoring
self._initialize_broker()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
 line 556, in _initialize_broker
m.get('options', {}))
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 89, in start_monitor
).format(t=type, o=options, e=e)
RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 
2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': 
None, 'network_test': None, 'tcp_t_port': None, 'addr': '192.168.200.1'}]

MainThread::ERROR::2019-12-07 
15:01:51,381::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
 Trying to restart agent
MainThread::INFO::2019-12-07 
15:01:51,382::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent 
shutting down

maybe someone can give me an advice 

thx shb

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UE4TMWWT27BHFJWITYRV52CHG5AA7ZYD/

[ovirt-users] ovirt-ha-agent failed to start

[mustafa . taha . mu95]

hi 
i try to start ovirt-ha-agent in my self-host machine but it does not work 
if i run   journalctl -xe   it shows : 

Oct 03 04:33:24 ovirtnode44.exalt.ps ovirt-ha-agent[81909]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start 
necessary mo
Oct 03 04:33:24 ovirtnode44.exalt.ps ovirt-ha-agent[81909]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call 
last):
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
131, in _run_agent
return 
action(he)
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 
55, in action_prope
return 
he.start_monitoring()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
 line 413, in sta

self._initialize_broker()
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
 line 535, in _in

m.get('options', {}))
  File 
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", 
line 83, in start_mon
.format(type, 
options, e))
RequestError: 
Failed to start monitor ping, options {'addr': '192.168.3.2'}: [Errno 2] No 
such file or di
Oct 03 04:33:24 ovirtnode44.exalt.ps ovirt-ha-agent[81909]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent


anybody have idea ? 
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/7YD34KNXTPGSKEUPDSX7TEZITABXIMCD/

Re: [ovirt-users] ovirt-ha-agent service errors

[Vadim]

Hi, All

after upgrade to 4.1.5 no any error messages ;-)

Thanks to TEAM

Пнд 21 Авг 2017 14:29:49 +0300, Vadim  написал:
> Hi, All
> 
> ovirt 4.1.4 fresh install on two hosts with hosted-engine on both.
> gluster volume is replica 3 with two vdsm hosts and one VM under esxi
> 
> working only one vm for HE 
> 
> sometimes have such errors in ha-agent
> 
> # service ovirt-ha-agent status
> Redirecting to /bin/systemctl status  ovirt-ha-agent.service
> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring 
> Agent
>Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; 
> vendor preset: disabled)
>Active: active (running) since Thu 2017-08-17 11:43:44 MSK; 3 days ago
>  Main PID: 2534 (ovirt-ha-agent)
>CGroup: /system.slice/ovirt-ha-agent.service
>└─2534 /usr/bin/python 
> /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon
> 
> Aug 21 00:29:11 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 00:48:32 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 01:12:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 02:12:09 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 03:55:08 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 08:14:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 08:25:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 08:46:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 09:20:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> Aug 21 09:21:40 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has 
> bad health status, timeout in 300 seconds
> 
> 
> in agent log:
> 
> MainThread::INFO::2017-08-21 
> 09:21:40,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
>  Found OVF_STORE: imgUUID:c7b33006-e7c7-4e39-8d80-2301149ac8f9, 
> volUUID:184f9e45-ab1b-44b8-8a68-238042dba1a7
> MainThread::INFO::2017-08-21 
> 09:21:40,594::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
>  Found OVF_STORE: imgUUID:e7a3c173-9f87-4f6c-a807-63118b9b7cb2, 
> volUUID:92317b81-1bb0-43e6-b029-8931aa5d0af0
> MainThread::INFO::2017-08-21 
> 09:21:40,716::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>  Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2017-08-21 
> 09:21:40,749::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>  OVF_STORE volume path: 
> /rhev/data-center/mnt/glusterSD/localhost:_ovha/7b6badfb-4986-4983-9f62-ae55da33d15e/images/e7a3c173-9f87-4f6c-a807-63118b9b7cb2/92317b81-1bb0-43e6-b029-8931aa5d0af0
> MainThread::INFO::2017-08-21 
> 09:21:40,787::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>  Found an OVF for HE VM, trying to convert
> MainThread::INFO::2017-08-21 
> 09:21:40,792::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
>  Got vm.conf from OVF_STORE
> MainThread::ERROR::2017-08-21 
> 09:21:40,792::states::606::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
>  Engine VM has bad health status, timeout in 300 seconds
> MainThread::INFO::2017-08-21 
> 09:21:40,792::states::430::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
>  Engine vm running on localhost
> MainThread::INFO::2017-08-21 
> 09:21:40,796::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
>  Timeout cleared while transitioning  'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> ->  'ovirt_hosted_engine_ha.agent.states.EngineUp'>
> MainThread::INFO::2017-08-21 
> 09:21:40,800::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
>  Trying: notify time=1503296500.8 type=state_transition 
> 

[ovirt-users] ovirt-ha-agent service errors

[Vadim]

Hi, All

ovirt 4.1.4 fresh install on two hosts with hosted-engine on both.
gluster volume is replica 3 with two vdsm hosts and one VM under esxi

working only one vm for HE 

sometimes have such errors in ha-agent

# service ovirt-ha-agent status
Redirecting to /bin/systemctl status  ovirt-ha-agent.service
● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring 
Agent
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; 
vendor preset: disabled)
   Active: active (running) since Thu 2017-08-17 11:43:44 MSK; 3 days ago
 Main PID: 2534 (ovirt-ha-agent)
   CGroup: /system.slice/ovirt-ha-agent.service
   └─2534 /usr/bin/python 
/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon

Aug 21 00:29:11 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 00:48:32 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 01:12:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 02:12:09 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 03:55:08 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 08:14:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 08:25:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 08:46:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 09:20:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds
Aug 21 09:21:40 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent 
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad 
health status, timeout in 300 seconds


in agent log:

MainThread::INFO::2017-08-21 
09:21:40,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
 Found OVF_STORE: imgUUID:c7b33006-e7c7-4e39-8d80-2301149ac8f9, 
volUUID:184f9e45-ab1b-44b8-8a68-238042dba1a7
MainThread::INFO::2017-08-21 
09:21:40,594::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
 Found OVF_STORE: imgUUID:e7a3c173-9f87-4f6c-a807-63118b9b7cb2, 
volUUID:92317b81-1bb0-43e6-b029-8931aa5d0af0
MainThread::INFO::2017-08-21 
09:21:40,716::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
 Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-08-21 
09:21:40,749::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
 OVF_STORE volume path: 
/rhev/data-center/mnt/glusterSD/localhost:_ovha/7b6badfb-4986-4983-9f62-ae55da33d15e/images/e7a3c173-9f87-4f6c-a807-63118b9b7cb2/92317b81-1bb0-43e6-b029-8931aa5d0af0
MainThread::INFO::2017-08-21 
09:21:40,787::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
 Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-08-21 
09:21:40,792::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store)
 Got vm.conf from OVF_STORE
MainThread::ERROR::2017-08-21 
09:21:40,792::states::606::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
 Engine VM has bad health status, timeout in 300 seconds
MainThread::INFO::2017-08-21 
09:21:40,792::states::430::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume)
 Engine vm running on localhost
MainThread::INFO::2017-08-21 
09:21:40,796::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check)
 Timeout cleared while transitioning  -> 
MainThread::INFO::2017-08-21 
09:21:40,800::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
 Trying: notify time=1503296500.8 type=state_transition 
detail=EngineUpBadHealth-EngineUp hostname='kvm03'
MainThread::INFO::2017-08-21 
09:21:40,853::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify)
 Success, was notification of state_transition (EngineUpBadHealth-EngineUp) 
sent? sent
MainThread::INFO::2017-08-21 
09:21:40,853::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
 Initializing VDSM

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 6:14 PM, Gianluca Cecchi 
wrote:

> On Wed, Apr 26, 2017 at 4:54 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi <
>> gianluca.cec...@gmail.com> wrote:
>>
>>> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
>>> wrote:
>>>



 On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
 connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
 and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
 kages/vdsm/api/vdsmapi.py

 So it's still the parsing of the api yaml schema.
 On master we already merged a patch to reuse an existing connection if
 available and this should mitigate/resolve the issue:
 https://gerrit.ovirt.org/73757/

 It's still not that clear why we are facing this kind of performance
 regression.


>>> Does this mean that I could try to patch the
>>>   ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
>>> verify if it solves...?
>>>
>>
>> Unfortunately it's not enough by itself since it also requires
>> https://gerrit.ovirt.org/#/c/58029/
>>
>>
>>> Or do I have to wait an official patched rpm?
>>>
>>> Gianluca
>>>
>>
>>
> And what if I change all the 3 files involved in the 2 gerrit entries:
> ovirt_hosted_engine_ha/lib/util.py
> lib/yajsonrpc/stomp.py
> lib/yajsonrpc/stompreactor.py
>
> ?
>
>
Yes, it should work although we never officially back-ported and tested it
for 4.1.z.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Wed, Apr 26, 2017 at 4:54 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>>
>>> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
>>> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
>>> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
>>> kages/vdsm/api/vdsmapi.py
>>>
>>> So it's still the parsing of the api yaml schema.
>>> On master we already merged a patch to reuse an existing connection if
>>> available and this should mitigate/resolve the issue:
>>> https://gerrit.ovirt.org/73757/
>>>
>>> It's still not that clear why we are facing this kind of performance
>>> regression.
>>>
>>>
>> Does this mean that I could try to patch the
>>   ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
>> verify if it solves...?
>>
>
> Unfortunately it's not enough by itself since it also requires
> https://gerrit.ovirt.org/#/c/58029/
>
>
>> Or do I have to wait an official patched rpm?
>>
>> Gianluca
>>
>
>
And what if I change all the 3 files involved in the 2 gerrit entries:
ovirt_hosted_engine_ha/lib/util.py
lib/yajsonrpc/stomp.py
lib/yajsonrpc/stompreactor.py

?
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi 
wrote:

> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>>
>> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
>> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
>> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
>> kages/vdsm/api/vdsmapi.py
>>
>> So it's still the parsing of the api yaml schema.
>> On master we already merged a patch to reuse an existing connection if
>> available and this should mitigate/resolve the issue:
>> https://gerrit.ovirt.org/73757/
>>
>> It's still not that clear why we are facing this kind of performance
>> regression.
>>
>>
> Does this mean that I could try to patch the   
> ovirt_hosted_engine_ha/lib/util.py
> file also in my 4.1.1 install and verify if it solves...?
>

Unfortunately it's not enough by itself since it also requires
https://gerrit.ovirt.org/#/c/58029/


> Or do I have to wait an official patched rpm?
>
> Gianluca
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
wrote:

>
>
>
> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-
> packages/vdsm/api/vdsmapi.py
>
> So it's still the parsing of the api yaml schema.
> On master we already merged a patch to reuse an existing connection if
> available and this should mitigate/resolve the issue:
> https://gerrit.ovirt.org/73757/
>
> It's still not that clear why we are facing this kind of performance
> regression.
>
>
Does this mean that I could try to patch the
  ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
verify if it solves...?
Or do I have to wait an official patched rpm?

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 1:28 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer  wrote:
>
>> On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <
>> gianluca.cec...@gmail.com> wrote:
>>
>>> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>>>


 Hi Gianluca,

 You can run this on the host:

 $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
 CLoader: True

 If you get "CLoader: False", you have some packaging issue, CLoader
 is available on all supported platforms.

 Nir


> Thanks,
> Gianluca
>
>
>>>
>>> It seems ok.
>>>
>>> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
>>> 'CLoader:', hasattr(yaml, 'CLoader')"
>>> CLoader: True
>>> [root@ovirt01 ovirt-hosted-engine-ha]#
>>>
>>> Anyway see here a sample of the spikes that it cntinues to have.. from
>>> 15% to 55% many times
>>> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE
>>> /view?usp=sharing
>>>
>>
>> There are two issues in this video:
>> - Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
>> that it needs so much memory.
>> - Unusual cpu usage - but not the kind of usage related to yaml parsing.
>>
>> I would open two bugs for this. We have seen the first issue few month
>> ago, and
>> we did nothing about it so the memory leak was not fixed.
>>
>> To understand the unusual cpu usage, we need to integrate yappi into
>> ovirt-ha-agent,
>> and do some profiling to understand where cpu time is spent.
>>
>> Simone, can you do something based on these patches?
>> https://gerrit.ovirt.org/#/q/topic:generic-profiler
>>
>> I hope to get these patches merged soon.
>>
>>
> Absolutely at this point.
>
>

On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
and the 95.98% is in Schema.__init__
in /usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py

So it's still the parsing of the api yaml schema.
On master we already merged a patch to reuse an existing connection if
available and this should mitigate/resolve the issue:
https://gerrit.ovirt.org/73757/

It's still not that clear why we are facing this kind of performance
regression.



> Nir
>>
>>
>>>
>>>
>>> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an
>>> F25 guest and a C7 desktop VMs running, without doing almost anything.
>>>
>>> Gianluca
>>>
>>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer  wrote:

> On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>>
>>>
>>>
>>> Hi Gianluca,
>>>
>>> You can run this on the host:
>>>
>>> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
>>> CLoader: True
>>>
>>> If you get "CLoader: False", you have some packaging issue, CLoader
>>> is available on all supported platforms.
>>>
>>> Nir
>>>
>>>
 Thanks,
 Gianluca


>>
>> It seems ok.
>>
>> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
>> 'CLoader:', hasattr(yaml, 'CLoader')"
>> CLoader: True
>> [root@ovirt01 ovirt-hosted-engine-ha]#
>>
>> Anyway see here a sample of the spikes that it cntinues to have.. from
>> 15% to 55% many times
>> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/
>> view?usp=sharing
>>
>
> There are two issues in this video:
> - Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
> that it needs so much memory.
> - Unusual cpu usage - but not the kind of usage related to yaml parsing.
>
> I would open two bugs for this. We have seen the first issue few month
> ago, and
> we did nothing about it so the memory leak was not fixed.
>
> To understand the unusual cpu usage, we need to integrate yappi into
> ovirt-ha-agent,
> and do some profiling to understand where cpu time is spent.
>
> Simone, can you do something based on these patches?
> https://gerrit.ovirt.org/#/q/topic:generic-profiler
>
> I hope to get these patches merged soon.
>
>
Absolutely at this point.


> Nir
>
>
>>
>>
>> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an
>> F25 guest and a C7 desktop VMs running, without doing almost anything.
>>
>> Gianluca
>>
>>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi 
wrote:

> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>
>>
>>
>> Hi Gianluca,
>>
>> You can run this on the host:
>>
>> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
>> CLoader: True
>>
>> If you get "CLoader: False", you have some packaging issue, CLoader
>> is available on all supported platforms.
>>
>> Nir
>>
>>
>>> Thanks,
>>> Gianluca
>>>
>>>
>
> It seems ok.
>
> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
> 'CLoader:', hasattr(yaml, 'CLoader')"
> CLoader: True
> [root@ovirt01 ovirt-hosted-engine-ha]#
>
> Anyway see here a sample of the spikes that it cntinues to have.. from 15%
> to 55% many times
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/view?usp=sharing
>

There are two issues in this video:
- Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
that it needs so much memory.
- Unusual cpu usage - but not the kind of usage related to yaml parsing.

I would open two bugs for this. We have seen the first issue few month ago,
and
we did nothing about it so the memory leak was not fixed.

To understand the unusual cpu usage, we need to integrate yappi into
ovirt-ha-agent,
and do some profiling to understand where cpu time is spent.

Simone, can you do something based on these patches?
https://gerrit.ovirt.org/#/q/topic:generic-profiler

I hope to get these patches merged soon.

Nir


>
>
> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an F25
> guest and a C7 desktop VMs running, without doing almost anything.
>
> Gianluca
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:

>
>
> Hi Gianluca,
>
> You can run this on the host:
>
> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
> CLoader: True
>
> If you get "CLoader: False", you have some packaging issue, CLoader
> is available on all supported platforms.
>
> Nir
>
>
>> Thanks,
>> Gianluca
>>
>>

It seems ok.

[root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
'CLoader:', hasattr(yaml, 'CLoader')"
CLoader: True
[root@ovirt01 ovirt-hosted-engine-ha]#

Anyway see here a sample of the spikes that it cntinues to have.. from 15%
to 55% many times
https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/view?usp=sharing

The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an F25
guest and a C7 desktop VMs running, without doing almost anything.

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi 
wrote:

>
>
>
> If I can apply the patch also to 4.0.3 I'm going to see if there is then a
> different behavior.
> Let me know,
>
>
> I'm trying it right now.
> Any other tests will be really appreciated.
>
> The patch is pretty simply, you can apply that on the fly.
> You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
> directly edit
> /usr/lib/python2.7/site-packages/api/vdsmapi.py
> around line 97 changing from
> loaded_schema = yaml.load(f)
> to
> loaded_schema = yaml.load(f, Loader=yaml.CLoader)
> Please pay attention to keep exactly the same amount of initial spaces.
>
> Then you can simply restart the HA agent and check.
>
>

Hello,
I'm again registering high spikes of ovirt-ha-agent with only 2-3 VMs up
and with almost no activity
The package of the involved file
/usr/lib/python2.7/site-packages/api/vdsmapi.py
is now at veriosn vdsm-api-4.19.4-1.el7.centos.noarch and I see that the
file contains this kind of lines

129 try:
130 for path in paths:
131 with open(path) as f:
132 if hasattr(yaml, 'CLoader'):
133 loader = yaml.CLoader
134 else:
135 loader = yaml.Loader
136 loaded_schema = yaml.load(f, Loader=loader)
137
138 types = loaded_schema.pop('types')
139 self._types.update(types)
140 self._methods.update(loaded_schema)
141 except EnvironmentError:
142 raise SchemaNotFound("Unable to find API schema file")

So there is a conditional statement...
How can I be sure that "loader" is set to "yaml.CLoader" that was what in
4.0 was able to lower the cpu usage of ovirt-ha-agent?
Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Sam Cappello]

it worked for me as well - load avg. < 1 now, ovirt-ha-agent pops up in 
top periodically, but not on top using 100% CPU all the time anymore.

thanks!
sam

Gianluca Cecchi wrote on 10/7/2016 10:13 AM:



On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi > wrote:




On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi
> wrote:

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer > wrote:


wasn’t it suppose to be fixed to reuse the connection?
Like all the other clients (vdsm migration code:-)


This is orthogonal issue.

Does schema validation matter then if there would be
only one connection at the start up?


Loading once does not help command line tools like
vdsClient, hosted-engine and
vdsm-tool.

Nir




Simone, reusing the connection is good idea
anyway, but what you describe is
a bug in the client library. The library does
*not* need to load and parse the
schema at all for sending requests to vdsm.

The schema is only needed if you want to
verify request parameters,
or provide online help, these are not needed
in a client library.

Please file an infra bug about it.


Done,
https://bugzilla.redhat.com/show_bug.cgi?id=1381899



Here is a patch that should eliminate most most of
the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system
showing this problem.

Cheers,
Nir
___




this is a video of 1 minute with the same system as the first
post, but in 4.0.3 now and the same 3 VMs powered on without
any particular load.
It seems very similar to the previous 3.6.6 in cpu used by
ovirt-ha-agent.


https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing



Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if
there is then a different behavior.
Let me know,


I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you
could directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial
spaces.

Then you can simply restart the HA agent and check.

Gianluca

___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users





What I've done (I didn't read your answer in between and this is a 
test system not so important... )


set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process 
or at least has ranges between 5% and 12% and not more


BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; 
vendor preset: enabled)

   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh 
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)

 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 
--write-pipe-fd 40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 
--write-pipe-fd 46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 
--write-pipe-fd 56 --max-threads 10 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 07 Oct 2016, at 16:10, Simone Tiraboschi  wrote:
> 
> 
> 
>> On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>>> 
>>> 
>>> 
>>> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek 
>>>  wrote:
 
> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>> 
 On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
  wrote:
 
 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  
> wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>  wrote:
>> 
>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  
>>> wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there 
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically 
>> reconnects over json rpc and this is CPU intensive since the client 
>> has to parse the yaml API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the 
>> other clients (vdsm migration code:-) 
> 
> This is orthogonal issue.
 
 Yes it is. And that’s the issue;-)
 Both are wrong, but by “fixing” the schema validation only you lose the 
 motivation to fix the meaningless wasteful reconnect
>>> 
>>> Yes, we are going to fix that too ( 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 )
>> 
>> that’s great! Also al the other vdsClient uses?:-)
> 
> https://gerrit.ovirt.org/#/c/62729/

Cool. It's not so important for one time actions but we need it to be able to 
drop xmlrpc finally, so it is important:)

>  
>> What is that periodic one call anyway? Is there only one? Maybe we don’t 
>> need it so much.
> 
> Currently ovirt-ha-agent is periodically reconnecting the hosted-engine 
> storage domain and checking its status. This is already on jsonrpc.

Ok. As long as it is low frequency it's ok. Just bear in mind for future that 
some of the vdsm calls may be heavy and impact performance, regardless the 
communication layer. 

Thanks and have a nice weekend,
michal

> In 4.1 all the monitoring will be moved to jsonrpc.
>  
>> 
>>> but it would require also 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 to be fixed.
>> 
>> This is less good. Well, worst case you can reconnect yourself, all you need 
>> is a notification when the existing connection breaks
>> 
>>>  
 
>  
>> Does schema validation matter then if there would be only one connection 
>> at the start up?
> 
> Loading once does not help command line tools like vdsClient, 
> hosted-engine and
> vdsm-tool. 
 
 none of the other tools is using json-rpc.
>>> 
>>> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
>>> remaining tools since xmlrpc has been deprecated with 4.0
>> 
>> ok. though setup is a one-time action so it’s not an issue there
>> 
>>>  
 
> 
> Nir
>  
>> 
> 
> Simone, reusing the connection is good idea anyway, but what you 
> describe is 
> a bug in the client library. The library does *not* need to load and 
> parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
 
 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
>>> 
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

>
> And a bright new 3-minutes video here:
> https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/
> view?usp=sharing
>

> It seems that now ovirt-ha-agent or is not present in top cpu process or
> at least has ranges between 5% and 12% and not more
>

5-12% is probably 10 times more than needed for the agent, profiling
should tell us where time is spent.

Since the agent depends on vdsm, we can reuse vdsm cpu profiler, see
lib/vdsm/profiling/cpu.py. It is not ready yet to be used by other
applications,
but making it more general should be easy.

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi 
wrote:

>
>
> On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi  > wrote:
>
>> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>>
>>>
 wasn’t it suppose to be fixed to reuse the connection? Like all the
 other clients (vdsm migration code:-)

>>>
>>> This is orthogonal issue.
>>>
>>>
 Does schema validation matter then if there would be only one
 connection at the start up?

>>>
>>> Loading once does not help command line tools like vdsClient,
>>> hosted-engine and
>>> vdsm-tool.
>>>
>>> Nir
>>>
>>>


>> Simone, reusing the connection is good idea anyway, but what you
>> describe is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

 Here is a patch that should eliminate most most of the problem:
 https://gerrit.ovirt.org/65230

 Would be nice if it can be tested on the system showing this problem.

 Cheers,
 Nir
 ___


>>
>> this is a video of 1 minute with the same system as the first post, but
>> in 4.0.3 now and the same 3 VMs powered on without any particular load.
>> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>>
>> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8
>> /view?usp=sharing
>>
>> Enjoy Nir ;-)
>>
>> If I can apply the patch also to 4.0.3 I'm going to see if there is then
>> a different behavior.
>> Let me know,
>>
>>
> I'm trying it right now.
> Any other tests will be really appreciated.
>
> The patch is pretty simply, you can apply that on the fly.
> You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
> directly edit
> /usr/lib/python2.7/site-packages/api/vdsmapi.py
> around line 97 changing from
> loaded_schema = yaml.load(f)
> to
> loaded_schema = yaml.load(f, Loader=yaml.CLoader)
> Please pay attention to keep exactly the same amount of initial spaces.
>
> Then you can simply restart the HA agent and check.
>
> Gianluca
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>

What I've done (I didn't read your answer in between and this is a test
system not so important... )

set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process or at
least has ranges between 5% and 12% and not more

BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: enabled)
   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 --write-pipe-fd
40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd
56 --max-threads 10 --max-queue...
   ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
64 --max-threads 10 --max-queue...
   ├─21149 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
72 --max-threads 10 --max-queue...
   ├─21156 /usr/libexec/ioprocess --read-pipe-fd 80 --write-pipe-fd
78 --max-threads 10 --max-queue...
   ├─21177 /usr/libexec/ioprocess --read-pipe-fd 88 --write-pipe-fd
87 --max-threads 10 --max-queue...
   ├─21204 /usr/libexec/ioprocess --read-pipe-fd 99 --write-pipe-fd
98 --max-threads 10 --max-queue...
   └─21239 /usr/libexec/ioprocess --read-pipe-fd 111
--write-pipe-fd 110 --max-threads 10 --max-que...

Oct 07 16:02:52 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:54 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:56 ovirt01.lutwyn.org vdsm[21023]: vdsm 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>
>
>
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek <
> michal.skriva...@redhat.com> wrote:
>
>>
>> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
>>
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
>> michal.skriva...@redhat.com> wrote:
>>
>>>
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>>
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi <
> stira...@redhat.com> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor 
>> wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>> Yes it is. And that’s the issue;-)
>> Both are wrong, but by “fixing” the schema validation only you lose the
>> motivation to fix the meaningless wasteful reconnect
>>
>
> Yes, we are going to fix that too ( https://bugzilla.redhat.com/
> show_bug.cgi?id=1349829 )
>
>
> that’s great! Also al the other vdsClient uses?:-)
>

https://gerrit.ovirt.org/#/c/62729/


> What is that periodic one call anyway? Is there only one? Maybe we don’t
> need it so much.
>

Currently ovirt-ha-agent is periodically reconnecting the hosted-engine
storage domain and checking its status. This is already on jsonrpc.
In 4.1 all the monitoring will be moved to jsonrpc.


>
> but it would require also https://bugzilla.redhat.com/
> show_bug.cgi?id=1376843 to be fixed.
>
>
> This is less good. Well, worst case you can reconnect yourself, all you
> need is a notification when the existing connection breaks
>
>
>
>>
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>>
>> none of the other tools is using json-rpc.
>>
>
> hosted-engine-setup is, and sooner or later we'll have to migrate also the
> remaining tools since xmlrpc has been deprecated with 4.0
>
>
> ok. though setup is a one-time action so it’s not an issue there
>
>
>
>>
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
> 
> 
> 
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:59, Nir Soffer > > wrote:
>> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>> > wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer >> > wrote:
>>> 
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer >> > wrote:
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor >> > wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>> 
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>>> json rpc and this is CPU intensive since the client has to parse the yaml 
>>> API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
>> clients (vdsm migration code:-) 
>> 
>> This is orthogonal issue.
> 
> Yes it is. And that’s the issue;-)
> Both are wrong, but by “fixing” the schema validation only you lose the 
> motivation to fix the meaningless wasteful reconnect
> 
> Yes, we are going to fix that too ( 
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 
>  )

that’s great! Also al the other vdsClient uses?:-)
What is that periodic one call anyway? Is there only one? Maybe we don’t need 
it so much.

> but it would require also https://bugzilla.redhat.com/show_bug.cgi?id=1376843 
>  to be fixed.

This is less good. Well, worst case you can reconnect yourself, all you need is 
a notification when the existing connection breaks

>  
> 
>>  
>> Does schema validation matter then if there would be only one connection at 
>> the start up?
>> 
>> Loading once does not help command line tools like vdsClient, hosted-engine 
>> and
>> vdsm-tool. 
> 
> none of the other tools is using json-rpc.
> 
> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
> remaining tools since xmlrpc has been deprecated with 4.0

ok. though setup is a one-time action so it’s not an issue there

>  
> 
>> 
>> Nir
>>  
>> 
>>> 
>>> Simone, reusing the connection is good idea anyway, but what you describe 
>>> is 
>>> a bug in the client library. The library does *not* need to load and parse 
>>> the
>>> schema at all for sending requests to vdsm.
>>> 
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>> 
>>> Please file an infra bug about it.
>>> 
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>>> 
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230 
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org 
>>> http://lists.ovirt.org/mailman/listinfo/users 
>>> 
>> 
>> 
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users 
> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi 
wrote:

> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>
>>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>>
>>>
>
> this is a video of 1 minute with the same system as the first post, but in
> 4.0.3 now and the same 3 VMs powered on without any particular load.
> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/
> view?usp=sharing
>
> Enjoy Nir ;-)
>
> If I can apply the patch also to 4.0.3 I'm going to see if there is then a
> different behavior.
> Let me know,
>
>
I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial spaces.

Then you can simply restart the HA agent and check.

Gianluca
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:42, Nir Soffer > > wrote:
>> 
>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer > > wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor > > wrote:
>> Hi,
>> 
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>> json rpc and this is CPU intensive since the client has to parse the yaml 
>> API specification each time it connects.
> 
> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
> clients (vdsm migration code:-) 
> 
> This is orthogonal issue.

Yes it is. And that’s the issue;-)
Both are wrong, but by “fixing” the schema validation only you lose the 
motivation to fix the meaningless wasteful reconnect

>  
> Does schema validation matter then if there would be only one connection at 
> the start up?
> 
> Loading once does not help command line tools like vdsClient, hosted-engine 
> and
> vdsm-tool. 

none of the other tools is using json-rpc.

> 
> Nir
>  
> 
>> 
>> Simone, reusing the connection is good idea anyway, but what you describe is 
>> a bug in the client library. The library does *not* need to load and parse 
>> the
>> schema at all for sending requests to vdsm.
>> 
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>> 
>> Please file an infra bug about it.
>> 
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>> 
>> 
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230 
>> 
>> Would be nice if it can be tested on the system showing this problem.
>> 
>> Cheers,
>> Nir
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:

>
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other
>> clients (vdsm migration code:-)
>>
>
> This is orthogonal issue.
>
>
>> Does schema validation matter then if there would be only one connection
>> at the start up?
>>
>
> Loading once does not help command line tools like vdsClient,
> hosted-engine and
> vdsm-tool.
>
> Nir
>
>
>>
>>
 Simone, reusing the connection is good idea anyway, but what you
 describe is
 a bug in the client library. The library does *not* need to load and
 parse the
 schema at all for sending requests to vdsm.

 The schema is only needed if you want to verify request parameters,
 or provide online help, these are not needed in a client library.

 Please file an infra bug about it.

>>>
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>>
>>
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230
>>
>> Would be nice if it can be tested on the system showing this problem.
>>
>> Cheers,
>> Nir
>> ___
>>
>>

this is a video of 1 minute with the same system as the first post, but in
4.0.3 now and the same 3 VMs powered on without any particular load.
It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.

https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing

Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if there is then a
different behavior.
Let me know,

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>>
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

 Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
 over json rpc and this is CPU intensive since the client has to parse the
 yaml API specification each time it connects.

>>>
> wasn’t it suppose to be fixed to reuse the connection? Like all the other
> clients (vdsm migration code:-)
>

This is orthogonal issue.


> Does schema validation matter then if there would be only one connection
> at the start up?
>

Loading once does not help command line tools like vdsClient, hosted-engine
and
vdsm-tool.

Nir


>
>
>>> Simone, reusing the connection is good idea anyway, but what you
>>> describe is
>>> a bug in the client library. The library does *not* need to load and
>>> parse the
>>> schema at all for sending requests to vdsm.
>>>
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>>
>>> Please file an infra bug about it.
>>>
>>
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>
>
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230
>
> Would be nice if it can be tested on the system showing this problem.
>
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
> 
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  > wrote:
> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  > wrote:
> Hi,
> 
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
> 
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
> json rpc and this is CPU intensive since the client has to parse the yaml API 
> specification each time it connects.

wasn’t it suppose to be fixed to reuse the connection? Like all the other 
clients (vdsm migration code:-) 
Does schema validation matter then if there would be only one connection at the 
start up?

> 
> Simone, reusing the connection is good idea anyway, but what you describe is 
> a bug in the client library. The library does *not* need to load and parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
> 
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
> 
> 
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230 
> 
> Would be nice if it can be tested on the system showing this problem.
> 
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>>
 Hi,

 did you found a solution or cause for this high CPU usage?
 I have installed the self hosted engine on another server and there is
 no VM running but ovirt-ha-agent uses heavily the CPU.

>>>
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>>> over json rpc and this is CPU intensive since the client has to parse the
>>> yaml API specification each time it connects.
>>>
>>
>> Simone, reusing the connection is good idea anyway, but what you describe
>> is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

Here is a patch that should eliminate most most of the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system showing this problem.

Cheers,
Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
> Simone, reusing the connection is good idea anyway, but what you describe
> is
> a bug in the client library. The library does *not* need to load and parse
> the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
Thanks.


> Nir
>
>
>> The issue is tracked here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
>> should reuse json-rpc connections
>> but it depends on:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
>> keep-alive with reconnect if needed logic for the python jsonrpc client
>>
>>
>>
>>>
>>> cheers
>>> gregor
>>>
>>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan >> > > wrote:
>>> >
>>> > Does the spikes correlates with info messages on extracting the
>>> ovf?
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > yes, it seems so and it happens every 14-15 seconds
>>> >
>>> > These are the lines I see scrolling in agent.log when I notice cpu
>>> > spikes in ovirt-ha-agent...
>>> >
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Connecting storage server
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Refreshing the storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.I
>>> mage::(prepare_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Reloading vm.conf from the shared storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > Extracting Engine VM OVF from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > OVF_STORE volume path:
>>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab2
>>> 2-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Found an OVF for HE VM, trying to convert
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Got vm.conf from OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> > Current state EngineUp (score: 3400)
>>> >
>>> >
>>> > ___
>>> > Users mailing list
>>> > Users@ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>> 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>
>> Hi,
>>
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>
>
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
> json rpc and this is CPU intensive since the client has to parse the yaml
> API specification each time it connects.
>

Simone, reusing the connection is good idea anyway, but what you describe
is
a bug in the client library. The library does *not* need to load and parse
the
schema at all for sending requests to vdsm.

The schema is only needed if you want to verify request parameters,
or provide online help, these are not needed in a client library.

Please file an infra bug about it.

Nir


> The issue is tracked here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
> should reuse json-rpc connections
> but it depends on:
> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
> keep-alive with reconnect if needed logic for the python jsonrpc client
>
>
>
>>
>> cheers
>> gregor
>>
>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan > > > wrote:
>> >
>> > Does the spikes correlates with info messages on extracting the ovf?
>> >
>> >
>> >
>> >
>> >
>> >
>> > yes, it seems so and it happens every 14-15 seconds
>> >
>> > These are the lines I see scrolling in agent.log when I notice cpu
>> > spikes in ovirt-ha-agent...
>> >
>> > MainThread::INFO::2016-08-08
>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Connecting storage server
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Refreshing the storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.
>> Image::(prepare_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Reloading vm.conf from the shared storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > Extracting Engine VM OVF from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > OVF_STORE volume path:
>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-
>> ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Found an OVF for HE VM, trying to convert
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Got vm.conf from OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(start_monitoring)
>> > Current state EngineUp (score: 3400)
>> >
>> >
>> > ___
>> > Users mailing list
>> > Users@ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
json rpc and this is CPU intensive since the client has to parse the yaml
API specification each time it connects.
The issue is tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent should
reuse json-rpc connections
but it depends on:
https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
keep-alive with reconnect if needed logic for the python jsonrpc client


>
> cheers
> gregor
>
> On 08/08/16 15:09, Gianluca Cecchi wrote:
> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > > wrote:
> >
> > Does the spikes correlates with info messages on extracting the ovf?
> >
> >
> >
> >
> >
> >
> > yes, it seems so and it happens every 14-15 seconds
> >
> > These are the lines I see scrolling in agent.log when I notice cpu
> > spikes in ovirt-ha-agent...
> >
> > MainThread::INFO::2016-08-08
> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Connecting storage server
> > MainThread::INFO::2016-08-08
> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Refreshing the storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.
> image.Image::(prepare_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Reloading vm.conf from the shared storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Trying to get a fresher copy of vm configuration from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> > MainThread::INFO::2016-08-08
> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > Extracting Engine VM OVF from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > OVF_STORE volume path:
> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/
> 31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-
> 01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Found an OVF for HE VM, trying to convert
> > MainThread::INFO::2016-08-08
> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Got vm.conf from OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(start_monitoring)
> > Current state EngineUp (score: 3400)
> >
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[gregor]

Hi,

did you found a solution or cause for this high CPU usage?
I have installed the self hosted engine on another server and there is
no VM running but ovirt-ha-agent uses heavily the CPU.

cheers
gregor

On 08/08/16 15:09, Gianluca Cecchi wrote:
> On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > wrote:
> 
> Does the spikes correlates with info messages on extracting the ovf?
> 
> 
> 
> 
> 
> 
> yes, it seems so and it happens every 14-15 seconds
> 
> These are the lines I see scrolling in agent.log when I notice cpu
> spikes in ovirt-ha-agent...
> 
> MainThread::INFO::2016-08-08
> 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2016-08-08
> 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2016-08-08
> 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2016-08-08
> 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> MainThread::INFO::2016-08-08
> 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> volUUID:1a18851e-6858-401c-be6e-af14415034b5
> MainThread::INFO::2016-08-08
> 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>  
> MainThread::INFO::2016-08-08
> 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2016-08-08
> 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent

[Sandro Bonazzola]

Adding Martin.

Il 26/Ago/2016 11:14, "David Gossage"  ha
scritto:
>
> On Fri, Aug 26, 2016 at 1:38 AM, Renout Gerrits  wrote:
>>
>> Depends on your systemd configuration. ovirt-ha-agent and broker
daemon's both log to stdout and it's own logfile. All messages to stdout
will go to journald and be forwarded to /var/log/messages
(ForwardToSyslog=yes in /etc/systemd/journald.conf I think).
>> So the ovirt-ha-agent doesn't log to /var/log/messages, journald does.
if it should log to stdout is another discussion, but maybe there's good
reason for that, backwards compatibility, don't know.
>>
>> An easy fix is redirecting the output of the daemon to /dev/null. in
/usr/lib/systemd/system/ovirt-ha-agent.service add StandardOutput=null to
the [service] section.
>
>
> This did not start occurring until after I updated ovirt on August 12th.
I jumped a couple updates I think so which update it started with not sure.
>
> Aug 12 21:54:58 Updated: ovirt-vmconsole-1.0.2-1.el7.centos.noarch
> Aug 12 21:55:23 Updated: ovirt-vmconsole-host-1.0.2-1.el7.centos.noarch
> Aug 12 21:55:34 Updated: ovirt-setup-lib-1.0.1-1.el7.centos.noarch
> Aug 12 21:55:42 Updated: libgovirt-0.3.3-1.el7_2.4.x86_64
> Aug 12 21:56:37 Updated:
ovirt-hosted-engine-ha-1.3.5.7-1.el7.centos.noarch
> Aug 12 21:56:37 Updated:
ovirt-engine-sdk-python-3.6.7.0-1.el7.centos.noarch
> Aug 12 21:56:38 Updated:
ovirt-hosted-engine-setup-1.3.7.2-1.el7.centos.noarch
> Aug 12 21:58:46 Updated: 1:ovirt-release36-3.6.7-1.noarch
>
> So is this considered normal behavior that everyone should need to
reconfigure their logging to deal with or did someone leave it going to
stdout for debugging and not turn it back off?  I know I can manipulate
logging myself if I need to, but I'd rather not have to customize that and
then check every update that it's still applied or that I still get logs at
all.
>
>
>>
>> Renout
>>
>> On Thu, Aug 25, 2016 at 10:39 PM, David Gossage <
dgoss...@carouselchecks.com> wrote:
>>>
>>> This service seems to be logging to both /var/log/messages
and /var/log/ovirt-hosted-engine-ha/agent.log
>>>
>>> Anything that may be causing that?  Centos7 ovirt 3.6.7
>>>
>>> MainThread::INFO::2016-08-25
15:38:36,912::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE
>>> MainThread::INFO::2016-08-25
15:38:36,976::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path:
/rhev/data-center/mnt/glusterSD/ccgl1.gl.local:HOST1/6a0bca4a-a1be-47d3-be51-64c6277d1f0f/images/c12c8000-0373-419b-963b-98b04adca760/fb6e2509-4786-433d-868f-a6303dd69cca
>>> MainThread::INFO::2016-08-25
15:38:37,097::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Found an OVF for HE VM, trying to convert
>>> MainThread::INFO::2016-08-25
15:38:37,102::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Got vm.conf from OVF_STORE
>>> MainThread::INFO::2016-08-25
15:38:37,200::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 3400)
>>> MainThread::INFO::2016-08-25
15:38:37,201::hosted_engine::467::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
>>> MainThread::INFO::2016-08-25
15:38:47,346::hosted_engine::613::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
>>> MainThread::INFO::2016-08-25
15:38:47,439::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
>>> MainThread::INFO::2016-08-25
15:38:47,454::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
>>> MainThread::INFO::2016-08-25
15:38:47,618::storage_server::222::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
>>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
closed
>>>
>>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:mem_free.MemFree:memFree: 16466
>>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:engine_health.CpuLoadNoEngine:VM not on this host
>>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:mgmt_bridge.MgmtBridge:Found bridge ovirtmgmt with ports
>>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:VM not on this host
>>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:System load total=0.1022,
engine=0., non-engine=0.1022
>>> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
VDSM
>>> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:

Re: [ovirt-users] ovirt-ha-agent

[David Gossage]

On Fri, Aug 26, 2016 at 1:38 AM, Renout Gerrits  wrote:

> Depends on your systemd configuration. ovirt-ha-agent and broker daemon's
> both log to stdout and it's own logfile. All messages to stdout will go to
> journald and be forwarded to /var/log/messages (ForwardToSyslog=yes in
> /etc/systemd/journald.conf I think).
> So the ovirt-ha-agent doesn't log to /var/log/messages, journald does. if
> it should log to stdout is another discussion, but maybe there's good
> reason for that, backwards compatibility, don't know.
>
> An easy fix is redirecting the output of the daemon to /dev/null. in
> /usr/lib/systemd/system/ovirt-ha-agent.service add StandardOutput=null to
> the [service] section.
>

This did not start occurring until after I updated ovirt on August 12th. I
jumped a couple updates I think so which update it started with not sure.

Aug 12 21:54:58 Updated: ovirt-vmconsole-1.0.2-1.el7.centos.noarch
Aug 12 21:55:23 Updated: ovirt-vmconsole-host-1.0.2-1.el7.centos.noarch
Aug 12 21:55:34 Updated: ovirt-setup-lib-1.0.1-1.el7.centos.noarch
Aug 12 21:55:42 Updated: libgovirt-0.3.3-1.el7_2.4.x86_64
Aug 12 21:56:37 Updated: ovirt-hosted-engine-ha-1.3.5.7-1.el7.centos.noarch
Aug 12 21:56:37 Updated: ovirt-engine-sdk-python-3.6.7.0-1.el7.centos.noarch
Aug 12 21:56:38 Updated:
ovirt-hosted-engine-setup-1.3.7.2-1.el7.centos.noarch
Aug 12 21:58:46 Updated: 1:ovirt-release36-3.6.7-1.noarch

So is this considered normal behavior that everyone should need to
reconfigure their logging to deal with or did someone leave it going to
stdout for debugging and not turn it back off?  I know I can manipulate
logging myself if I need to, but I'd rather not have to customize that and
then check every update that it's still applied or that I still get logs at
all.



> Renout
>
> On Thu, Aug 25, 2016 at 10:39 PM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> This service seems to be logging to both /var/log/messages
>> and /var/log/ovirt-hosted-engine-ha/agent.log
>>
>> Anything that may be causing that?  Centos7 ovirt 3.6.7
>>
>> MainThread::INFO::2016-08-25 15:38:36,912::ovf_store::109::
>> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>> Extracting Engine VM OVF from the OVF_STORE
>> MainThread::INFO::2016-08-25 15:38:36,976::ovf_store::116::
>> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
>> OVF_STORE volume path: /rhev/data-center/mnt/glusterS
>> D/ccgl1.gl.local:HOST1/6a0bca4a-a1be-47d3-be51-64c627
>> 7d1f0f/images/c12c8000-0373-419b-963b-98b04adca760/fb6e250
>> 9-4786-433d-868f-a6303dd69cca
>> MainThread::INFO::2016-08-25 15:38:37,097::config::226::ovi
>> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>> Found an OVF for HE VM, trying to convert
>> MainThread::INFO::2016-08-25 15:38:37,102::config::231::ovi
>> rt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
>> Got vm.conf from OVF_STORE
>> MainThread::INFO::2016-08-25 15:38:37,200::hosted_engine::4
>> 62::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Current state EngineDown (score: 3400)
>> MainThread::INFO::2016-08-25 15:38:37,201::hosted_engine::4
>> 67::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
>> MainThread::INFO::2016-08-25 15:38:47,346::hosted_engine::6
>> 13::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
>> Initializing VDSM
>> MainThread::INFO::2016-08-25 15:38:47,439::hosted_engine::6
>> 58::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
>> Connecting the storage
>> MainThread::INFO::2016-08-25 15:38:47,454::storage_server::218::
>> ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>> Connecting storage server
>> MainThread::INFO::2016-08-25 15:38:47,618::storage_server::222::
>> ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
>> Connecting storage server
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.br
>> oker.listener.ConnectionHandler:Connection closed
>>
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
>> 16466
>> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: 
>> INFO:engine_health.CpuLoadNoEngine:VM
>> not on this host
>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: INFO:mgmt_bridge.MgmtBridge:Found
>> bridge ovirtmgmt with ports
>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: 
>> INFO:cpu_load_no_engine.EngineHealth:VM
>> not on this host
>> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: 
>> INFO:cpu_load_no_engine.EngineHealth:System
>> load total=0.1022, engine=0., non-engine=0.1022
>> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.ag
>> ent.hosted_engine.HostedEngine:Initializing VDSM
>> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.ag
>> 

Re: [ovirt-users] ovirt-ha-agent

[Renout Gerrits]

Depends on your systemd configuration. ovirt-ha-agent and broker daemon's
both log to stdout and it's own logfile. All messages to stdout will go to
journald and be forwarded to /var/log/messages (ForwardToSyslog=yes in
/etc/systemd/journald.conf I think).
So the ovirt-ha-agent doesn't log to /var/log/messages, journald does. if
it should log to stdout is another discussion, but maybe there's good
reason for that, backwards compatibility, don't know.

An easy fix is redirecting the output of the daemon to /dev/null. in
/usr/lib/systemd/system/ovirt-ha-agent.service add StandardOutput=null to
the [service] section.

Renout

On Thu, Aug 25, 2016 at 10:39 PM, David Gossage  wrote:

> This service seems to be logging to both /var/log/messages
> and /var/log/ovirt-hosted-engine-ha/agent.log
>
> Anything that may be causing that?  Centos7 ovirt 3.6.7
>
> MainThread::INFO::2016-08-25 15:38:36,912::ovf_store::109::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-08-25 15:38:36,976::ovf_store::116::
> ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ccgl1.gl.local:
> HOST1/6a0bca4a-a1be-47d3-be51-64c6277d1f0f/images/c12c8000-
> 0373-419b-963b-98b04adca760/fb6e2509-4786-433d-868f-a6303dd69cca
> MainThread::INFO::2016-08-25 15:38:37,097::config::226::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Found an OVF for HE VM, trying to
> convert
> MainThread::INFO::2016-08-25 15:38:37,102::config::231::
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.
> config::(refresh_local_conf_file) Got vm.conf from OVF_STORE
> MainThread::INFO::2016-08-25 15:38:37,200::hosted_engine::
> 462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineDown (score: 3400)
> MainThread::INFO::2016-08-25 15:38:37,201::hosted_engine::
> 467::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
> MainThread::INFO::2016-08-25 15:38:47,346::hosted_engine::
> 613::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_vdsm) Initializing VDSM
> MainThread::INFO::2016-08-25 15:38:47,439::hosted_engine::
> 658::ovirt_hosted_engine_ha.agent.hosted_engine.
> HostedEngine::(_initialize_storage_images) Connecting the storage
> MainThread::INFO::2016-08-25 15:38:47,454::storage_server::
> 218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2016-08-25 15:38:47,618::storage_server::
> 222::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:ovirt_hosted_engine_ha.
> broker.listener.ConnectionHandler:Connection closed
>
> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
> 16466
> Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: 
> INFO:engine_health.CpuLoadNoEngine:VM
> not on this host
> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: INFO:mgmt_bridge.MgmtBridge:Found
> bridge ovirtmgmt with ports
> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: 
> INFO:cpu_load_no_engine.EngineHealth:VM
> not on this host
> Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: 
> INFO:cpu_load_no_engine.EngineHealth:System
> load total=0.1022, engine=0., non-engine=0.1022
> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine:Initializing VDSM
> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine:Connecting the storage
> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer:Connecting storage server
> Aug 25 15:38:47 ccovirt3 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer:Connecting storage server
>
> *David Gossage*
> *Carousel Checks Inc. | System Administrator*
> *Office* 708.613.2284
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] ovirt-ha-agent

[David Gossage]

This service seems to be logging to both /var/log/messages
and /var/log/ovirt-hosted-engine-ha/agent.log

Anything that may be causing that?  Centos7 ovirt 3.6.7

MainThread::INFO::2016-08-25
15:38:36,912::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-08-25
15:38:36,976::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path:
/rhev/data-center/mnt/glusterSD/ccgl1.gl.local:HOST1/6a0bca4a-a1be-47d3-be51-64c6277d1f0f/images/c12c8000-0373-419b-963b-98b04adca760/fb6e2509-4786-433d-868f-a6303dd69cca
MainThread::INFO::2016-08-25
15:38:37,097::config::226::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-08-25
15:38:37,102::config::231::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Got vm.conf from OVF_STORE
MainThread::INFO::2016-08-25
15:38:37,200::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineDown (score: 3400)
MainThread::INFO::2016-08-25
15:38:37,201::hosted_engine::467::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host ccovirt1.carouselchecks.local (id: 1, score: 3400)
MainThread::INFO::2016-08-25
15:38:47,346::hosted_engine::613::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
MainThread::INFO::2016-08-25
15:38:47,439::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
MainThread::INFO::2016-08-25
15:38:47,454::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2016-08-25
15:38:47,618::storage_server::222::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:ovirt_hosted_engine_ha.broker.listener.ConnectionHandler:Connection
closed

Aug 25 15:38:40 ccovirt3 ovirt-ha-broker: INFO:mem_free.MemFree:memFree:
16466
Aug 25 15:38:40 ccovirt3 ovirt-ha-broker:
INFO:engine_health.CpuLoadNoEngine:VM not on this host
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker: INFO:mgmt_bridge.MgmtBridge:Found
bridge ovirtmgmt with ports
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:VM not on this host
Aug 25 15:38:45 ccovirt3 ovirt-ha-broker:
INFO:cpu_load_no_engine.EngineHealth:System load total=0.1022,
engine=0., non-engine=0.1022
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing
VDSM
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting the
storage
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server
Aug 25 15:38:47 ccovirt3 ovirt-ha-agent:
INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting
storage server

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent and too many open files error

[Gianluca Cecchi]

On Tue, Aug 9, 2016 at 6:54 PM, Simone Tiraboschi 
wrote:

>
>
> Ciao Gianluca,
> can you please report which vdsm version are using there?
> we had a similar issue in the past but it should be already solved:
> https://bugzilla.redhat.com/1343005
>
> > Gianluca
> >
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
>

Ciao,
the version installed at the moment is vdsm-4.18.4.1-0.el7.centos.x86_64
(what provided by version 4.0 of oVirt)
I see in referred bugzilla that it should be fixed in 4.0.1, the version
where I want to upgrade to...

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent and too many open files error

[Simone Tiraboschi]

On Tue, Aug 9, 2016 at 4:59 PM, Gianluca Cecchi
 wrote:
> Hello,
> I have a 4.0 test environment (single host with self hosted engine) where I
> have 6 VMs defined (5 running) and no much activity.
>
> I do't monitor this system very much.
>
> Now I have connected to it to evaluate upgrade to 4.0.1 and see that about
> 15 days ago the ovirt-ha-agent died because of too many open files
>
> [root@ractor ovirt-hosted-engine-ha]# systemctl status ovirt-ha-agent -l
> ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring
> Agent
>Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled;
> vendor preset: disabled)
>Active: inactive (dead) since Fri 2016-07-22 16:39:49 CEST; 2 weeks 4
> days ago
>  Main PID: 72795 (code=exited, status=0/SUCCESS)
>
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.set_file(fd)
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File
> "/usr/lib64/python2.7/asyncore.py", line 657, in set_file
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.socket =
> file_wrapper(fd)
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File
> "/usr/lib64/python2.7/asyncore.py", line 616, in __init__
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.fd = os.dup(fd)
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: OSError: [Errno 24]
> Too many open files
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Shutting down
> the agent because of 3 failures in a row!
> Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]:
> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Shutting down
> the agent because of 3 failures in a row!
> Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]:
> WARNING:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:The VM is
> running locally or we have no data, keeping the domain monitor.
> Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]:
> INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down
>
> Is this sort of known problem or any reason to investigate?
> It seems very strange to have reached this limit
>
> I presume the agent runs as vdsm user and that the oVirt installation
> creates the file
> /etc/security/limits.d/99-vdsm.conf
>
> with
> # This limits are intended for medium VDSM hosts, for large hosts scale
> these
> # numbers appropriately.
>
> # nproc should be the maximum amount of storage operations usage.
> # VMs run by "qemu" user, vm processes are not relavent to "vdsm" user
> limits.
> vdsm - nproc 4096
>
> # nofile should be at least 3(stdin,stdour,stderr) * each external process.
> # 3 * 4096 = 12288
> vdsm - nofile 12288
>
> As a rough estimation (over estimation actually , due to many duplicates) I
> have now:
> # lsof -u vdsm | wc -l
> 488
>
> Anything else to check?

Ciao Gianluca,
can you please report which vdsm version are using there?
we had a similar issue in the past but it should be already solved:
https://bugzilla.redhat.com/1343005

> Gianluca
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] ovirt-ha-agent and too many open files error

[Gianluca Cecchi]

Hello,
I have a 4.0 test environment (single host with self hosted engine) where I
have 6 VMs defined (5 running) and no much activity.

I do't monitor this system very much.

Now I have connected to it to evaluate upgrade to 4.0.1 and see that about
15 days ago the ovirt-ha-agent died because of too many open files

[root@ractor ovirt-hosted-engine-ha]# systemctl status ovirt-ha-agent -l
● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring
Agent
   Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled;
vendor preset: disabled)
   Active: inactive (dead) since Fri 2016-07-22 16:39:49 CEST; 2 weeks 4
days ago
 Main PID: 72795 (code=exited, status=0/SUCCESS)

Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.set_file(fd)
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File
"/usr/lib64/python2.7/asyncore.py", line 657, in set_file
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.socket =
file_wrapper(fd)
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: File
"/usr/lib64/python2.7/asyncore.py", line 616, in __init__
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: self.fd = os.dup(fd)
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: OSError: [Errno 24]
Too many open files
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Shutting down
the agent because of 3 failures in a row!
Jul 22 16:39:47 ractor.mydomain ovirt-ha-agent[72795]:
ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Shutting down
the agent because of 3 failures in a row!
Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]:
WARNING:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:The VM is
running locally or we have no data, keeping the domain monitor.
Jul 22 16:39:49 ractor.mydomain ovirt-ha-agent[72795]:
INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down

Is this sort of known problem or any reason to investigate?
It seems very strange to have reached this limit

I presume the agent runs as vdsm user and that the oVirt installation
creates the file
/etc/security/limits.d/99-vdsm.conf

with
# This limits are intended for medium VDSM hosts, for large hosts scale
these
# numbers appropriately.

# nproc should be the maximum amount of storage operations usage.
# VMs run by "qemu" user, vm processes are not relavent to "vdsm" user
limits.
vdsm - nproc 4096

# nofile should be at least 3(stdin,stdour,stderr) * each external process.
# 3 * 4096 = 12288
vdsm - nofile 12288

As a rough estimation (over estimation actually , due to many duplicates) I
have now:
# lsof -u vdsm | wc -l
488

Anything else to check?

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  wrote:

> Does the spikes correlates with info messages on extracting the ovf?
>
>>
>>
>>


yes, it seems so and it happens every 14-15 seconds

These are the lines I see scrolling in agent.log when I notice cpu spikes
in ovirt-ha-agent...

MainThread::INFO::2016-08-08
15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2016-08-08
15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain
MainThread::INFO::2016-08-08
15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images
MainThread::INFO::2016-08-08
15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images
MainThread::INFO::2016-08-08
15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-08-08
15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-08-08
15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
MainThread::INFO::2016-08-08
15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
volUUID:1a18851e-6858-401c-be6e-af14415034b5
MainThread::INFO::2016-08-08
15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-08-08
15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path: /rhev/data-center/mnt/ovirt01.lutwyn.org:
_SHE__DOMAIN/31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
MainThread::INFO::2016-08-08
15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-08-08
15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Got vm.conf from OVF_STORE
MainThread::INFO::2016-08-08
15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 3400)
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Roy Golan]

Does the spikes correlates with info messages on extracting the ovf?

On Aug 8, 2016 1:33 PM, "Gianluca Cecchi"  wrote:

> Hello,
> I have a test machine that is a nuc6 with an i5 and 32G of ram and SSD
> disks.
> It is configured as a single host environment with Self Hosted Engine VM.
> Both host and SHE are CentOS 7.2 and oVirt version is 3.6.6.2-1.el7
>
> I notice that having 3 VMs powered on and making nothing special (engine
> VM, a CentOS 7 VM and a Fedora 24 VM) the ovirt-ha-agent process on host
> often spikes its cpu usage.
> See for example this quick video with top command running on host that
> reflects what happens continuously.
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvYUVRMFlLVmxRdXM/
> view?usp=sharing
>
> Is it normal that ovirt-ha-agent consumes all this amount of cpu?
> Going into /var/log/ovirt-hosted-engine-ha/agent.log I see nothing
> special, only messages of type "INFO". The same for broker.log
>
> Thanks,
> Gianluca
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

Hello,
I have a test machine that is a nuc6 with an i5 and 32G of ram and SSD
disks.
It is configured as a single host environment with Self Hosted Engine VM.
Both host and SHE are CentOS 7.2 and oVirt version is 3.6.6.2-1.el7

I notice that having 3 VMs powered on and making nothing special (engine
VM, a CentOS 7 VM and a Fedora 24 VM) the ovirt-ha-agent process on host
often spikes its cpu usage.
See for example this quick video with top command running on host that
reflects what happens continuously.

https://drive.google.com/file/d/0BwoPbcrMv8mvYUVRMFlLVmxRdXM/view?usp=sharing

Is it normal that ovirt-ha-agent consumes all this amount of cpu?
Going into /var/log/ovirt-hosted-engine-ha/agent.log I see nothing special,
only messages of type "INFO". The same for broker.log

Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent keeps quitting - 4.0.0

[Yaniv Dary]

The other issue will be fixed in 4.0.2:
https://bugzilla.redhat.com/show_bug.cgi?id=1348907

Yaniv Dary
Technical Product Manager
Red Hat Israel Ltd.
34 Jerusalem Road
Building A, 4th floor
Ra'anana, Israel 4350109

Tel : +972 (9) 7692306
8272306
Email: yd...@redhat.com
IRC : ydary


On Sun, Jul 17, 2016 at 1:04 PM, Artyom Lukianov 
wrote:

> We had the bug related to this issue
> https://bugzilla.redhat.com/show_bug.cgi?id=1343005.
> It must be fixed in recent versions.
> Best Regards
>
> On Thu, Jul 14, 2016 at 8:14 PM, Gervais de Montbrun <
> gerv...@demontbrun.com> wrote:
>
>> Hey Folks,
>>
>> I upgraded my oVirt cluster from 3.6.7 to 4.0.0 yesterday and am
>> experiencing a bunch of issues.
>>
>> 1) I can't update the Compatibility Version to 4.0 because it tells me
>> that all my VMs have to be off to do so, but I have a hosted engine. I
>> found some info online about how you plan to fix this. Do we know if the
>> fix will be in 4.0.1?
>>
>> 2) More alarming... the ovirt-ha-agent keeps quitting. The agent.log
>> shows:
>>
>> MainThread::ERROR::2016-07-13
>> 16:38:57,100::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 16:39:02,104::config::122::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_load)
>> Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not
>> available [[Errno 24] Too many open files:
>> '/etc/ovirt-hosted-engine/hosted-engine.conf']
>> MainThread::ERROR::2016-07-13
>> 16:39:02,105::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 16:39:07,110::agent::210::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Too many errors occurred, giving up. Please review the log and consider
>> filing a bug.
>> MainThread::ERROR::2016-07-13
>> 17:44:03,499::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Shutting down the agent because of 3 failures in a row!
>> MainThread::ERROR::2016-07-13
>> 17:44:03,515::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '(24, 'Sanlock lockspace remove failure', 'Too many open files')' -
>> trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:08,520::config::122::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_load)
>> Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not
>> available [[Errno 24] Too many open files:
>> '/etc/ovirt-hosted-engine/hosted-engine.conf']
>> MainThread::ERROR::2016-07-13
>> 17:44:08,523::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:13,529::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:18,535::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:23,541::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:28,546::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:33,552::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:38,556::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:43,561::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:48,566::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Error: '[Errno 24] Too many open files' - trying to restart agent
>> MainThread::ERROR::2016-07-13
>> 17:44:53,571::agent::210::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
>> Too many errors occurred, giving up. Please review the log and consider
>> filing a bug.
>> MainThread::ERROR::2016-07-13
>> 18:47:40,048::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Shutting down the agent because of 3 failures in a row!
>> MainThread::ERROR::2016-07-14
>> 10:32:29,184::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
>> Shutting down the agent because of 3 failures in a row!
>> 

Re: [ovirt-users] ovirt-ha-agent keeps quitting - 4.0.0

[Artyom Lukianov]

We had the bug related to this issue
https://bugzilla.redhat.com/show_bug.cgi?id=1343005.
It must be fixed in recent versions.
Best Regards

On Thu, Jul 14, 2016 at 8:14 PM, Gervais de Montbrun  wrote:

> Hey Folks,
>
> I upgraded my oVirt cluster from 3.6.7 to 4.0.0 yesterday and am
> experiencing a bunch of issues.
>
> 1) I can't update the Compatibility Version to 4.0 because it tells me
> that all my VMs have to be off to do so, but I have a hosted engine. I
> found some info online about how you plan to fix this. Do we know if the
> fix will be in 4.0.1?
>
> 2) More alarming... the ovirt-ha-agent keeps quitting. The agent.log shows:
>
> MainThread::ERROR::2016-07-13
> 16:38:57,100::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 16:39:02,104::config::122::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_load)
> Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not
> available [[Errno 24] Too many open files:
> '/etc/ovirt-hosted-engine/hosted-engine.conf']
> MainThread::ERROR::2016-07-13
> 16:39:02,105::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 16:39:07,110::agent::210::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Too many errors occurred, giving up. Please review the log and consider
> filing a bug.
> MainThread::ERROR::2016-07-13
> 17:44:03,499::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Shutting down the agent because of 3 failures in a row!
> MainThread::ERROR::2016-07-13
> 17:44:03,515::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '(24, 'Sanlock lockspace remove failure', 'Too many open files')' -
> trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:08,520::config::122::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_load)
> Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not
> available [[Errno 24] Too many open files:
> '/etc/ovirt-hosted-engine/hosted-engine.conf']
> MainThread::ERROR::2016-07-13
> 17:44:08,523::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:13,529::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:18,535::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:23,541::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:28,546::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:33,552::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:38,556::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:43,561::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:48,566::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: '[Errno 24] Too many open files' - trying to restart agent
> MainThread::ERROR::2016-07-13
> 17:44:53,571::agent::210::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Too many errors occurred, giving up. Please review the log and consider
> filing a bug.
> MainThread::ERROR::2016-07-13
> 18:47:40,048::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Shutting down the agent because of 3 failures in a row!
> MainThread::ERROR::2016-07-14
> 10:32:29,184::hosted_engine::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Shutting down the agent because of 3 failures in a row!
> MainThread::ERROR::2016-07-14
> 11:10:07,223::brokerlink::279::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate)
> Connection closed: Connection closed
> MainThread::ERROR::2016-07-14
> 11:10:07,224::brokerlink::148::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(get_monitor_status)
> Exception getting monitor status: Connection closed
> MainThread::ERROR::2016-07-14
> 11:10:07,224::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Error: