Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 6:14 PM, Gianluca Cecchi 
wrote:

> On Wed, Apr 26, 2017 at 4:54 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi <
>> gianluca.cec...@gmail.com> wrote:
>>
>>> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
>>> wrote:
>>>



 On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
 connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
 and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
 kages/vdsm/api/vdsmapi.py

 So it's still the parsing of the api yaml schema.
 On master we already merged a patch to reuse an existing connection if
 available and this should mitigate/resolve the issue:
 https://gerrit.ovirt.org/73757/

 It's still not that clear why we are facing this kind of performance
 regression.


>>> Does this mean that I could try to patch the
>>>   ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
>>> verify if it solves...?
>>>
>>
>> Unfortunately it's not enough by itself since it also requires
>> https://gerrit.ovirt.org/#/c/58029/
>>
>>
>>> Or do I have to wait an official patched rpm?
>>>
>>> Gianluca
>>>
>>
>>
> And what if I change all the 3 files involved in the 2 gerrit entries:
> ovirt_hosted_engine_ha/lib/util.py
> lib/yajsonrpc/stomp.py
> lib/yajsonrpc/stompreactor.py
>
> ?
>
>
Yes, it should work although we never officially back-ported and tested it
for 4.1.z.
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Wed, Apr 26, 2017 at 4:54 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>>
>>> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
>>> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
>>> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
>>> kages/vdsm/api/vdsmapi.py
>>>
>>> So it's still the parsing of the api yaml schema.
>>> On master we already merged a patch to reuse an existing connection if
>>> available and this should mitigate/resolve the issue:
>>> https://gerrit.ovirt.org/73757/
>>>
>>> It's still not that clear why we are facing this kind of performance
>>> regression.
>>>
>>>
>> Does this mean that I could try to patch the
>>   ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
>> verify if it solves...?
>>
>
> Unfortunately it's not enough by itself since it also requires
> https://gerrit.ovirt.org/#/c/58029/
>
>
>> Or do I have to wait an official patched rpm?
>>
>> Gianluca
>>
>
>
And what if I change all the 3 files involved in the 2 gerrit entries:
ovirt_hosted_engine_ha/lib/util.py
lib/yajsonrpc/stomp.py
lib/yajsonrpc/stompreactor.py

?
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 4:52 PM, Gianluca Cecchi 
wrote:

> On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>>
>> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
>> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
>> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-pac
>> kages/vdsm/api/vdsmapi.py
>>
>> So it's still the parsing of the api yaml schema.
>> On master we already merged a patch to reuse an existing connection if
>> available and this should mitigate/resolve the issue:
>> https://gerrit.ovirt.org/73757/
>>
>> It's still not that clear why we are facing this kind of performance
>> regression.
>>
>>
> Does this mean that I could try to patch the   
> ovirt_hosted_engine_ha/lib/util.py
> file also in my 4.1.1 install and verify if it solves...?
>

Unfortunately it's not enough by itself since it also requires
https://gerrit.ovirt.org/#/c/58029/


> Or do I have to wait an official patched rpm?
>
> Gianluca
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Wed, Apr 26, 2017 at 4:28 PM, Simone Tiraboschi 
wrote:

>
>
>
> On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
> connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
> and the 95.98% is in Schema.__init__ in /usr/lib/python2.7/site-
> packages/vdsm/api/vdsmapi.py
>
> So it's still the parsing of the api yaml schema.
> On master we already merged a patch to reuse an existing connection if
> available and this should mitigate/resolve the issue:
> https://gerrit.ovirt.org/73757/
>
> It's still not that clear why we are facing this kind of performance
> regression.
>
>
Does this mean that I could try to patch the
  ovirt_hosted_engine_ha/lib/util.py file also in my 4.1.1 install and
verify if it solves...?
Or do I have to wait an official patched rpm?

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 1:28 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer  wrote:
>
>> On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <
>> gianluca.cec...@gmail.com> wrote:
>>
>>> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>>>


 Hi Gianluca,

 You can run this on the host:

 $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
 CLoader: True

 If you get "CLoader: False", you have some packaging issue, CLoader
 is available on all supported platforms.

 Nir


> Thanks,
> Gianluca
>
>
>>>
>>> It seems ok.
>>>
>>> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
>>> 'CLoader:', hasattr(yaml, 'CLoader')"
>>> CLoader: True
>>> [root@ovirt01 ovirt-hosted-engine-ha]#
>>>
>>> Anyway see here a sample of the spikes that it cntinues to have.. from
>>> 15% to 55% many times
>>> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE
>>> /view?usp=sharing
>>>
>>
>> There are two issues in this video:
>> - Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
>> that it needs so much memory.
>> - Unusual cpu usage - but not the kind of usage related to yaml parsing.
>>
>> I would open two bugs for this. We have seen the first issue few month
>> ago, and
>> we did nothing about it so the memory leak was not fixed.
>>
>> To understand the unusual cpu usage, we need to integrate yappi into
>> ovirt-ha-agent,
>> and do some profiling to understand where cpu time is spent.
>>
>> Simone, can you do something based on these patches?
>> https://gerrit.ovirt.org/#/q/topic:generic-profiler
>>
>> I hope to get these patches merged soon.
>>
>>
> Absolutely at this point.
>
>

On 4.1.1, the 96% of the cpu time of ovirt-ha-agent is still spent in
connect() in /usr/lib/python2.7/site-packages/vdsm/jsonrpcvdscli.py
and the 95.98% is in Schema.__init__
in /usr/lib/python2.7/site-packages/vdsm/api/vdsmapi.py

So it's still the parsing of the api yaml schema.
On master we already merged a patch to reuse an existing connection if
available and this should mitigate/resolve the issue:
https://gerrit.ovirt.org/73757/

It's still not that clear why we are facing this kind of performance
regression.



> Nir
>>
>>
>>>
>>>
>>> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an
>>> F25 guest and a C7 desktop VMs running, without doing almost anything.
>>>
>>> Gianluca
>>>
>>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Apr 26, 2017 at 12:52 PM, Nir Soffer  wrote:

> On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi <
> gianluca.cec...@gmail.com> wrote:
>
>> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>>
>>>
>>>
>>> Hi Gianluca,
>>>
>>> You can run this on the host:
>>>
>>> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
>>> CLoader: True
>>>
>>> If you get "CLoader: False", you have some packaging issue, CLoader
>>> is available on all supported platforms.
>>>
>>> Nir
>>>
>>>
 Thanks,
 Gianluca


>>
>> It seems ok.
>>
>> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
>> 'CLoader:', hasattr(yaml, 'CLoader')"
>> CLoader: True
>> [root@ovirt01 ovirt-hosted-engine-ha]#
>>
>> Anyway see here a sample of the spikes that it cntinues to have.. from
>> 15% to 55% many times
>> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/
>> view?usp=sharing
>>
>
> There are two issues in this video:
> - Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
> that it needs so much memory.
> - Unusual cpu usage - but not the kind of usage related to yaml parsing.
>
> I would open two bugs for this. We have seen the first issue few month
> ago, and
> we did nothing about it so the memory leak was not fixed.
>
> To understand the unusual cpu usage, we need to integrate yappi into
> ovirt-ha-agent,
> and do some profiling to understand where cpu time is spent.
>
> Simone, can you do something based on these patches?
> https://gerrit.ovirt.org/#/q/topic:generic-profiler
>
> I hope to get these patches merged soon.
>
>
Absolutely at this point.


> Nir
>
>
>>
>>
>> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an
>> F25 guest and a C7 desktop VMs running, without doing almost anything.
>>
>> Gianluca
>>
>>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Apr 26, 2017 at 11:36 AM Gianluca Cecchi 
wrote:

> On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:
>
>>
>>
>> Hi Gianluca,
>>
>> You can run this on the host:
>>
>> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
>> CLoader: True
>>
>> If you get "CLoader: False", you have some packaging issue, CLoader
>> is available on all supported platforms.
>>
>> Nir
>>
>>
>>> Thanks,
>>> Gianluca
>>>
>>>
>
> It seems ok.
>
> [root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
> 'CLoader:', hasattr(yaml, 'CLoader')"
> CLoader: True
> [root@ovirt01 ovirt-hosted-engine-ha]#
>
> Anyway see here a sample of the spikes that it cntinues to have.. from 15%
> to 55% many times
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/view?usp=sharing
>

There are two issues in this video:
- Memory leak, ovirt-ha-agent is using 1g of memory. It is very unlikely
that it needs so much memory.
- Unusual cpu usage - but not the kind of usage related to yaml parsing.

I would open two bugs for this. We have seen the first issue few month ago,
and
we did nothing about it so the memory leak was not fixed.

To understand the unusual cpu usage, we need to integrate yappi into
ovirt-ha-agent,
and do some profiling to understand where cpu time is spent.

Simone, can you do something based on these patches?
https://gerrit.ovirt.org/#/q/topic:generic-profiler

I hope to get these patches merged soon.

Nir


>
>
> The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an F25
> guest and a C7 desktop VMs running, without doing almost anything.
>
> Gianluca
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Tue, Apr 25, 2017 at 11:28 PM, Nir Soffer  wrote:

>
>
> Hi Gianluca,
>
> You can run this on the host:
>
> $ python -c "import yaml; print 'CLoader:', hasattr(yaml, 'CLoader')"
> CLoader: True
>
> If you get "CLoader: False", you have some packaging issue, CLoader
> is available on all supported platforms.
>
> Nir
>
>
>> Thanks,
>> Gianluca
>>
>>

It seems ok.

[root@ovirt01 ovirt-hosted-engine-ha]#  python -c "import yaml; print
'CLoader:', hasattr(yaml, 'CLoader')"
CLoader: True
[root@ovirt01 ovirt-hosted-engine-ha]#

Anyway see here a sample of the spikes that it cntinues to have.. from 15%
to 55% many times
https://drive.google.com/file/d/0BwoPbcrMv8mvMy1xVUE3YzI2YVE/view?usp=sharing

The host is an Intel NUC6i5 with 32Gb of ram. There are the engine, an F25
guest and a C7 desktop VMs running, without doing almost anything.

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi 
wrote:

>
>
>
> If I can apply the patch also to 4.0.3 I'm going to see if there is then a
> different behavior.
> Let me know,
>
>
> I'm trying it right now.
> Any other tests will be really appreciated.
>
> The patch is pretty simply, you can apply that on the fly.
> You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
> directly edit
> /usr/lib/python2.7/site-packages/api/vdsmapi.py
> around line 97 changing from
> loaded_schema = yaml.load(f)
> to
> loaded_schema = yaml.load(f, Loader=yaml.CLoader)
> Please pay attention to keep exactly the same amount of initial spaces.
>
> Then you can simply restart the HA agent and check.
>
>

Hello,
I'm again registering high spikes of ovirt-ha-agent with only 2-3 VMs up
and with almost no activity
The package of the involved file
/usr/lib/python2.7/site-packages/api/vdsmapi.py
is now at veriosn vdsm-api-4.19.4-1.el7.centos.noarch and I see that the
file contains this kind of lines

129 try:
130 for path in paths:
131 with open(path) as f:
132 if hasattr(yaml, 'CLoader'):
133 loader = yaml.CLoader
134 else:
135 loader = yaml.Loader
136 loaded_schema = yaml.load(f, Loader=loader)
137
138 types = loaded_schema.pop('types')
139 self._types.update(types)
140 self._methods.update(loaded_schema)
141 except EnvironmentError:
142 raise SchemaNotFound("Unable to find API schema file")

So there is a conditional statement...
How can I be sure that "loader" is set to "yaml.CLoader" that was what in
4.0 was able to lower the cpu usage of ovirt-ha-agent?
Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Sam Cappello]

it worked for me as well - load avg. < 1 now, ovirt-ha-agent pops up in 
top periodically, but not on top using 100% CPU all the time anymore.

thanks!
sam

Gianluca Cecchi wrote on 10/7/2016 10:13 AM:



On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi > wrote:




On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi
> wrote:

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer > wrote:


wasn’t it suppose to be fixed to reuse the connection?
Like all the other clients (vdsm migration code:-)


This is orthogonal issue.

Does schema validation matter then if there would be
only one connection at the start up?


Loading once does not help command line tools like
vdsClient, hosted-engine and
vdsm-tool.

Nir




Simone, reusing the connection is good idea
anyway, but what you describe is
a bug in the client library. The library does
*not* need to load and parse the
schema at all for sending requests to vdsm.

The schema is only needed if you want to
verify request parameters,
or provide online help, these are not needed
in a client library.

Please file an infra bug about it.


Done,
https://bugzilla.redhat.com/show_bug.cgi?id=1381899



Here is a patch that should eliminate most most of
the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system
showing this problem.

Cheers,
Nir
___




this is a video of 1 minute with the same system as the first
post, but in 4.0.3 now and the same 3 VMs powered on without
any particular load.
It seems very similar to the previous 3.6.6 in cpu used by
ovirt-ha-agent.


https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing



Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if
there is then a different behavior.
Let me know,


I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you
could directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial
spaces.

Then you can simply restart the HA agent and check.

Gianluca

___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users





What I've done (I didn't read your answer in between and this is a 
test system not so important... )


set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process 
or at least has ranges between 5% and 12% and not more


BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; 
vendor preset: enabled)

   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh 
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)

 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 
--write-pipe-fd 40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 
--write-pipe-fd 46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 
--write-pipe-fd 56 --max-threads 10 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 07 Oct 2016, at 16:10, Simone Tiraboschi  wrote:
> 
> 
> 
>> On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>>> 
>>> 
>>> 
>>> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek 
>>>  wrote:
 
> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>> 
 On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
  wrote:
 
 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  
> wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>  wrote:
>> 
>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  
>>> wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there 
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically 
>> reconnects over json rpc and this is CPU intensive since the client 
>> has to parse the yaml API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the 
>> other clients (vdsm migration code:-) 
> 
> This is orthogonal issue.
 
 Yes it is. And that’s the issue;-)
 Both are wrong, but by “fixing” the schema validation only you lose the 
 motivation to fix the meaningless wasteful reconnect
>>> 
>>> Yes, we are going to fix that too ( 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 )
>> 
>> that’s great! Also al the other vdsClient uses?:-)
> 
> https://gerrit.ovirt.org/#/c/62729/

Cool. It's not so important for one time actions but we need it to be able to 
drop xmlrpc finally, so it is important:)

>  
>> What is that periodic one call anyway? Is there only one? Maybe we don’t 
>> need it so much.
> 
> Currently ovirt-ha-agent is periodically reconnecting the hosted-engine 
> storage domain and checking its status. This is already on jsonrpc.

Ok. As long as it is low frequency it's ok. Just bear in mind for future that 
some of the vdsm calls may be heavy and impact performance, regardless the 
communication layer. 

Thanks and have a nice weekend,
michal

> In 4.1 all the monitoring will be moved to jsonrpc.
>  
>> 
>>> but it would require also 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 to be fixed.
>> 
>> This is less good. Well, worst case you can reconnect yourself, all you need 
>> is a notification when the existing connection breaks
>> 
>>>  
 
>  
>> Does schema validation matter then if there would be only one connection 
>> at the start up?
> 
> Loading once does not help command line tools like vdsClient, 
> hosted-engine and
> vdsm-tool. 
 
 none of the other tools is using json-rpc.
>>> 
>>> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
>>> remaining tools since xmlrpc has been deprecated with 4.0
>> 
>> ok. though setup is a one-time action so it’s not an issue there
>> 
>>>  
 
> 
> Nir
>  
>> 
> 
> Simone, reusing the connection is good idea anyway, but what you 
> describe is 
> a bug in the client library. The library does *not* need to load and 
> parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
 
 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
>>> 
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

>
> And a bright new 3-minutes video here:
> https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/
> view?usp=sharing
>

> It seems that now ovirt-ha-agent or is not present in top cpu process or
> at least has ranges between 5% and 12% and not more
>

5-12% is probably 10 times more than needed for the agent, profiling
should tell us where time is spent.

Since the agent depends on vdsm, we can reuse vdsm cpu profiler, see
lib/vdsm/profiling/cpu.py. It is not ready yet to be used by other
applications,
but making it more general should be easy.

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi 
wrote:

>
>
> On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi  > wrote:
>
>> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>>
>>>
 wasn’t it suppose to be fixed to reuse the connection? Like all the
 other clients (vdsm migration code:-)

>>>
>>> This is orthogonal issue.
>>>
>>>
 Does schema validation matter then if there would be only one
 connection at the start up?

>>>
>>> Loading once does not help command line tools like vdsClient,
>>> hosted-engine and
>>> vdsm-tool.
>>>
>>> Nir
>>>
>>>


>> Simone, reusing the connection is good idea anyway, but what you
>> describe is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

 Here is a patch that should eliminate most most of the problem:
 https://gerrit.ovirt.org/65230

 Would be nice if it can be tested on the system showing this problem.

 Cheers,
 Nir
 ___


>>
>> this is a video of 1 minute with the same system as the first post, but
>> in 4.0.3 now and the same 3 VMs powered on without any particular load.
>> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>>
>> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8
>> /view?usp=sharing
>>
>> Enjoy Nir ;-)
>>
>> If I can apply the patch also to 4.0.3 I'm going to see if there is then
>> a different behavior.
>> Let me know,
>>
>>
> I'm trying it right now.
> Any other tests will be really appreciated.
>
> The patch is pretty simply, you can apply that on the fly.
> You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
> directly edit
> /usr/lib/python2.7/site-packages/api/vdsmapi.py
> around line 97 changing from
> loaded_schema = yaml.load(f)
> to
> loaded_schema = yaml.load(f, Loader=yaml.CLoader)
> Please pay attention to keep exactly the same amount of initial spaces.
>
> Then you can simply restart the HA agent and check.
>
> Gianluca
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>

What I've done (I didn't read your answer in between and this is a test
system not so important... )

set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process or at
least has ranges between 5% and 12% and not more

BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: enabled)
   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 --write-pipe-fd
40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd
56 --max-threads 10 --max-queue...
   ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
64 --max-threads 10 --max-queue...
   ├─21149 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
72 --max-threads 10 --max-queue...
   ├─21156 /usr/libexec/ioprocess --read-pipe-fd 80 --write-pipe-fd
78 --max-threads 10 --max-queue...
   ├─21177 /usr/libexec/ioprocess --read-pipe-fd 88 --write-pipe-fd
87 --max-threads 10 --max-queue...
   ├─21204 /usr/libexec/ioprocess --read-pipe-fd 99 --write-pipe-fd
98 --max-threads 10 --max-queue...
   └─21239 /usr/libexec/ioprocess --read-pipe-fd 111
--write-pipe-fd 110 --max-threads 10 --max-que...

Oct 07 16:02:52 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:54 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:56 ovirt01.lutwyn.org vdsm[21023]: vdsm 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>
>
>
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek <
> michal.skriva...@redhat.com> wrote:
>
>>
>> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
>>
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
>> michal.skriva...@redhat.com> wrote:
>>
>>>
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>>
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi <
> stira...@redhat.com> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor 
>> wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>> Yes it is. And that’s the issue;-)
>> Both are wrong, but by “fixing” the schema validation only you lose the
>> motivation to fix the meaningless wasteful reconnect
>>
>
> Yes, we are going to fix that too ( https://bugzilla.redhat.com/
> show_bug.cgi?id=1349829 )
>
>
> that’s great! Also al the other vdsClient uses?:-)
>

https://gerrit.ovirt.org/#/c/62729/


> What is that periodic one call anyway? Is there only one? Maybe we don’t
> need it so much.
>

Currently ovirt-ha-agent is periodically reconnecting the hosted-engine
storage domain and checking its status. This is already on jsonrpc.
In 4.1 all the monitoring will be moved to jsonrpc.


>
> but it would require also https://bugzilla.redhat.com/
> show_bug.cgi?id=1376843 to be fixed.
>
>
> This is less good. Well, worst case you can reconnect yourself, all you
> need is a notification when the existing connection breaks
>
>
>
>>
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>>
>> none of the other tools is using json-rpc.
>>
>
> hosted-engine-setup is, and sooner or later we'll have to migrate also the
> remaining tools since xmlrpc has been deprecated with 4.0
>
>
> ok. though setup is a one-time action so it’s not an issue there
>
>
>
>>
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
> 
> 
> 
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:59, Nir Soffer > > wrote:
>> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>> > wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer >> > wrote:
>>> 
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer >> > wrote:
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor >> > wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>> 
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>>> json rpc and this is CPU intensive since the client has to parse the yaml 
>>> API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
>> clients (vdsm migration code:-) 
>> 
>> This is orthogonal issue.
> 
> Yes it is. And that’s the issue;-)
> Both are wrong, but by “fixing” the schema validation only you lose the 
> motivation to fix the meaningless wasteful reconnect
> 
> Yes, we are going to fix that too ( 
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 
>  )

that’s great! Also al the other vdsClient uses?:-)
What is that periodic one call anyway? Is there only one? Maybe we don’t need 
it so much.

> but it would require also https://bugzilla.redhat.com/show_bug.cgi?id=1376843 
>  to be fixed.

This is less good. Well, worst case you can reconnect yourself, all you need is 
a notification when the existing connection breaks

>  
> 
>>  
>> Does schema validation matter then if there would be only one connection at 
>> the start up?
>> 
>> Loading once does not help command line tools like vdsClient, hosted-engine 
>> and
>> vdsm-tool. 
> 
> none of the other tools is using json-rpc.
> 
> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
> remaining tools since xmlrpc has been deprecated with 4.0

ok. though setup is a one-time action so it’s not an issue there

>  
> 
>> 
>> Nir
>>  
>> 
>>> 
>>> Simone, reusing the connection is good idea anyway, but what you describe 
>>> is 
>>> a bug in the client library. The library does *not* need to load and parse 
>>> the
>>> schema at all for sending requests to vdsm.
>>> 
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>> 
>>> Please file an infra bug about it.
>>> 
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>>> 
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230 
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org 
>>> http://lists.ovirt.org/mailman/listinfo/users 
>>> 
>> 
>> 
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users 
> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi 
wrote:

> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>
>>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>>
>>>
>
> this is a video of 1 minute with the same system as the first post, but in
> 4.0.3 now and the same 3 VMs powered on without any particular load.
> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/
> view?usp=sharing
>
> Enjoy Nir ;-)
>
> If I can apply the patch also to 4.0.3 I'm going to see if there is then a
> different behavior.
> Let me know,
>
>
I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial spaces.

Then you can simply restart the HA agent and check.

Gianluca
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:42, Nir Soffer > > wrote:
>> 
>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer > > wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor > > wrote:
>> Hi,
>> 
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>> json rpc and this is CPU intensive since the client has to parse the yaml 
>> API specification each time it connects.
> 
> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
> clients (vdsm migration code:-) 
> 
> This is orthogonal issue.

Yes it is. And that’s the issue;-)
Both are wrong, but by “fixing” the schema validation only you lose the 
motivation to fix the meaningless wasteful reconnect

>  
> Does schema validation matter then if there would be only one connection at 
> the start up?
> 
> Loading once does not help command line tools like vdsClient, hosted-engine 
> and
> vdsm-tool. 

none of the other tools is using json-rpc.

> 
> Nir
>  
> 
>> 
>> Simone, reusing the connection is good idea anyway, but what you describe is 
>> a bug in the client library. The library does *not* need to load and parse 
>> the
>> schema at all for sending requests to vdsm.
>> 
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>> 
>> Please file an infra bug about it.
>> 
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>> 
>> 
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230 
>> 
>> Would be nice if it can be tested on the system showing this problem.
>> 
>> Cheers,
>> Nir
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:

>
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other
>> clients (vdsm migration code:-)
>>
>
> This is orthogonal issue.
>
>
>> Does schema validation matter then if there would be only one connection
>> at the start up?
>>
>
> Loading once does not help command line tools like vdsClient,
> hosted-engine and
> vdsm-tool.
>
> Nir
>
>
>>
>>
 Simone, reusing the connection is good idea anyway, but what you
 describe is
 a bug in the client library. The library does *not* need to load and
 parse the
 schema at all for sending requests to vdsm.

 The schema is only needed if you want to verify request parameters,
 or provide online help, these are not needed in a client library.

 Please file an infra bug about it.

>>>
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>>
>>
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230
>>
>> Would be nice if it can be tested on the system showing this problem.
>>
>> Cheers,
>> Nir
>> ___
>>
>>

this is a video of 1 minute with the same system as the first post, but in
4.0.3 now and the same 3 VMs powered on without any particular load.
It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.

https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing

Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if there is then a
different behavior.
Let me know,

Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>>
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

 Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
 over json rpc and this is CPU intensive since the client has to parse the
 yaml API specification each time it connects.

>>>
> wasn’t it suppose to be fixed to reuse the connection? Like all the other
> clients (vdsm migration code:-)
>

This is orthogonal issue.


> Does schema validation matter then if there would be only one connection
> at the start up?
>

Loading once does not help command line tools like vdsClient, hosted-engine
and
vdsm-tool.

Nir


>
>
>>> Simone, reusing the connection is good idea anyway, but what you
>>> describe is
>>> a bug in the client library. The library does *not* need to load and
>>> parse the
>>> schema at all for sending requests to vdsm.
>>>
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>>
>>> Please file an infra bug about it.
>>>
>>
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>
>
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230
>
> Would be nice if it can be tested on the system showing this problem.
>
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
> 
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  > wrote:
> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  > wrote:
> Hi,
> 
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
> 
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
> json rpc and this is CPU intensive since the client has to parse the yaml API 
> specification each time it connects.

wasn’t it suppose to be fixed to reuse the connection? Like all the other 
clients (vdsm migration code:-) 
Does schema validation matter then if there would be only one connection at the 
start up?

> 
> Simone, reusing the connection is good idea anyway, but what you describe is 
> a bug in the client library. The library does *not* need to load and parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
> 
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
> 
> 
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230 
> 
> Would be nice if it can be tested on the system showing this problem.
> 
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>>
 Hi,

 did you found a solution or cause for this high CPU usage?
 I have installed the self hosted engine on another server and there is
 no VM running but ovirt-ha-agent uses heavily the CPU.

>>>
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>>> over json rpc and this is CPU intensive since the client has to parse the
>>> yaml API specification each time it connects.
>>>
>>
>> Simone, reusing the connection is good idea anyway, but what you describe
>> is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

Here is a patch that should eliminate most most of the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system showing this problem.

Cheers,
Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
> Simone, reusing the connection is good idea anyway, but what you describe
> is
> a bug in the client library. The library does *not* need to load and parse
> the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
Thanks.


> Nir
>
>
>> The issue is tracked here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
>> should reuse json-rpc connections
>> but it depends on:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
>> keep-alive with reconnect if needed logic for the python jsonrpc client
>>
>>
>>
>>>
>>> cheers
>>> gregor
>>>
>>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan >> > > wrote:
>>> >
>>> > Does the spikes correlates with info messages on extracting the
>>> ovf?
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > yes, it seems so and it happens every 14-15 seconds
>>> >
>>> > These are the lines I see scrolling in agent.log when I notice cpu
>>> > spikes in ovirt-ha-agent...
>>> >
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Connecting storage server
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Refreshing the storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.I
>>> mage::(prepare_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Reloading vm.conf from the shared storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > Extracting Engine VM OVF from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > OVF_STORE volume path:
>>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab2
>>> 2-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Found an OVF for HE VM, trying to convert
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Got vm.conf from OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> > Current state EngineUp (score: 3400)
>>> >
>>> >
>>> > ___
>>> > Users mailing list
>>> > Users@ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>> 

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>
>> Hi,
>>
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>
>
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
> json rpc and this is CPU intensive since the client has to parse the yaml
> API specification each time it connects.
>

Simone, reusing the connection is good idea anyway, but what you describe
is
a bug in the client library. The library does *not* need to load and parse
the
schema at all for sending requests to vdsm.

The schema is only needed if you want to verify request parameters,
or provide online help, these are not needed in a client library.

Please file an infra bug about it.

Nir


> The issue is tracked here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
> should reuse json-rpc connections
> but it depends on:
> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
> keep-alive with reconnect if needed logic for the python jsonrpc client
>
>
>
>>
>> cheers
>> gregor
>>
>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan > > > wrote:
>> >
>> > Does the spikes correlates with info messages on extracting the ovf?
>> >
>> >
>> >
>> >
>> >
>> >
>> > yes, it seems so and it happens every 14-15 seconds
>> >
>> > These are the lines I see scrolling in agent.log when I notice cpu
>> > spikes in ovirt-ha-agent...
>> >
>> > MainThread::INFO::2016-08-08
>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Connecting storage server
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Refreshing the storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.
>> Image::(prepare_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Reloading vm.conf from the shared storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > Extracting Engine VM OVF from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > OVF_STORE volume path:
>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-
>> ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Found an OVF for HE VM, trying to convert
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Got vm.conf from OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(start_monitoring)
>> > Current state EngineUp (score: 3400)
>> >
>> >
>> > ___
>> > Users mailing list
>> > Users@ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
json rpc and this is CPU intensive since the client has to parse the yaml
API specification each time it connects.
The issue is tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent should
reuse json-rpc connections
but it depends on:
https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
keep-alive with reconnect if needed logic for the python jsonrpc client


>
> cheers
> gregor
>
> On 08/08/16 15:09, Gianluca Cecchi wrote:
> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > > wrote:
> >
> > Does the spikes correlates with info messages on extracting the ovf?
> >
> >
> >
> >
> >
> >
> > yes, it seems so and it happens every 14-15 seconds
> >
> > These are the lines I see scrolling in agent.log when I notice cpu
> > spikes in ovirt-ha-agent...
> >
> > MainThread::INFO::2016-08-08
> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Connecting storage server
> > MainThread::INFO::2016-08-08
> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Refreshing the storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.
> image.Image::(prepare_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Reloading vm.conf from the shared storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Trying to get a fresher copy of vm configuration from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> > MainThread::INFO::2016-08-08
> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > Extracting Engine VM OVF from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > OVF_STORE volume path:
> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/
> 31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-
> 01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Found an OVF for HE VM, trying to convert
> > MainThread::INFO::2016-08-08
> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Got vm.conf from OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(start_monitoring)
> > Current state EngineUp (score: 3400)
> >
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[gregor]

Hi,

did you found a solution or cause for this high CPU usage?
I have installed the self hosted engine on another server and there is
no VM running but ovirt-ha-agent uses heavily the CPU.

cheers
gregor

On 08/08/16 15:09, Gianluca Cecchi wrote:
> On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > wrote:
> 
> Does the spikes correlates with info messages on extracting the ovf?
> 
> 
> 
> 
> 
> 
> yes, it seems so and it happens every 14-15 seconds
> 
> These are the lines I see scrolling in agent.log when I notice cpu
> spikes in ovirt-ha-agent...
> 
> MainThread::INFO::2016-08-08
> 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2016-08-08
> 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2016-08-08
> 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2016-08-08
> 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> MainThread::INFO::2016-08-08
> 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> volUUID:1a18851e-6858-401c-be6e-af14415034b5
> MainThread::INFO::2016-08-08
> 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>  
> MainThread::INFO::2016-08-08
> 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2016-08-08
> 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  wrote:

> Does the spikes correlates with info messages on extracting the ovf?
>
>>
>>
>>


yes, it seems so and it happens every 14-15 seconds

These are the lines I see scrolling in agent.log when I notice cpu spikes
in ovirt-ha-agent...

MainThread::INFO::2016-08-08
15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2016-08-08
15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain
MainThread::INFO::2016-08-08
15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images
MainThread::INFO::2016-08-08
15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images
MainThread::INFO::2016-08-08
15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Reloading vm.conf from the shared storage domain
MainThread::INFO::2016-08-08
15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2016-08-08
15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
MainThread::INFO::2016-08-08
15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
volUUID:1a18851e-6858-401c-be6e-af14415034b5
MainThread::INFO::2016-08-08
15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2016-08-08
15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
OVF_STORE volume path: /rhev/data-center/mnt/ovirt01.lutwyn.org:
_SHE__DOMAIN/31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
MainThread::INFO::2016-08-08
15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Found an OVF for HE VM, trying to convert
MainThread::INFO::2016-08-08
15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Got vm.conf from OVF_STORE
MainThread::INFO::2016-08-08
15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 3400)
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] ovirt-ha-agent cpu usage

[Roy Golan]

Does the spikes correlates with info messages on extracting the ovf?

On Aug 8, 2016 1:33 PM, "Gianluca Cecchi"  wrote:

> Hello,
> I have a test machine that is a nuc6 with an i5 and 32G of ram and SSD
> disks.
> It is configured as a single host environment with Self Hosted Engine VM.
> Both host and SHE are CentOS 7.2 and oVirt version is 3.6.6.2-1.el7
>
> I notice that having 3 VMs powered on and making nothing special (engine
> VM, a CentOS 7 VM and a Fedora 24 VM) the ovirt-ha-agent process on host
> often spikes its cpu usage.
> See for example this quick video with top command running on host that
> reflects what happens continuously.
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvYUVRMFlLVmxRdXM/
> view?usp=sharing
>
> Is it normal that ovirt-ha-agent consumes all this amount of cpu?
> Going into /var/log/ovirt-hosted-engine-ha/agent.log I see nothing
> special, only messages of type "INFO". The same for broker.log
>
> Thanks,
> Gianluca
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] ovirt-ha-agent cpu usage

[Gianluca Cecchi]

Hello,
I have a test machine that is a nuc6 with an i5 and 32G of ram and SSD
disks.
It is configured as a single host environment with Self Hosted Engine VM.
Both host and SHE are CentOS 7.2 and oVirt version is 3.6.6.2-1.el7

I notice that having 3 VMs powered on and making nothing special (engine
VM, a CentOS 7 VM and a Fedora 24 VM) the ovirt-ha-agent process on host
often spikes its cpu usage.
See for example this quick video with top command running on host that
reflects what happens continuously.

https://drive.google.com/file/d/0BwoPbcrMv8mvYUVRMFlLVmxRdXM/view?usp=sharing

Is it normal that ovirt-ha-agent consumes all this amount of cpu?
Going into /var/log/ovirt-hosted-engine-ha/agent.log I see nothing special,
only messages of type "INFO". The same for broker.log

Thanks,
Gianluca
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users