[ovirt-users] Re: ovirt-ha-agent cpu usage

[Nir Soffer]

>
> And a bright new 3-minutes video here:
> https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/
> view?usp=sharing
>

> It seems that now ovirt-ha-agent or is not present in top cpu process or
> at least has ranges between 5% and 12% and not more
>

5-12% is probably 10 times more than needed for the agent, profiling
should tell us where time is spent.

Since the agent depends on vdsm, we can reuse vdsm cpu profiler, see
lib/vdsm/profiling/cpu.py. It is not ready yet to be used by other
applications,
but making it more general should be easy.

Nir

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/A2JOBFOHETQIMVJHLK6CIYF4ECCJ65NI/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
> 
> 
> 
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:59, Nir Soffer > > wrote:
>> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>> mailto:michal.skriva...@redhat.com>> wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer >> > wrote:
>>> 
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer >> > wrote:
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi >> > wrote:
>>> 
>>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor >> > wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>> 
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>>> json rpc and this is CPU intensive since the client has to parse the yaml 
>>> API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
>> clients (vdsm migration code:-) 
>> 
>> This is orthogonal issue.
> 
> Yes it is. And that’s the issue;-)
> Both are wrong, but by “fixing” the schema validation only you lose the 
> motivation to fix the meaningless wasteful reconnect
> 
> Yes, we are going to fix that too ( 
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 
>  )

that’s great! Also al the other vdsClient uses?:-)
What is that periodic one call anyway? Is there only one? Maybe we don’t need 
it so much.

> but it would require also https://bugzilla.redhat.com/show_bug.cgi?id=1376843 
>  to be fixed.

This is less good. Well, worst case you can reconnect yourself, all you need is 
a notification when the existing connection breaks

>  
> 
>>  
>> Does schema validation matter then if there would be only one connection at 
>> the start up?
>> 
>> Loading once does not help command line tools like vdsClient, hosted-engine 
>> and
>> vdsm-tool. 
> 
> none of the other tools is using json-rpc.
> 
> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
> remaining tools since xmlrpc has been deprecated with 4.0

ok. though setup is a one-time action so it’s not an issue there

>  
> 
>> 
>> Nir
>>  
>> 
>>> 
>>> Simone, reusing the connection is good idea anyway, but what you describe 
>>> is 
>>> a bug in the client library. The library does *not* need to load and parse 
>>> the
>>> schema at all for sending requests to vdsm.
>>> 
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>> 
>>> Please file an infra bug about it.
>>> 
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>>> 
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230 
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org 
>>> http://lists.ovirt.org/mailman/listinfo/users 
>>> 
>> 
>> 
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org 
> http://lists.ovirt.org/mailman/listinfo/users 
> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users


--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to 

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Sam Cappello]

it worked for me as well - load avg. < 1 now, ovirt-ha-agent pops up in 
top periodically, but not on top using 100% CPU all the time anymore.

thanks!
sam

Gianluca Cecchi wrote on 10/7/2016 10:13 AM:



On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi > wrote:




On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi
mailto:gianluca.cec...@gmail.com>> wrote:

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer mailto:nsof...@redhat.com>> wrote:


wasn’t it suppose to be fixed to reuse the connection?
Like all the other clients (vdsm migration code:-)


This is orthogonal issue.

Does schema validation matter then if there would be
only one connection at the start up?


Loading once does not help command line tools like
vdsClient, hosted-engine and
vdsm-tool.

Nir




Simone, reusing the connection is good idea
anyway, but what you describe is
a bug in the client library. The library does
*not* need to load and parse the
schema at all for sending requests to vdsm.

The schema is only needed if you want to
verify request parameters,
or provide online help, these are not needed
in a client library.

Please file an infra bug about it.


Done,
https://bugzilla.redhat.com/show_bug.cgi?id=1381899



Here is a patch that should eliminate most most of
the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system
showing this problem.

Cheers,
Nir
___




this is a video of 1 minute with the same system as the first
post, but in 4.0.3 now and the same 3 VMs powered on without
any particular load.
It seems very similar to the previous 3.6.6 in cpu used by
ovirt-ha-agent.


https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing



Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if
there is then a different behavior.
Let me know,


I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you
could directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial
spaces.

Then you can simply restart the HA agent and check.

Gianluca

___
Users mailing list
Users@ovirt.org 
http://lists.ovirt.org/mailman/listinfo/users





What I've done (I didn't read your answer in between and this is a 
test system not so important... )


set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process 
or at least has ranges between 5% and 12% and not more


BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; 
vendor preset: enabled)

   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh 
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh 
--pre-start (code=exited, status=0/SUCCESS)

 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 
--write-pipe-fd 40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 
--write-pipe-fd 46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 
--write-pipe-fd 56 --max-threads 10 --max-queue...
   ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>
>
>
> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek <
> michal.skriva...@redhat.com> wrote:
>
>>
>> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
>>
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
>> michal.skriva...@redhat.com> wrote:
>>
>>>
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>>
>>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi <
> stira...@redhat.com> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor 
>> wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>> Yes it is. And that’s the issue;-)
>> Both are wrong, but by “fixing” the schema validation only you lose the
>> motivation to fix the meaningless wasteful reconnect
>>
>
> Yes, we are going to fix that too ( https://bugzilla.redhat.com/
> show_bug.cgi?id=1349829 )
>
>
> that’s great! Also al the other vdsClient uses?:-)
>

https://gerrit.ovirt.org/#/c/62729/


> What is that periodic one call anyway? Is there only one? Maybe we don’t
> need it so much.
>

Currently ovirt-ha-agent is periodically reconnecting the hosted-engine
storage domain and checking its status. This is already on jsonrpc.
In 4.1 all the monitoring will be moved to jsonrpc.


>
> but it would require also https://bugzilla.redhat.com/
> show_bug.cgi?id=1376843 to be fixed.
>
>
> This is less good. Well, worst case you can reconnect yourself, all you
> need is a notification when the existing connection breaks
>
>
>
>>
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>>
>> none of the other tools is using json-rpc.
>>
>
> hosted-engine-setup is, and sooner or later we'll have to migrate also the
> remaining tools since xmlrpc has been deprecated with 4.0
>
>
> ok. though setup is a one-time action so it’s not an issue there
>
>
>
>>
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>>
>>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Nir Soffer]

On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>>
>>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>> wrote:
>>>


 On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

 Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
 over json rpc and this is CPU intensive since the client has to parse the
 yaml API specification each time it connects.

>>>
> wasn’t it suppose to be fixed to reuse the connection? Like all the other
> clients (vdsm migration code:-)
>

This is orthogonal issue.


> Does schema validation matter then if there would be only one connection
> at the start up?
>

Loading once does not help command line tools like vdsClient, hosted-engine
and
vdsm-tool.

Nir


>
>
>>> Simone, reusing the connection is good idea anyway, but what you
>>> describe is
>>> a bug in the client library. The library does *not* need to load and
>>> parse the
>>> schema at all for sending requests to vdsm.
>>>
>>> The schema is only needed if you want to verify request parameters,
>>> or provide online help, these are not needed in a client library.
>>>
>>> Please file an infra bug about it.
>>>
>>
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>
>
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230
>
> Would be nice if it can be tested on the system showing this problem.
>
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/SWDGG5TVQ54Q3SB2TAOKEQI6SVNTQS5W/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek  > wrote:
> 
>> On 7 Oct 2016, at 14:42, Nir Soffer > > wrote:
>> 
>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer > > wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi > > wrote:
>> 
>> 
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor > > wrote:
>> Hi,
>> 
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
>> json rpc and this is CPU intensive since the client has to parse the yaml 
>> API specification each time it connects.
> 
> wasn’t it suppose to be fixed to reuse the connection? Like all the other 
> clients (vdsm migration code:-) 
> 
> This is orthogonal issue.

Yes it is. And that’s the issue;-)
Both are wrong, but by “fixing” the schema validation only you lose the 
motivation to fix the meaningless wasteful reconnect

>  
> Does schema validation matter then if there would be only one connection at 
> the start up?
> 
> Loading once does not help command line tools like vdsClient, hosted-engine 
> and
> vdsm-tool. 

none of the other tools is using json-rpc.

> 
> Nir
>  
> 
>> 
>> Simone, reusing the connection is good idea anyway, but what you describe is 
>> a bug in the client library. The library does *not* need to load and parse 
>> the
>> schema at all for sending requests to vdsm.
>> 
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>> 
>> Please file an infra bug about it.
>> 
>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
>> 
>> 
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230 
>> 
>> Would be nice if it can be tested on the system showing this problem.
>> 
>> Cheers,
>> Nir
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users 
>> 
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users


--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OFZU3EZTXNPNGGIJ2RJWWMFJBXRI3VHJ/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:

>
>> wasn’t it suppose to be fixed to reuse the connection? Like all the other
>> clients (vdsm migration code:-)
>>
>
> This is orthogonal issue.
>
>
>> Does schema validation matter then if there would be only one connection
>> at the start up?
>>
>
> Loading once does not help command line tools like vdsClient,
> hosted-engine and
> vdsm-tool.
>
> Nir
>
>
>>
>>
 Simone, reusing the connection is good idea anyway, but what you
 describe is
 a bug in the client library. The library does *not* need to load and
 parse the
 schema at all for sending requests to vdsm.

 The schema is only needed if you want to verify request parameters,
 or provide online help, these are not needed in a client library.

 Please file an infra bug about it.

>>>
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>>
>>
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230
>>
>> Would be nice if it can be tested on the system showing this problem.
>>
>> Cheers,
>> Nir
>> ___
>>
>>

this is a video of 1 minute with the same system as the first post, but in
4.0.3 now and the same 3 VMs powered on without any particular load.
It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.

https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing

Enjoy Nir ;-)

If I can apply the patch also to 4.0.3 I'm going to see if there is then a
different behavior.
Let me know,

Gianluca

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EBOEALLKXJJJ7CFGO3ZARA6DZSV3F3YJ/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi 
wrote:

> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>
>>
>>> wasn’t it suppose to be fixed to reuse the connection? Like all the
>>> other clients (vdsm migration code:-)
>>>
>>
>> This is orthogonal issue.
>>
>>
>>> Does schema validation matter then if there would be only one connection
>>> at the start up?
>>>
>>
>> Loading once does not help command line tools like vdsClient,
>> hosted-engine and
>> vdsm-tool.
>>
>> Nir
>>
>>
>>>
>>>
> Simone, reusing the connection is good idea anyway, but what you
> describe is
> a bug in the client library. The library does *not* need to load and
> parse the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899

>>>
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>>
>>> Would be nice if it can be tested on the system showing this problem.
>>>
>>> Cheers,
>>> Nir
>>> ___
>>>
>>>
>
> this is a video of 1 minute with the same system as the first post, but in
> 4.0.3 now and the same 3 VMs powered on without any particular load.
> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>
> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/
> view?usp=sharing
>
> Enjoy Nir ;-)
>
> If I can apply the patch also to 4.0.3 I'm going to see if there is then a
> different behavior.
> Let me know,
>
>
I'm trying it right now.
Any other tests will be really appreciated.

The patch is pretty simply, you can apply that on the fly.
You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
directly edit
/usr/lib/python2.7/site-packages/api/vdsmapi.py
around line 97 changing from
loaded_schema = yaml.load(f)
to
loaded_schema = yaml.load(f, Loader=yaml.CLoader)
Please pay attention to keep exactly the same amount of initial spaces.

Then you can simply restart the HA agent and check.

Gianluca
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PNLP5ARR2OU6XN74AIA6O26ZERCF3DTR/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 07 Oct 2016, at 16:10, Simone Tiraboschi  wrote:
> 
> 
> 
>> On Fri, Oct 7, 2016 at 4:02 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 15:28, Simone Tiraboschi  wrote:
>>> 
>>> 
>>> 
>>> On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek 
>>>  wrote:
 
> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
> 
>> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek 
>>  wrote:
>> 
>>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>> 
 On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
  wrote:
 
 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  
> wrote:
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>>  wrote:
>> 
>> 
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  
>>> wrote:
>>> Hi,
>>> 
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there 
>>> is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>> 
>> Yes, it's due to the fact that ovirt-ha-agent periodically 
>> reconnects over json rpc and this is CPU intensive since the client 
>> has to parse the yaml API specification each time it connects.
>> 
>> wasn’t it suppose to be fixed to reuse the connection? Like all the 
>> other clients (vdsm migration code:-) 
> 
> This is orthogonal issue.
 
 Yes it is. And that’s the issue;-)
 Both are wrong, but by “fixing” the schema validation only you lose the 
 motivation to fix the meaningless wasteful reconnect
>>> 
>>> Yes, we are going to fix that too ( 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 )
>> 
>> that’s great! Also al the other vdsClient uses?:-)
> 
> https://gerrit.ovirt.org/#/c/62729/

Cool. It's not so important for one time actions but we need it to be able to 
drop xmlrpc finally, so it is important:)

>  
>> What is that periodic one call anyway? Is there only one? Maybe we don’t 
>> need it so much.
> 
> Currently ovirt-ha-agent is periodically reconnecting the hosted-engine 
> storage domain and checking its status. This is already on jsonrpc.

Ok. As long as it is low frequency it's ok. Just bear in mind for future that 
some of the vdsm calls may be heavy and impact performance, regardless the 
communication layer. 

Thanks and have a nice weekend,
michal

> In 4.1 all the monitoring will be moved to jsonrpc.
>  
>> 
>>> but it would require also 
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 to be fixed.
>> 
>> This is less good. Well, worst case you can reconnect yourself, all you need 
>> is a notification when the existing connection breaks
>> 
>>>  
 
>  
>> Does schema validation matter then if there would be only one connection 
>> at the start up?
> 
> Loading once does not help command line tools like vdsClient, 
> hosted-engine and
> vdsm-tool. 
 
 none of the other tools is using json-rpc.
>>> 
>>> hosted-engine-setup is, and sooner or later we'll have to migrate also the 
>>> remaining tools since xmlrpc has been deprecated with 4.0
>> 
>> ok. though setup is a one-time action so it’s not an issue there
>> 
>>>  
 
> 
> Nir
>  
>> 
> 
> Simone, reusing the connection is good idea anyway, but what you 
> describe is 
> a bug in the client library. The library does *not* need to load and 
> parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
 
 Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>> 
>>> Here is a patch that should eliminate most most of the problem:
>>> https://gerrit.ovirt.org/65230
>>> 
>>> Would be nice if it can be tested on the system showing this problem.
>>> 
>>> Cheers,
>>> Nir
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
 
 
 ___
 Users mailing list
 Users@ovirt.org
 http://lists.ovirt.org/mailman/listinfo/users
>>> 
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
> 

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Gianluca Cecchi]

On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi 
wrote:

>
>
> On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi  > wrote:
>
>> On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer  wrote:
>>
>>>
 wasn’t it suppose to be fixed to reuse the connection? Like all the
 other clients (vdsm migration code:-)

>>>
>>> This is orthogonal issue.
>>>
>>>
 Does schema validation matter then if there would be only one
 connection at the start up?

>>>
>>> Loading once does not help command line tools like vdsClient,
>>> hosted-engine and
>>> vdsm-tool.
>>>
>>> Nir
>>>
>>>


>> Simone, reusing the connection is good idea anyway, but what you
>> describe is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

 Here is a patch that should eliminate most most of the problem:
 https://gerrit.ovirt.org/65230

 Would be nice if it can be tested on the system showing this problem.

 Cheers,
 Nir
 ___


>>
>> this is a video of 1 minute with the same system as the first post, but
>> in 4.0.3 now and the same 3 VMs powered on without any particular load.
>> It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
>>
>> https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8
>> /view?usp=sharing
>>
>> Enjoy Nir ;-)
>>
>> If I can apply the patch also to 4.0.3 I'm going to see if there is then
>> a different behavior.
>> Let me know,
>>
>>
> I'm trying it right now.
> Any other tests will be really appreciated.
>
> The patch is pretty simply, you can apply that on the fly.
> You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could
> directly edit
> /usr/lib/python2.7/site-packages/api/vdsmapi.py
> around line 97 changing from
> loaded_schema = yaml.load(f)
> to
> loaded_schema = yaml.load(f, Loader=yaml.CLoader)
> Please pay attention to keep exactly the same amount of initial spaces.
>
> Then you can simply restart the HA agent and check.
>
> Gianluca
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>

What I've done (I didn't read your answer in between and this is a test
system not so important... )

set to global maintenance
patch vdsmapi.py
restart vdsmd
restart ovirt-ha-agent
set maintenance to none

And a bright new 3-minutes video here:
https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing

It seems that now ovirt-ha-agent or is not present in top cpu process or at
least has ranges between 5% and 12% and not more

BTW: I see this in vdsm status now

[root@ovirt01 api]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor
preset: enabled)
   Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago
  Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh
--post-stop (code=exited, status=0/SUCCESS)
  Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh
--pre-start (code=exited, status=0/SUCCESS)
 Main PID: 21023 (vdsm)
   CGroup: /system.slice/vdsmd.service
   ├─21023 /usr/bin/python /usr/share/vdsm/vdsm
   ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 --write-pipe-fd
40 --max-threads 10 --max-queue...
   ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd
46 --max-threads 10 --max-queue...
   ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd
56 --max-threads 10 --max-queue...
   ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd
64 --max-threads 10 --max-queue...
   ├─21149 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd
72 --max-threads 10 --max-queue...
   ├─21156 /usr/libexec/ioprocess --read-pipe-fd 80 --write-pipe-fd
78 --max-threads 10 --max-queue...
   ├─21177 /usr/libexec/ioprocess --read-pipe-fd 88 --write-pipe-fd
87 --max-threads 10 --max-queue...
   ├─21204 /usr/libexec/ioprocess --read-pipe-fd 99 --write-pipe-fd
98 --max-threads 10 --max-queue...
   └─21239 /usr/libexec/ioprocess --read-pipe-fd 111
--write-pipe-fd 110 --max-threads 10 --max-que...

Oct 07 16:02:52 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:54 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:56 ovirt01.lutwyn.org vdsm[21023]: vdsm vds.dispatcher ERROR
SSL error during reading data... eof
Oct 07 16:02:58 

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Fri, Oct 7, 2016 at 3:25 PM, Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On 7 Oct 2016, at 14:59, Nir Soffer  wrote:
>
> On Fri, Oct 7, 2016 at 3:52 PM, Michal Skrivanek <
> michal.skriva...@redhat.com> wrote:
>
>>
>> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
>>
>> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>>>
 On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi >>> > wrote:

>
>
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>
>> Hi,
>>
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>
>
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
> over json rpc and this is CPU intensive since the client has to parse the
> yaml API specification each time it connects.
>

>> wasn’t it suppose to be fixed to reuse the connection? Like all the other
>> clients (vdsm migration code:-)
>>
>
> This is orthogonal issue.
>
>
> Yes it is. And that’s the issue;-)
> Both are wrong, but by “fixing” the schema validation only you lose the
> motivation to fix the meaningless wasteful reconnect
>

Yes, we are going to fix that too (
https://bugzilla.redhat.com/show_bug.cgi?id=1349829 ) but it would require
also https://bugzilla.redhat.com/show_bug.cgi?id=1376843 to be fixed.


>
>
>
>> Does schema validation matter then if there would be only one connection
>> at the start up?
>>
>
> Loading once does not help command line tools like vdsClient,
> hosted-engine and
> vdsm-tool.
>
>
> none of the other tools is using json-rpc.
>

hosted-engine-setup is, and sooner or later we'll have to migrate also the
remaining tools since xmlrpc has been deprecated with 4.0


>
>
> Nir
>
>
>>
>>
 Simone, reusing the connection is good idea anyway, but what you
 describe is
 a bug in the client library. The library does *not* need to load and
 parse the
 schema at all for sending requests to vdsm.

 The schema is only needed if you want to verify request parameters,
 or provide online help, these are not needed in a client library.

 Please file an infra bug about it.

>>>
>>> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>>>
>>
>> Here is a patch that should eliminate most most of the problem:
>> https://gerrit.ovirt.org/65230
>>
>> Would be nice if it can be tested on the system showing this problem.
>>
>> Cheers,
>> Nir
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/C45E4PS6WM6UPOJHATKPVSAS57EAGH35/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Michal Skrivanek]

> On 7 Oct 2016, at 14:42, Nir Soffer  wrote:
> 
> On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  > wrote:
> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi  > wrote:
> 
> 
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  > wrote:
> Hi,
> 
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
> 
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over 
> json rpc and this is CPU intensive since the client has to parse the yaml API 
> specification each time it connects.

wasn’t it suppose to be fixed to reuse the connection? Like all the other 
clients (vdsm migration code:-) 
Does schema validation matter then if there would be only one connection at the 
start up?

> 
> Simone, reusing the connection is good idea anyway, but what you describe is 
> a bug in the client library. The library does *not* need to load and parse the
> schema at all for sending requests to vdsm.
> 
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
> 
> Please file an infra bug about it.
> 
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 
> 
> 
> Here is a patch that should eliminate most most of the problem:
> https://gerrit.ovirt.org/65230 
> 
> Would be nice if it can be tested on the system showing this problem.
> 
> Cheers,
> Nir
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users


--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/33ETNQXRLHTTICZSOHJOA76QGIUCRBOJ/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 1:33 PM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:
>
>> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>>
 Hi,

 did you found a solution or cause for this high CPU usage?
 I have installed the self hosted engine on another server and there is
 no VM running but ovirt-ha-agent uses heavily the CPU.

>>>
>>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>>> over json rpc and this is CPU intensive since the client has to parse the
>>> yaml API specification each time it connects.
>>>
>>
>> Simone, reusing the connection is good idea anyway, but what you describe
>> is
>> a bug in the client library. The library does *not* need to load and
>> parse the
>> schema at all for sending requests to vdsm.
>>
>> The schema is only needed if you want to verify request parameters,
>> or provide online help, these are not needed in a client library.
>>
>> Please file an infra bug about it.
>>
>
> Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
>

Here is a patch that should eliminate most most of the problem:
https://gerrit.ovirt.org/65230

Would be nice if it can be tested on the system showing this problem.

Cheers,
Nir

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZBONUMRRA2POLODZPFSRHZVG3YOTGHSV/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 10:34 AM, Nir Soffer  wrote:

> On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
> wrote:
>
>>
>>
>> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>>
>>> Hi,
>>>
>>> did you found a solution or cause for this high CPU usage?
>>> I have installed the self hosted engine on another server and there is
>>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>>
>>
>> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects
>> over json rpc and this is CPU intensive since the client has to parse the
>> yaml API specification each time it connects.
>>
>
> Simone, reusing the connection is good idea anyway, but what you describe
> is
> a bug in the client library. The library does *not* need to load and parse
> the
> schema at all for sending requests to vdsm.
>
> The schema is only needed if you want to verify request parameters,
> or provide online help, these are not needed in a client library.
>
> Please file an infra bug about it.
>

Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899
Thanks.


> Nir
>
>
>> The issue is tracked here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
>> should reuse json-rpc connections
>> but it depends on:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
>> keep-alive with reconnect if needed logic for the python jsonrpc client
>>
>>
>>
>>>
>>> cheers
>>> gregor
>>>
>>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan >> > > wrote:
>>> >
>>> > Does the spikes correlates with info messages on extracting the
>>> ovf?
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > yes, it seems so and it happens every 14-15 seconds
>>> >
>>> > These are the lines I see scrolling in agent.log when I notice cpu
>>> > spikes in ovirt-ha-agent...
>>> >
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Connecting storage server
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>>> b.storage_server.StorageServer::(connect_storage_server)
>>> > Refreshing the storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.I
>>> mage::(prepare_images)
>>> > Preparing images
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>>> > Reloading vm.conf from the shared storage domain
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(scan)
>>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > Extracting Engine VM OVF from the OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>>> .ovf_store.OVFStore::(getEngineVMOVF)
>>> > OVF_STORE volume path:
>>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab2
>>> 2-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Found an OVF for HE VM, trying to convert
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>>> > Got vm.conf from OVF_STORE
>>> > MainThread::INFO::2016-08-08
>>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>>> nt.hosted_engine.HostedEngine::(start_monitoring)
>>> > Current state EngineUp (score: 3400)
>>> >
>>> >
>>> > ___
>>> > Users mailing list
>>> > Users@ovirt.org
>>> > http://lists.ovirt.org/mailman/listinfo/users
>>> >
>>> ___
>>> Users mailing list
>>> Users@ovirt.org
>>> 

[ovirt-users] Re: ovirt-ha-agent cpu usage

[gregor]

Hi,

did you found a solution or cause for this high CPU usage?
I have installed the self hosted engine on another server and there is
no VM running but ovirt-ha-agent uses heavily the CPU.

cheers
gregor

On 08/08/16 15:09, Gianluca Cecchi wrote:
> On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > wrote:
> 
> Does the spikes correlates with info messages on extracting the ovf?
> 
> 
> 
> 
> 
> 
> yes, it seems so and it happens every 14-15 seconds
> 
> These are the lines I see scrolling in agent.log when I notice cpu
> spikes in ovirt-ha-agent...
> 
> MainThread::INFO::2016-08-08
> 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Connecting storage server
> MainThread::INFO::2016-08-08
> 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
> Refreshing the storage domain
> MainThread::INFO::2016-08-08
> 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
> Preparing images
> MainThread::INFO::2016-08-08
> 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> Reloading vm.conf from the shared storage domain
> MainThread::INFO::2016-08-08
> 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Trying to get a fresher copy of vm configuration from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> MainThread::INFO::2016-08-08
> 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
> Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> volUUID:1a18851e-6858-401c-be6e-af14415034b5
> MainThread::INFO::2016-08-08
> 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> Extracting Engine VM OVF from the OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF)
> OVF_STORE volume path:
> /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>  
> MainThread::INFO::2016-08-08
> 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Found an OVF for HE VM, trying to convert
> MainThread::INFO::2016-08-08
> 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> Got vm.conf from OVF_STORE
> MainThread::INFO::2016-08-08
> 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
> Current state EngineUp (score: 3400)
> 
> 
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QJCHWW5M6M4WTEXW7GXYGOK2U3YNJNEO/

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Nir Soffer]

On Wed, Oct 5, 2016 at 10:24 AM, Simone Tiraboschi 
wrote:

>
>
> On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:
>
>> Hi,
>>
>> did you found a solution or cause for this high CPU usage?
>> I have installed the self hosted engine on another server and there is
>> no VM running but ovirt-ha-agent uses heavily the CPU.
>>
>
> Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
> json rpc and this is CPU intensive since the client has to parse the yaml
> API specification each time it connects.
>

Simone, reusing the connection is good idea anyway, but what you describe
is
a bug in the client library. The library does *not* need to load and parse
the
schema at all for sending requests to vdsm.

The schema is only needed if you want to verify request parameters,
or provide online help, these are not needed in a client library.

Please file an infra bug about it.

Nir


> The issue is tracked here:
> https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent
> should reuse json-rpc connections
> but it depends on:
> https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
> keep-alive with reconnect if needed logic for the python jsonrpc client
>
>
>
>>
>> cheers
>> gregor
>>
>> On 08/08/16 15:09, Gianluca Cecchi wrote:
>> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan > > > wrote:
>> >
>> > Does the spikes correlates with info messages on extracting the ovf?
>> >
>> >
>> >
>> >
>> >
>> >
>> > yes, it seems so and it happens every 14-15 seconds
>> >
>> > These are the lines I see scrolling in agent.log when I notice cpu
>> > spikes in ovirt-ha-agent...
>> >
>> > MainThread::INFO::2016-08-08
>> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Connecting storage server
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.li
>> b.storage_server.StorageServer::(connect_storage_server)
>> > Refreshing the storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.image.
>> Image::(prepare_images)
>> > Preparing images
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(_initialize_storage_images)
>> > Reloading vm.conf from the shared storage domain
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Trying to get a fresher copy of vm configuration from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
>> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(scan)
>> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
>> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > Extracting Engine VM OVF from the OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.ovf
>> .ovf_store.OVFStore::(getEngineVMOVF)
>> > OVF_STORE volume path:
>> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/31a9e9
>> fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-01f7-41ab-
>> ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Found an OVF for HE VM, trying to convert
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.host
>> ed_engine.HostedEngine.config::(refresh_local_conf_file)
>> > Got vm.conf from OVF_STORE
>> > MainThread::INFO::2016-08-08
>> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.age
>> nt.hosted_engine.HostedEngine::(start_monitoring)
>> > Current state EngineUp (score: 3400)
>> >
>> >
>> > ___
>> > Users mailing list
>> > Users@ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.

[ovirt-users] Re: ovirt-ha-agent cpu usage

[Simone Tiraboschi]

On Wed, Oct 5, 2016 at 9:17 AM, gregor  wrote:

> Hi,
>
> did you found a solution or cause for this high CPU usage?
> I have installed the self hosted engine on another server and there is
> no VM running but ovirt-ha-agent uses heavily the CPU.
>

Yes, it's due to the fact that ovirt-ha-agent periodically reconnects over
json rpc and this is CPU intensive since the client has to parse the yaml
API specification each time it connects.
The issue is tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=1349829 - ovirt-ha-agent should
reuse json-rpc connections
but it depends on:
https://bugzilla.redhat.com/show_bug.cgi?id=1376843 - [RFE] Implement a
keep-alive with reconnect if needed logic for the python jsonrpc client


>
> cheers
> gregor
>
> On 08/08/16 15:09, Gianluca Cecchi wrote:
> > On Mon, Aug 8, 2016 at 1:03 PM, Roy Golan  > > wrote:
> >
> > Does the spikes correlates with info messages on extracting the ovf?
> >
> >
> >
> >
> >
> >
> > yes, it seems so and it happens every 14-15 seconds
> >
> > These are the lines I see scrolling in agent.log when I notice cpu
> > spikes in ovirt-ha-agent...
> >
> > MainThread::INFO::2016-08-08
> > 15:03:07,815::storage_server::212::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Connecting storage server
> > MainThread::INFO::2016-08-08
> > 15:03:08,144::storage_server::220::ovirt_hosted_engine_ha.
> lib.storage_server.StorageServer::(connect_storage_server)
> > Refreshing the storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::hosted_engine::685::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:08,705::image::126::ovirt_hosted_engine_ha.lib.
> image.Image::(prepare_images)
> > Preparing images
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::hosted_engine::688::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(_initialize_storage_images)
> > Reloading vm.conf from the shared storage domain
> > MainThread::INFO::2016-08-08
> > 15:03:09,653::config::205::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Trying to get a fresher copy of vm configuration from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:09,843::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:223d26c2-1668-493c-a322-8054923d135f,
> > volUUID:108a362c-f5a9-440e-8817-1ed8a129afe8
> > MainThread::INFO::2016-08-08
> > 15:03:10,309::ovf_store::100::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(scan)
> > Found OVF_STORE: imgUUID:12ca2fc6-01f7-41ab-ab22-e75c822ac9b6,
> > volUUID:1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:10,652::ovf_store::109::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > Extracting Engine VM OVF from the OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:10,974::ovf_store::116::ovirt_hosted_engine_ha.lib.
> ovf.ovf_store.OVFStore::(getEngineVMOVF)
> > OVF_STORE volume path:
> > /rhev/data-center/mnt/ovirt01.lutwyn.org:_SHE__DOMAIN/
> 31a9e9fd-8dcb-4475-aac4-09f897ee1b45/images/12ca2fc6-
> 01f7-41ab-ab22-e75c822ac9b6/1a18851e-6858-401c-be6e-af14415034b5
> > MainThread::INFO::2016-08-08
> > 15:03:11,494::config::225::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Found an OVF for HE VM, trying to convert
> > MainThread::INFO::2016-08-08
> > 15:03:11,497::config::230::ovirt_hosted_engine_ha.agent.
> hosted_engine.HostedEngine.config::(refresh_local_conf_file)
> > Got vm.conf from OVF_STORE
> > MainThread::INFO::2016-08-08
> > 15:03:11,675::hosted_engine::462::ovirt_hosted_engine_ha.
> agent.hosted_engine.HostedEngine::(start_monitoring)
> > Current state EngineUp (score: 3400)
> >
> >
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>

--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se



--
IMPORTANT!
This message has been scanned for viruses and phishing links.
However, it is your responsibility to evaluate the links and attachments you 
choose to click.
If you are uncertain, we always try to help.
Greetings helpd...@actnet.se


___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: