Re: [ovirt-users] vdsm memory consumption (ovirt 4.0)

2016-09-01 Thread Nir Soffer 
On Wed, Aug 31, 2016 at 9:06 PM, Nir Soffer  wrote:

> On Wed, Aug 31, 2016 at 8:06 PM, Federico Alberto Sayd 
> wrote:
>
>> Hello Nir:
>>
>>
>> I followed your instructions , added the config file, restarted vdsm, and
>> today I have the vdsm logs from a host:
>>
>> https://drive.google.com/file/d/0ByrwZ1AkYuyeR1hmRm90a1R6MEk
>> /view?usp=sharing
>>
>> Please tell me if you see anything related to the memory issue.
>>
>
> This logs start when vdsm is using 567640 kB (554 MiB) - very unusual.
>
> The memory usage grow by 18 MiB during one day. No garbage collection
> issues. This smells like we keep some data forever for no reason.
>
> $ grep rss= vdsm-leak.log | head -n 1
> Thread-33::DEBUG::2016-08-30 
> 12:01:43,845::health::122::health::(_check_resources)
> user=1.73%, sys=1.65%, rss=567640 kB (+44), threads=57
>
>  $ grep rss= vdsm-leak.log | tail -n 1
> Thread-33::DEBUG::2016-08-31 
> 13:00:36,913::health::122::health::(_check_resources)
> user=4.18%, sys=1.87%, rss=586584 kB (+0), threads=52
>
> I would like to see the logs since vdsm was started  - do you have them?
>
> Also, can you describe the workload on this hypervisor?
>
> - how many vms are running at the same time
> - how many vms are started and stopped per hour
> - using default vdsm.conf? if not, please attach your conf
>

I could reproduce similar leak in master - it seems that we leak about 1MiB
for each vm started and stopped.

I opened this bug:
https://bugzilla.redhat.com/1372205

Please check if this bug match your issue. If it does, please add
your logs and other info to this bug.

Thanks,
Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm memory consumption (ovirt 4.0)

2016-08-31 Thread Nir Soffer 
On Wed, Aug 31, 2016 at 8:06 PM, Federico Alberto Sayd 
wrote:

> Hello Nir:
>
>
> I followed your instructions , added the config file, restarted vdsm, and
> today I have the vdsm logs from a host:
>
> https://drive.google.com/file/d/0ByrwZ1AkYuyeR1hmRm90a1R6MEk/
> view?usp=sharing
>
> Please tell me if you see anything related to the memory issue.
>

This logs start when vdsm is using 567640 kB (554 MiB) - very unusual.

The memory usage grow by 18 MiB during one day. No garbage collection
issues. This smells like we keep some data forever for no reason.

$ grep rss= vdsm-leak.log | head -n 1
Thread-33::DEBUG::2016-08-30
12:01:43,845::health::122::health::(_check_resources) user=1.73%,
sys=1.65%, rss=567640 kB (+44), threads=57

 $ grep rss= vdsm-leak.log | tail -n 1
Thread-33::DEBUG::2016-08-31
13:00:36,913::health::122::health::(_check_resources) user=4.18%,
sys=1.87%, rss=586584 kB (+0), threads=52

I would like to see the logs since vdsm was started  - do you have them?

Also, can you describe the workload on this hypervisor?

- how many vms are running at the same time
- how many vms are started and stopped per hour
- using default vdsm.conf? if not, please attach your conf

Nir

>
> Thanks
>
> Federico
>
> El 30/08/16 a las 03:47, Nir Soffer escribió:
>
> On Tue, Aug 30, 2016 at 1:30 AM, Federico Alberto Sayd <
> fs...@uncu.edu.ar> wrote:
>
>> I have issues with my ovirt setup related to memory consumption. After
>> upgrading to 4.0 I noted a considerable grow in vdsm memory consumption.
>> I suspect that the grow is related to a memory leak.
>>
>
> We need more details, see bellow...
>
>
>>
>> When I boot up the system and activate the host the memory consumption
>> is about 600MB. After 5 days running and host in maintenance mode the
>> memory consumption is about 1,4 GB.
>>
>> I need to put my hosts in maintenance and reboot to free memory.
>>
>
> You can restart vdsm (systemctl restart vdsmd) instead, running vms
> are not effected by this.
>
>
>>
>> Can anyone help me to debug this problem?
>>
>
> We had a memory in vdsm-4.18.5, fixed  in vdsm-4.18.11. Since you
> are running 4.18.11, there may be another leak.
>
> Please enable health monitoring by creating
> /etc/vdsm/vdsm.conf.d/50-health.conf
>
> [devel]
> health_monitor_enable = true
>
> And restart vdsm.
>
> Please run with this setting for couple of hours, maybe one day,
> and then share the vdsm logs from this timeframe.
>
> You may disable health monitoring by setting
>
> [devel]
> health_monitor_enable = false
>
> Or by renaming or deleting this configuration file:
>
> /etc/vdsm/vdsm.conf.d/50-health.conf.disabled
>
> Nir
>
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm memory consumption (ovirt 4.0)

2016-08-31 Thread Federico Alberto Sayd 

Hello Nir:


I followed your instructions , added the config file, restarted vdsm, 
and today I have the vdsm logs from a host:


https://drive.google.com/file/d/0ByrwZ1AkYuyeR1hmRm90a1R6MEk/view?usp=sharing

Please tell me if you see anything related to the memory issue.


Thanks


Federico

El 30/08/16 a las 03:47, Nir Soffer escribió:
On Tue, Aug 30, 2016 at 1:30 AM, Federico Alberto Sayd 
> wrote:


I have issues with my ovirt setup related to memory consumption. After
upgrading to 4.0 I noted a considerable grow in vdsm memory
consumption.
I suspect that the grow is related to a memory leak.


We need more details, see bellow...


When I boot up the system and activate the host the memory consumption
is about 600MB. After 5 days running and host in maintenance mode the
memory consumption is about 1,4 GB.

I need to put my hosts in maintenance and reboot to free memory.


You can restart vdsm (systemctl restart vdsmd) instead, running vms
are not effected by this.


Can anyone help me to debug this problem?


We had a memory in vdsm-4.18.5, fixed  in vdsm-4.18.11. Since you
are running 4.18.11, there may be another leak.

Please enable health monitoring by creating
/etc/vdsm/vdsm.conf.d/50-health.conf

[devel]
health_monitor_enable = true

And restart vdsm.

Please run with this setting for couple of hours, maybe one day,
and then share the vdsm logs from this timeframe.

You may disable health monitoring by setting

[devel]
health_monitor_enable = false

Or by renaming or deleting this configuration file:

/etc/vdsm/vdsm.conf.d/50-health.conf.disabled

Nir



___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm memory consumption (ovirt 4.0)

2016-08-30 Thread Nir Soffer 
On Tue, Aug 30, 2016 at 1:30 AM, Federico Alberto Sayd 
wrote:

> I have issues with my ovirt setup related to memory consumption. After
> upgrading to 4.0 I noted a considerable grow in vdsm memory consumption.
> I suspect that the grow is related to a memory leak.
>

We need more details, see bellow...


>
> When I boot up the system and activate the host the memory consumption
> is about 600MB. After 5 days running and host in maintenance mode the
> memory consumption is about 1,4 GB.
>
> I need to put my hosts in maintenance and reboot to free memory.
>

You can restart vdsm (systemctl restart vdsmd) instead, running vms
are not effected by this.


>
> Can anyone help me to debug this problem?
>

We had a memory in vdsm-4.18.5, fixed  in vdsm-4.18.11. Since you
are running 4.18.11, there may be another leak.

Please enable health monitoring by creating
/etc/vdsm/vdsm.conf.d/50-health.conf

[devel]
health_monitor_enable = true

And restart vdsm.

Please run with this setting for couple of hours, maybe one day,
and then share the vdsm logs from this timeframe.

You may disable health monitoring by setting

[devel]
health_monitor_enable = false

Or by renaming or deleting this configuration file:

/etc/vdsm/vdsm.conf.d/50-health.conf.disabled

Nir
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

[ovirt-users] vdsm memory consumption (ovirt 4.0)

2016-08-29 Thread Federico Alberto Sayd 
I have issues with my ovirt setup related to memory consumption. After
upgrading to 4.0 I noted a considerable grow in vdsm memory consumption.
I suspect that the grow is related to a memory leak.

When I boot up the system and activate the host the memory consumption
is about 600MB. After 5 days running and host in maintenance mode the
memory consumption is about 1,4 GB.

I need to put my hosts in maintenance and reboot to free memory.

Can anyone help me to debug this problem?

OS Version:
RHEL - 7 - 2.1511.el7.centos.2.10
Kernel Version:
3.10.0 - 327.22.2.el7.x86_64
KVM Version:
2.3.0 - 31.el7.16.1
LIBVIRT Version:
libvirt-1.2.17-13.el7_2.5
VDSM Version:
vdsm-4.18.11-1.el7.centos

Thank you
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users