Re: [ovirt-users] vdsm high mem usage

[Michal Skrivanek]

On Sep 10, 2015, at 20:35 , Michael Kleinpaste 
 wrote:

> Hi everybody.
> 
> So I ran into that high mem usage thing. The problem I have with patching is 
> that this is a live system so I can't do it mid day.  Can anybody tell me if 
> it is possible to just restart the vdsm service or does the host have to be 
> in "maintenance mode" before restarting it?  It is using gluster storage, if 
> that makes a difference as well.

Hi,
you can restart vdsm without any effect on running VMs. Other than short 
interruption of communication between engine and host. It can cause a short CPU 
spike on startup, so do that with caution when you run tend or hundreds of VMs 
on a same overloaded host.
Obviously, while vdsm is not running, the system is a bit more vulnerable to 
failures, but as long as you don't do that in the middle of a migration or a 
power failure you're good:)

Thanks,
michal
> 
> Thanks,
> -- 
> Michael Kleinpaste
> Senior Systems Administrator
> SharperLending, LLC.
> www.SharperLending.com
> michael.kleinpa...@sharperlending.com
> (509) 324-1230   Fax: (509) 324-1234
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm high mem usage

[Michael Kleinpaste]

Thanks all.  I restarted it and that fixed the issue temporarily freeing up
memory but it continued the leak process.  I updated the vdsm package and
that fixed it overall.

On Tue, Sep 15, 2015 at 12:57 AM Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
> On Sep 10, 2015, at 20:35 , Michael Kleinpaste <
> michael.kleinpa...@sharperlending.com> wrote:
>
> > Hi everybody.
> >
> > So I ran into that high mem usage thing. The problem I have with
> patching is that this is a live system so I can't do it mid day.  Can
> anybody tell me if it is possible to just restart the vdsm service or does
> the host have to be in "maintenance mode" before restarting it?  It is
> using gluster storage, if that makes a difference as well.
>
> Hi,
> you can restart vdsm without any effect on running VMs. Other than short
> interruption of communication between engine and host. It can cause a short
> CPU spike on startup, so do that with caution when you run tend or hundreds
> of VMs on a same overloaded host.
> Obviously, while vdsm is not running, the system is a bit more vulnerable
> to failures, but as long as you don't do that in the middle of a migration
> or a power failure you're good:)
>
> Thanks,
> michal
> >
> > Thanks,
> > --
> > Michael Kleinpaste
> > Senior Systems Administrator
> > SharperLending, LLC.
> > www.SharperLending.com
> > michael.kleinpa...@sharperlending.com
> > (509) 324-1230   Fax: (509) 324-1234
> > ___
> > Users mailing list
> > Users@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
>
> --
*Michael Kleinpaste*
Senior Systems Administrator
SharperLending, LLC.
www.SharperLending.com
michael.kleinpa...@sharperlending.com
(509) 324-1230   Fax: (509) 324-1234
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm high mem usage

[Daniel Helgenberger]

Hello Michael,

I ran into the issue myself and can confirm restarting vdsm with nfs 
mitgates the issue. I even had a cron job for that

On 11.09.2015 04:30, Darrell Budic wrote:
> If you’re using nfs mounts (even if they are gluster based), it’s safe to
> restart vdsmd, you’ll see it change status in ovirt, but your VMs will 
> continue
> running. If you’re mounting gluster based storage as glusterfs shares directly
> (not over nfs), there’s another issue that will cause all your VMs to pause 
> and
> the only way to recover is to stop them and restart them, but that’s going to
> happen to them anyway when vdsmd runs out of ram and crashes… Best solution is
> to migrate them yourself in this case, then restart and migrate back.
This is what I have done. The easiest way to do so is to set the host in 
maintenance, wait for the migration finishes and then restart vdsm. You 
sould do this only at one host and then wait a while so you do not run 
into OOM on the whole cluster at once.

  Or live
> migrate them to NFS mounted storage so when vdsm crashes they don’t lock up, 
> and
> clean up after you’ve had an opportunity to upgrade or patch.
>
> Upgrade to 3.5.3 or later at your earliest opportunity, the mem leak is 
> resolved
> there. Sounds like you already found the patch you can apply if upgrading 
> isn’t
> an option, but it will still require you to restart your vdsms.

I can confirm 3.5.3 finally solved the issue for us and VDSM keeps below 
100MB RSS.


>
> -Darrell
>
>> On Sep 10, 2015, at 1:45 PM, Michael Kleinpaste
>> > > wrote:
>>
>> Hi everybody.
>>
>> So I ran into that high mem usage thing. The problem I have with patching is
>> that this is a live system so I can't do it mid day.  Can anybody tell me if
>> it is possible to just restart the vdsm service or does the host have to be 
>> in
>> "maintenance mode" before restarting it?  It is using gluster storage, if 
>> that
>> makes a difference as well.
>>
>> Thanks,
>>
>> --
>> *Michael Kleinpaste*
>> Senior Systems Administrator
>> SharperLending, LLC.
>> www.SharperLending.com
>> michael.kleinpa...@sharperlending.com
>> 
>> (509) 324-1230   Fax: (509) 324-1234
>> ___
>> Users mailing list
>> Users@ovirt.org 
>> http://lists.ovirt.org/mailman/listinfo/users
>

-- 
Daniel Helgenberger
m box bewegtbild GmbH

P: +49/30/2408781-22
F: +49/30/2408781-10

ACKERSTR. 19
D-10115 BERLIN


www.m-box.de  www.monkeymen.tv

Geschäftsführer: Martin Retschitzegger / Michaela Göllner
Handeslregister: Amtsgericht Charlottenburg / HRB 112767
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

Re: [ovirt-users] vdsm high mem usage

[Darrell Budic]

If you’re using nfs mounts (even if they are gluster based), it’s safe to 
restart vdsmd, you’ll see it change status in ovirt, but your VMs will continue 
running. If you’re mounting gluster based storage as glusterfs shares directly 
(not over nfs), there’s another issue that will cause all your VMs to pause and 
the only way to recover is to stop them and restart them, but that’s going to 
happen to them anyway when vdsmd runs out of ram and crashes… Best solution is 
to migrate them yourself in this case, then restart and migrate back. Or live 
migrate them to NFS mounted storage so when vdsm crashes they don’t lock up, 
and clean up after you’ve had an opportunity to upgrade or patch.

Upgrade to 3.5.3 or later at your earliest opportunity, the mem leak is 
resolved there. Sounds like you already found the patch you can apply if 
upgrading isn’t an option, but it will still require you to restart your vdsms.

  -Darrell

> On Sep 10, 2015, at 1:45 PM, Michael Kleinpaste 
>  wrote:
> 
> Hi everybody.
> 
> So I ran into that high mem usage thing. The problem I have with patching is 
> that this is a live system so I can't do it mid day.  Can anybody tell me if 
> it is possible to just restart the vdsm service or does the host have to be 
> in "maintenance mode" before restarting it?  It is using gluster storage, if 
> that makes a difference as well.
> 
> Thanks,
> 
> -- 
> Michael Kleinpaste
> Senior Systems Administrator
> SharperLending, LLC.
> www.SharperLending.com <>
> michael.kleinpa...@sharperlending.com
> (509) 324-1230   Fax: (509) 324-1234
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users