Screenshots attached
We have 2 hosts and 24 running vms, all vms are pretty small.
As you can see, top and the dashboard does not agree.
An interesting thing is that if I view the host itself in the engine,
it says under "General" that
Max free Memory for scheduling new VMs: 413360 MB
So maybe it's some sort of caching that's using the memory.
/tony
On Wed, 2018-12-12 at 09:59 -0600, Darrell Budic wrote:
> Yeah, you’re right about 400G, I dropped a digit reading it out of
> your top display.
>
> So what are you seeing in the dashboard, I’m not sure I understand
> the disconnect between the top you shared and what you’re seeing
> there. It shows lots more than 110G in use, I gather? Or are you
> seeing this on the hosts page per host mem use?
>
> > On Dec 12, 2018, at 12:34 AM, Tony Brian Albers <[email protected]> wrote:
> >
> > I'm not following you on the 42G available, the way I see it
> > there's
> > 400+G available:
> >
> > [root@man-001 ~]# free -h
> > total used free shared buff/cache
> > a
> > vailable
> > Mem: 503G 96G 19G 205M 387G
> >
> > 405G
> > Swap: 4.0G 520K 4.0G
> >
> > And here's top sorted by %mem usage:
> >
> > top - 07:29:00 up 104 days, 20:56, 1 user, load average: 0.59,
> > 0.68,
> > 0.67
> > Tasks: 564 total, 1 running, 563 sleeping, 0 stopped, 0
> > zombie
> > %Cpu(s): 1.5 us, 0.2 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0
> > si, 0.0 st
> > KiB Mem : 52807689+total, 20085144 free, 10132981+used,
> > 40666195+buff/cache
> > KiB Swap: 4194300 total, 4193780 free, 520 used.
> > 42491062+avail
> > Mem
> >
> > PID USER PR NI VIRT RES SHR S %CPU
> > %MEM TIME+
> > COMMAND
> > 5517 qemu 20 0 9128268 8.1g 14084
> > S 3.0 1.6 5892:07
> > qemu-kvm
> > 14187 qemu 20 0 9210236 8.1g 14072
> > S 5.3 1.6 6586:00
> > qemu-kvm
> > 12791 qemu 20 0 9272448 8.1g 14140
> > S 14.2 1.6 17452:10
> > qemu-kvm
> > 135526 qemu 20 0 9117748 8.1g 13664
> > S 2.3 1.6 5874:48
> > qemu-kvm
> > 7938 qemu 20 0 9129936 8.1g 13744
> > S 2.3 1.6 22109:28
> > qemu-kvm
> > 11764 qemu 20 0 9275520 8.1g 13720
> > S 3.3 1.6 10679:25
> > qemu-kvm
> > 12066 qemu 20 0 9360552 8.1g 13708
> > S 3.0 1.6 10724:34
> > qemu-kvm
> > 11153 qemu 20 0 9113544 8.1g 13700
> > S 15.6 1.6 19050:12
> > qemu-kvm
> > 12436 qemu 20 0 9161800 8.1g 13712
> > S 16.2 1.6 21268:00
> > qemu-kvm
> > 6902 qemu 20 0 9110480 8.0g 13580
> > S 0.7 1.6 1804:16
> > qemu-kvm
> > 7621 qemu 20 0 9203816 4.8g 14264
> > S 1.7 1.0 3143:35
> > qemu-kvm
> > 6587 qemu 20 0 4880980 4.1g 13744
> > S 0.7 0.8 2354:56
> > qemu-kvm
> > 7249 qemu 20 0 4913084 1.6g 13712
> > S 0.7 0.3 1380:38
> > qemu-kvm
> > 111877 qemu 20 0 1911088 1.1g 14076 S 0.3 0.2
> > 419:58.70
> > qemu-kvm
> > 4602 vdsm 0 -20 4803160 114184 13860
> > S 1.3 0.0 2143:44
> > vdsmd
> > 4058 root 15 -5 1154020 38804 9588
> > S 0.0 0.0 0:00.81
> > supervdsmd
> > 818 root 20 0 84576 35356 34940
> > S 0.0 0.0 1:05.60
> > systemd-journal
> > 3602 root 20 0 1496796 32536 9232 S 0.0 0.0
> > 123:53.70
> > python
> > 2672 root 20 0 358328 30228 7984
> > S 0.0 0.0 0:14.76
> > firewalld
> > 4801 vdsm 20 0 1640996 28904 5484
> > S 0.0 0.0 1265:14
> > python
> >
> >
> > Rebooting a host doesn't help, (I've tried that earlier) the only
> > thing
> > that works is to stop all vm's, reboot all hosts at the same time
> > and
> > start vm's again. Then memory usage shown in the dashboard slowly
> > increases over time again.
> >
> > /tony
> >
> >
> >
> >
> >
> >
> > On Tue, 2018-12-11 at 14:09 -0600, Darrell Budic wrote:
> > > That’s only reporting 42G available of your 512, ok but something
> > > still using it. Try sorting the top by memory %, should be ‘>’
> > > while
> > > it’s running.
> > >
> > > > On Dec 11, 2018, at 1:39 AM, Tony Brian Albers <[email protected]>
> > > > wrote:
> > > >
> > > > Looks ok to me:
> > > >
> > > > top - 08:38:07 up 103 days, 22:05, 1 user, load average:
> > > > 0.68,
> > > > 0.62,
> > > > 0.57
> > > > Tasks: 565 total, 1 running, 564 sleeping, 0 stopped, 0
> > > > zombie
> > > > %Cpu(s): 1.0 us, 0.5 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0
> > > > hi, 0.0
> > > > si, 0.0 st
> > > > KiB Mem : 52807689+total, 22355988 free, 10132873+used,
> > > > 40439219+buff/cache
> > > > KiB Swap: 4194300 total, 4193780 free, 520 used.
> > > > 42492028+avail
> > > > Mem
> > > >
> > > > PID USER PR NI VIRT RES SHR S %CPU
> > > > %MEM TIME+
> > > > COMMAND
> > > > 14187 qemu 20 0 9144668 8.1g 14072
> > > > S 12.6 1.6 6506:46
> > > > qemu-kvm
> > > > 11153 qemu 20 0 9244680 8.1g 13700
> > > > S 4.3 1.6 18881:11
> > > > qemu-kvm
> > > > 12436 qemu 20 0 9292936 8.1g 13712
> > > > S 3.3 1.6 21071:56
> > > > qemu-kvm
> > > > 5517 qemu 20 0 9128268 8.1g 14084
> > > > S 3.0 1.6 5801:03
> > > > qemu-kvm
> > > > 11764 qemu 20 0 9185364 8.1g 13720
> > > > S 3.0 1.6 10585:14
> > > > qemu-kvm
> > > > 7938 qemu 20 0 9252876 8.1g 13744
> > > > S 2.6 1.6 21912:46
> > > > qemu-kvm
> > > > 12791 qemu 20 0 9182292 8.1g 14140
> > > > S 2.6 1.6 17299:36
> > > > qemu-kvm
> > > > 4602 vdsm 0 -20 4803160 <tel:20%204803160>
> > > > 114132 13860
> > > > S 2.3 0.0 2123:45
> > > > vdsmd
> > > > 7621 qemu 20 0 9187424 4.8g 14264
> > > > S 2.3 1.0 3114:25
> > > > qemu-kvm
> > > > 12066 qemu 20 0 9188436 8.1g 13708
> > > > S 2.3 1.6 10629:53
> > > > qemu-kvm
> > > > 135526 qemu 20 0 9298060 8.1g 13664
> > > > S 2.0 1.6 5792:05
> > > > qemu-kvm
> > > > 6587 qemu 20 0 4883036 4.1g 13744
> > > > S 1.3 0.8 2334:54
> > > > qemu-kvm
> > > > 3814 root 20 0 1450200 <tel:1450200> 25096 14208
> > > > S 1.0 0.0 368:03.80
> > > > libvirtd
> > > > 6902 qemu 20 0 9110480 8.0g 13580
> > > > S 1.0 1.6 1787:57
> > > > qemu-kvm
> > > > 7249 qemu 20 0 4913084 1.6g 13712
> > > > S 0.7 0.3 1367:32
> > > > qemu-kvm
> > > >
> > > >
> > > > It looks like it's only in oVirt-engine that there's an issue.
> > > > The
> > > > host
> > > > seems happy enough.
> > > >
> > > > /tony
> > > >
> > > >
> > > >
> > > > On Mon, 2018-12-10 at 20:14 -0600, Darrell Budic wrote:
> > > > > Grab a shell on your hosts and check top memory use quick.
> > > > > Could
> > > > > be
> > > > > VDSMD, in which case restarting the process will give you a
> > > > > temp
> > > > > fix.
> > > > > If you’re running hyperconvered, check your gluster version,
> > > > > there
> > > > > was a leak in versions 3.12.7 - 3.1.12 or so, updating
> > > > > ovirt/gluster
> > > > > is the best fix for that.
> > > > >
> > > > > > On Dec 10, 2018, at 7:36 AM, Tony Brian Albers <[email protected]>
> > > > > > wrote:
> > > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > We have a small test installation here running around 30
> > > > > > vms on
> > > > > > 2
> > > > > > hosts.
> > > > > >
> > > > > > oVirt 4.2.5.3
> > > > > >
> > > > > > The hosts each have 512 GB memory, and the vms are sized
> > > > > > with
> > > > > > 4-8
> > > > > > GB
> > > > > > each.
> > > > > >
> > > > > > I have noticed that over the last months, the memory usage
> > > > > > in
> > > > > > the
> > > > > > dashboard has been increasing and is now showing 946.8 GB
> > > > > > used
> > > > > > of
> > > > > > 1007.2 GB.
> > > > > >
> > > > > > What can be causing this?
> > > > > >
> > > > > > TIA,
> > > > > >
> > > > > > --
> > > > > > --
> > > > > > Tony Albers
> > > > > > Systems Architect
> > > > > > Systems Director, National Cultural Heritage Cluster
> > > > > > Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C,
> > > > > > Denmark.
> > > > > > Tel: +45 2566 2383 / +45 8946 2316
> > > > > > _______________________________________________
> > > > > > Users mailing list -- [email protected]
> > > > > > To unsubscribe send an email to [email protected]
> > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-polic
> > > > > > y/
> > > > > > oVirt Code of Conduct: https://www.ovirt.org/community/abou
> > > > > > t/co
> > > > > > mmun
> > > > > > ity-guidelines/
> > > > > > List Archives: https://lists.ovirt.org/archives/list/users@
> > > > > > ovir
> > > > > > t.or
> > > > > > g/message/SDDH2OC5RBOVYYCLGPOUF6HO676HWI5U/
> > > >
> > > > --
> > > > Tony Albers
> > > > Systems Architect
> > > > Systems Director, National Cultural Heritage Cluster
> > > > Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C,
> > > > Denmark.
> > > > Tel: +45 2566 2383 <tel:+45%202566%202383> / +45 8946 2316
> > > > <tel:+45%208946%202316>
> >
> > --
> > --
> > Tony Albers
> > Systems Architect
> > Systems Director, National Cultural Heritage Cluster
> > Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
> > Tel: +45 2566 2383 / +45 8946 2316
>
>
--
Tony Albers
Systems Architect
Systems Director, National Cultural Heritage Cluster
Royal Danish Library, Victor Albecks Vej 1, 8000 Aarhus C, Denmark.
Tel: +45 2566 2383 / +45 8946 2316
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/6QT5KZZKT4B53V4F6EG4DMEDOBVR7GMH/
