issue resolved (somewhat)
dmesg|Memory
shows 1GB reserved (of which 850MB memmap = 1.3% of total memory)
it also shows 650MB absent, apparently this is due to holes in the
memory mapping, which is something hardware specific (it's an AMD Dell
C6145, so it's amd specific or Dell/Bios specific)
anyway, thanks for the feedback
stijn
% slabtop -s c --once |head -10
Active / Total Objects (% used) : 185625 / 205302 (90.4%)
Active / Total Slabs (% used) : 16756 / 16757 (100.0%)
Active / Total Caches (% used) : 101 / 182 (55.5%)
Active / Total Size (% used) : 856865.31K / 859451.65K (99.7%)
Minimum / Average / Maximum Object : 0.02K / 4.19K / 4096.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
384 384 100% 2048.00K 384 1 786432K size-2097152
26276 26250 99% 1.00K 6569 4 26276K ext4_inode_cache
181 181 100% 32.12K 181 1 11584K kmem_cache
very different indeed.
(btw size-2097152 sounds like one of the default name used by hugectl
(or hugeadm) from the hugetlbfs tools). is that mounted in your case?
and are there any hugepages reserved? )
Not that I'd know of. But wasn't there a new feature called
"transparent hugepage support" in 6.1?
thp shows in /proc/meminfo as AnonHugePages
]# grep Huge /proc/meminfo
AnonHugePages: 18432 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
you show check those there too to see where the come from
stijn
Cheers,
Stephan
stijn
On 09/05/2011 05:10 PM, Stephan Wiesand wrote:
Hi Stijn,
On Sep 5, 2011, at 16:24, Stijn De Weirdt wrote:
hi all,
we are having an "issue" with some SL61 nodes. after a reboot, free
reports 1.4GB of memory in use, of which 24+163=187MB buffers+cache.
i'm unable to identify what is holding the memory, and i'd like to
know if others see this too and how i could proceed to find the
culprit.
yes, we see this as well. On a 48 GB system without users or special
processes:
# free -m
total used free shared buffers cached
Mem: 48388 1374 47013 0 30 186
-/+ buffers/cache: 1157 47231
In /proc/meminfo, I find that the difference to what I'd consider
reasonable (and see on a 48GB SL5 system) is due to slabs.
A "slabtop -s c" reveals that it's a "size-2097152" pool accounting
for this. Do you see this as well?
Cheers,
Stephan
(it is a 32core/64GB machine; kernel commandline has
crashkernel=128M@16M (but no difference then eg crashkernel=auto
and kdump is off))
many thanks,
stijn
free
# free -m
total used free shared buffers cached
Mem: 64554 1604 62949 0 24 166
-/+ buffers/cache: 1413 63140
Swap: 16394 0 16394
mem sorted top
top - 16:13:52 up 13 min, 1 user, load average: 0.00, 0.01, 0.01
Tasks: 694 total, 1 running, 693 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 66103768k total, 1643336k used, 64460432k free, 25164k buffers
Swap: 16787916k total, 0k used, 16787916k free, 170552k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2788 root 20 0 37988 25m 2876 S 0.0 0.0 0:00.06 pbs_mom
2653 root 20 0 159m 12m 1472 S 0.0 0.0 0:00.19 ncm-cdispd
2643 root 20 0 138m 5604 840 S 0.0 0.0 0:00.00 cdp-listend
3276 root 20 0 120m 4156 3232 S 0.0 0.0 0:00.07 sshd
2620 root 20 0 745m 3788 1764 S 0.0 0.0 0:00.12 automount
3102 nslcd 20 0 427m 2936 488 S 0.0 0.0 0:00.00 nslcd
3301 root 20 0 103m 1688 1336 S 0.0 0.0 0:00.05 bash
3623 root 20 0 13528 1604 844 R 0.3 0.0 0:00.14 top
1 root 20 0 21416 1544 1240 S 0.0 0.0 0:06.23 init
2482 root 20 0 194m 1484 1108 S 0.0 0.0 0:00.14 qlgc_dsc
2325 root 20 0 242m 1412 928 S 0.0 0.0 0:00.04 rsyslogd
2459 rpcuser 20 0 23112 1168 884 S 0.0 0.0 0:00.00 rpc.statd
2606 root 18 -2 10956 1144 412 S 0.0 0.0 0:00.03 udevd
3164 nscd 20 0 583m 1132 788 S 0.0 0.0 0:00.02 nscd
2697 root 20 0 62040 1064 464 S 0.0 0.0 0:00.00 sshd
943 root 16 -4 10960 1052 316 S 0.0 0.0 0:00.12 udevd
2607 root 18 -2 10956 1052 320 S 0.0 0.0 0:00.00 udevd
2723 root 20 0 112m 1012 380 S 0.0 0.0 0:00.00 crond
2707 root 20 0 22488 992 752 S 0.0 0.0 0:00.03 xinetd
2439 rpc 20 0 18940 908 672 S 0.0 0.0 0:00.04 rpcbind
2568 dbus 20 0 23448 876 604 S 0.0 0.0 0:00.01 dbus-daemon
2972 nagios 20 0 37096 796 452 S 0.0 0.0 0:00.00 nrpe
