Hello, 3 node cluster (2 storage with 2 OSDs one dedicated mon), 3 mons total. Debian Jessie, thus 3.13 kernel and Ceph 0.72.2.
2 of the mons (including the leader) are using around 100MB RSS and one was using about 1.1GB. I did my homework and scoured the ML archives and found at least 2 threads that looked very similar to what I saw with older versions of Ceph. So I first presumed this to be a still (unresolved) issue. One of the threads suggest to use "ceph heap stats" (on the respective nodes) and when I did that the numbers were nearly identical, around 100MB. Then I looked again at the first line of output and realized that I was seeing the data for the leader and the unaffected mon. The "ceph heap" stuff seems to work in a random manner, not even round-robin. After some iterations I got the output for the affected mon and it had 1GB in the freelist like so: --- mon.ceph-02tcmalloc heap stats:------------------------------------------------ MALLOC: 43344408 ( 41.3 MiB) Bytes in use by application MALLOC: + 1110376448 ( 1058.9 MiB) Bytes in page heap freelist MALLOC: + 17100240 ( 16.3 MiB) Bytes in central cache freelist MALLOC: + 42337616 ( 40.4 MiB) Bytes in transfer cache freelist MALLOC: + 23628488 ( 22.5 MiB) Bytes in thread cache freelists MALLOC: + 5615768 ( 5.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1242402968 ( 1184.8 MiB) Actual memory used (physical + swap) MALLOC: + 2990080 ( 2.9 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1245393048 ( 1187.7 MiB) Virtual address space used MALLOC: MALLOC: 8285 Spans in use MALLOC: 22 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ A couple of "ceph heap release" commands later that freelist was emptied and the mon back to normal size. Questions remaining: a) Is that non-deterministic "ceph heap" behavior expected and if yes can it be fixed? b) Any idea what might have caused the heap to grow to that size? c) Shouldn't it have released that memory by itself at some point in time? Regards, Christian -- Christian Balzer Network/Systems Engineer [email protected] Global OnLine Japan/Fusion Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
