On Fri, Apr 11, 2014 at 11:12 PM, Christian Balzer <[email protected]> wrote: > > Hello, > > 3 node cluster (2 storage with 2 OSDs one dedicated mon), 3 mons total. > Debian Jessie, thus 3.13 kernel and Ceph 0.72.2. > > 2 of the mons (including the leader) are using around 100MB RSS and one > was using about 1.1GB. > > I did my homework and scoured the ML archives and found at least 2 threads > that looked very similar to what I saw with older versions of Ceph. > So I first presumed this to be a still (unresolved) issue. > > One of the threads suggest to use "ceph heap stats" (on the respective > nodes) and when I did that the numbers were nearly identical, around > 100MB. > Then I looked again at the first line of output and realized that I was > seeing the data for the leader and the unaffected mon. > > The "ceph heap" stuff seems to work in a random manner, not even > round-robin. > > After some iterations I got the output for the affected mon and it had 1GB > in the freelist like so: > --- > mon.ceph-02tcmalloc heap > stats:------------------------------------------------ > MALLOC: 43344408 ( 41.3 MiB) Bytes in use by application > MALLOC: + 1110376448 ( 1058.9 MiB) Bytes in page heap freelist > MALLOC: + 17100240 ( 16.3 MiB) Bytes in central cache freelist > MALLOC: + 42337616 ( 40.4 MiB) Bytes in transfer cache freelist > MALLOC: + 23628488 ( 22.5 MiB) Bytes in thread cache freelists > MALLOC: + 5615768 ( 5.4 MiB) Bytes in malloc metadata > MALLOC: ------------ > MALLOC: = 1242402968 ( 1184.8 MiB) Actual memory used (physical + swap) > MALLOC: + 2990080 ( 2.9 MiB) Bytes released to OS (aka unmapped) > MALLOC: ------------ > MALLOC: = 1245393048 ( 1187.7 MiB) Virtual address space used > MALLOC: > MALLOC: 8285 Spans in use > MALLOC: 22 Thread heaps in use > MALLOC: 8192 Tcmalloc page size > ------------------------------------------------ > > A couple of "ceph heap release" commands later that freelist was emptied > and the mon back to normal size. > > Questions remaining: > > a) Is that non-deterministic "ceph heap" behavior expected and if yes can > it be fixed?
You can specify the monitor you want to connect to (by IP) with the "-m" option. > b) Any idea what might have caused the heap to grow to that size? > c) Shouldn't it have released that memory by itself at some point in time? This is some issue between tcmalloc (the memory allocator we use) and the OS that pops up rarely and hasn't been nailed down. Given that, we're not really sure what the issue is. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
