This is what i get: -------------------------------------------------------- -------------------------------------------------------- -------------------------------------------------------- :/# ceph tell mds.kavehome-mgto-pro-fs01 heap dump 2018-07-24 09:05:19.350720 7fc562ffd700 0 client.1452545 ms_handle_reset on 10.22.0.168:6800/1685786126 2018-07-24 09:05:29.103903 7fc563fff700 0 client.1452548 ms_handle_reset on 10.22.0.168:6800/1685786126 mds.kavehome-mgto-pro-fs01 dumping heap profile now. ------------------------------------------------ MALLOC: 760199640 ( 725.0 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 246962320 ( 235.5 MiB) Bytes in central cache freelist MALLOC: + 43933664 ( 41.9 MiB) Bytes in transfer cache freelist MALLOC: + 41012664 ( 39.1 MiB) Bytes in thread cache freelists MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1102295200 ( 1051.2 MiB) Actual memory used (physical + swap) MALLOC: + 4268335104 ( 4070.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used MALLOC: MALLOC: 33027 Spans in use MALLOC: 19 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory.
-------------------------------------------------------- -------------------------------------------------------- -------------------------------------------------------- :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats 2018-07-24 09:14:25.747706 7f94fffff700 0 client.1452578 ms_handle_reset on 10.22.0.168:6800/1685786126 2018-07-24 09:14:25.754034 7f95057fa700 0 client.1452581 ms_handle_reset on 10.22.0.168:6800/1685786126 mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:------------------------------------------------ MALLOC: 960649328 ( 916.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 108867288 ( 103.8 MiB) Bytes in central cache freelist MALLOC: + 37179424 ( 35.5 MiB) Bytes in transfer cache freelist MALLOC: + 40143000 ( 38.3 MiB) Bytes in thread cache freelists MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1157025952 ( 1103.4 MiB) Actual memory used (physical + swap) MALLOC: + 4213604352 ( 4018.4 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used MALLOC: MALLOC: 33028 Spans in use MALLOC: 19 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. -------------------------------------------------------- -------------------------------------------------------- -------------------------------------------------------- After heap release: :/# ceph tell mds.kavehome-mgto-pro-fs01 heap stats 2018-07-24 09:15:28.540203 7f2f7affd700 0 client.1443339 ms_handle_reset on 10.22.0.168:6800/1685786126 2018-07-24 09:15:28.547153 7f2f7bfff700 0 client.1443342 ms_handle_reset on 10.22.0.168:6800/1685786126 mds.kavehome-mgto-pro-fs01 tcmalloc heap stats:------------------------------------------------ MALLOC: 710315776 ( 677.4 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 246471880 ( 235.1 MiB) Bytes in central cache freelist MALLOC: + 40802848 ( 38.9 MiB) Bytes in transfer cache freelist MALLOC: + 38689304 ( 36.9 MiB) Bytes in thread cache freelists MALLOC: + 10186912 ( 9.7 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1046466720 ( 998.0 MiB) Actual memory used (physical + swap) MALLOC: + 4324163584 ( 4123.8 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 5370630304 ( 5121.8 MiB) Virtual address space used MALLOC: MALLOC: 33177 Spans in use MALLOC: 19 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. The other commands fails with a curl error: Failed to get profile: curl 'http:///pprof/profile?seconds=30' > /root/pprof/.tmp.ceph-mds.1532416424.: Greetings!! 2018-07-24 5:35 GMT+02:00 Yan, Zheng <[email protected]>: > could you profile memory allocation of mds > > http://docs.ceph.com/docs/mimic/rados/troubleshooting/memory-profiling/ > On Tue, Jul 24, 2018 at 7:54 AM Daniel Carrasco <[email protected]> > wrote: > > > > Yeah, is also my thread. This thread was created before lower the cache > size from 512Mb to 8Mb. I thought that maybe was my fault and I did a > misconfiguration, so I've ignored the problem until now. > > > > Greetings! > > > > El mar., 24 jul. 2018 1:00, Gregory Farnum <[email protected]> > escribió: > >> > >> On Mon, Jul 23, 2018 at 11:08 AM Patrick Donnelly <[email protected]> > wrote: > >>> > >>> On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco <[email protected]> > wrote: > >>> > Hi, thanks for your response. > >>> > > >>> > Clients are about 6, and 4 of them are the most of time on standby. > Only two > >>> > are active servers that are serving the webpage. Also we've a > varnish on > >>> > front, so are not getting all the load (below 30% in PHP is not > much). > >>> > About the MDS cache, now I've the mds_cache_memory_limit at 8Mb. > >>> > >>> What! Please post `ceph daemon mds.<name> config diff`, `... perf > >>> dump`, and `... dump_mempools ` from the server the active MDS is on. > >>> > >>> > I've tested > >>> > also 512Mb, but the CPU usage is the same and the MDS RAM usage > grows up to > >>> > 15GB (on a 16Gb server it starts to swap and all fails). With 8Mb, > at least > >>> > the memory usage is stable on less than 6Gb (now is using about 1GB > of RAM). > >>> > >>> We've seen reports of possible memory leaks before and the potential > >>> fixes for those were in 12.2.6. How fast does your MDS reach 15GB? > >>> Your MDS cache size should be configured to 1-8GB (depending on your > >>> preference) so it's disturbing to see you set it so low. > >> > >> > >> See also the thread "[ceph-users] Fwd: MDS memory usage is very high", > which had more discussion of that. The MDS daemon seemingly had 9.5GB of > allocated RSS but only believed 489MB was in use for the cache... > >> -Greg > >> > >>> > >>> > >>> -- > >>> Patrick Donnelly > >>> _______________________________________________ > >>> ceph-users mailing list > >>> [email protected] > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- _________________________________________ Daniel Carrasco Marín Ingeniería para la Innovación i2TIC, S.L. Tlf: +34 911 12 32 84 Ext: 223 www.i2tic.com _________________________________________
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
