Re: [ceph-users] Ceph MDS and hard links
I switched configs to use ms_type: simple and restarted all of our MDS
(there are 3 but only 1 active). It looks like the memory usage crept
back up to the same levels as before. I've included new mempool dump
and heap stat. If I can provide other debug info let me know.
ceph daemon mds.xxx config show | grep simple
"ms_type": "simple",
---
ceph daemon mds.xxx dump_mempools
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 4691601,
"bytes": 4691601
},
"bluestore_alloc": {
"items": 0,
"bytes": 0
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 0,
"bytes": 0
},
"bluestore_cache_other": {
"items": 0,
"bytes": 0
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 0,
"bytes": 0
},
"bluestore_writing_deferred": {
"items": 0,
"bytes": 0
},
"bluestore_writing": {
"items": 0,
"bytes": 0
},
"bluefs": {
"items": 0,
"bytes": 0
},
"buffer_anon": {
"items": 1304976,
"bytes": 6740803514
},
"buffer_meta": {
"items": 1506,
"bytes": 96384
},
"osd": {
"items": 0,
"bytes": 0
},
"osd_mapbl": {
"items": 0,
"bytes": 0
},
"osd_pglog": {
"items": 0,
"bytes": 0
},
"osdmap": {
"items": 8205,
"bytes": 185760
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 163871317,
"bytes": 4080249950
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 169877605,
"bytes": 10826027209
}
}
}
---
ceph tell mds.xxx heap stats
mds.um-mds01 tcmalloc heap
stats:
MALLOC:12521041136 (11941.0 MiB) Bytes in use by application
MALLOC: +49152 (0.0 MiB) Bytes in page heap freelist
MALLOC: +246633144 ( 235.2 MiB) Bytes in central cache freelist
MALLOC: + 6692352 (6.4 MiB) Bytes in transfer cache freelist
MALLOC: + 27288152 ( 26.0 MiB) Bytes in thread cache freelists
MALLOC: + 67895296 ( 64.8 MiB) Bytes in malloc metadata
MALLOC:
MALLOC: = 12869599232 (12273.4 MiB) Actual memory used (physical + swap)
MALLOC: +436740096 ( 416.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:
MALLOC: = 13306339328 (12689.9 MiB) Virtual address space used
MALLOC:
MALLOC: 901356 Spans in use
MALLOC: 1099 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
On Fri, Aug 3, 2018 at 9:14 AM, Yan, Zheng wrote:
> On Fri, Aug 3, 2018 at 8:53 PM Benjeman Meekhof wrote:
>>
>> Thanks, that's useful to know. I've pasted the output you asked for
>> below, thanks for taking a look.
>>
>> Here's the output of dump_mempools:
>>
>> {
>> "mempool": {
>> "by_pool": {
>> "bloom_filter": {
>> "items": 4806709,
>> "bytes": 4806709
>> },
>> "bluestore_alloc": {
>> "items": 0,
>> "bytes": 0
>> },
>> "bluestore_cache_data": {
>> "items": 0,
>> "bytes": 0
>> },
>> "bluestore_cache_onode": {
>> "items": 0,
>> "bytes": 0
>> },
>> "bluestore_cache_other": {
>> "items": 0,
>> "bytes": 0
>> },
>> "bluestore_fsck": {
>> "items": 0,
>> "bytes": 0
>> },
>> "bluestore_txc": {
>> "items": 0,
>> "bytes": 0
>>
Re: [ceph-users] Ceph MDS and hard links
On Fri, Aug 3, 2018 at 8:53 PM Benjeman Meekhof wrote:
>
> Thanks, that's useful to know. I've pasted the output you asked for
> below, thanks for taking a look.
>
> Here's the output of dump_mempools:
>
> {
> "mempool": {
> "by_pool": {
> "bloom_filter": {
> "items": 4806709,
> "bytes": 4806709
> },
> "bluestore_alloc": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_cache_data": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_cache_onode": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_cache_other": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_fsck": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_txc": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_writing_deferred": {
> "items": 0,
> "bytes": 0
> },
> "bluestore_writing": {
> "items": 0,
> "bytes": 0
> },
> "bluefs": {
> "items": 0,
> "bytes": 0
> },
> "buffer_anon": {
> "items": 1303621,
> "bytes": 6643324694
> },
> "buffer_meta": {
> "items": 2397,
> "bytes": 153408
> },
> "osd": {
> "items": 0,
> "bytes": 0
> },
> "osd_mapbl": {
> "items": 0,
> "bytes": 0
> },
> "osd_pglog": {
> "items": 0,
> "bytes": 0
> },
> "osdmap": {
> "items": 8222,
> "bytes": 185840
> },
> "osdmap_mapping": {
> "items": 0,
> "bytes": 0
> },
> "pgmap": {
> "items": 0,
> "bytes": 0
> },
> "mds_co": {
> "items": 160660321,
> "bytes": 4080240182
> },
> "unittest_1": {
> "items": 0,
> "bytes": 0
> },
> "unittest_2": {
> "items": 0,
> "bytes": 0
> }
> },
> "total": {
> "items": 166781270,
> "bytes": 10728710833
> }
> }
> }
>
> and heap_stats:
>
> MALLOC:12418630040 (11843.3 MiB) Bytes in use by application
> MALLOC: + 1310720 (1.2 MiB) Bytes in page heap freelist
> MALLOC: +378986760 ( 361.4 MiB) Bytes in central cache freelist
> MALLOC: + 4713472 (4.5 MiB) Bytes in transfer cache freelist
> MALLOC: + 20722016 ( 19.8 MiB) Bytes in thread cache freelists
> MALLOC: + 62652416 ( 59.8 MiB) Bytes in malloc metadata
> MALLOC:
> MALLOC: = 12887015424 (12290.0 MiB) Actual memory used (physical + swap)
> MALLOC: +309624832 ( 295.3 MiB) Bytes released to OS (aka unmapped)
> MALLOC:
> MALLOC: = 13196640256 (12585.3 MiB) Virtual address space used
> MALLOC:
> MALLOC: 921411 Spans in use
> MALLOC: 20 Thread heaps in use
> MALLOC: 8192 Tcmalloc page size
>
mds does use 4GB memory for its cache, but there are 6GB memory used
by bufferlist. It seems like there is memory leak. Could you try
simple messenger (add "ms type = simple" to 'global' section of
ceph.conf)
> On Wed, Aug 1, 2018 at 10:31 PM, Yan, Zheng wrote:
> > On Thu, Aug 2, 2018 at 3:36 AM Benjeman Meekhof wrote:
> >>
> >> I've been encountering lately a much higher than expected memory usage
> >> on our MDS which doesn't align with the cache_memory limit even
> >> accounting for potential over-runs. Our memory limit is 4GB but the
> >> MDS process is steadily at around 11GB used.
> >>
> >> Coincidentally we also have a new user heavily relying on hard links.
> >> This led me to the following (old) document which says "Hard links are
> >> also supported, although in their current implementation each link
> >> requires a small bit of MDS memory and so there is an implied limit
> >> based on your available memory. "
> >> (https://ceph.com/geen-categorie/cephfs-mds-status-discussion/)
> >>
> >> Is that statement still correct, could it potentially explain why our
> >> memory usage appears so high? As far as I know this is a recent
> >> development and it does very closely correspond to a new user doing a
> >> lot of hardlinking. Ceph Mimic 13.2.1, though we first saw the issue
> >> while still running 13.2.0.
> >>
> >
> > That statement is no longer correct. what are
Re: [ceph-users] Ceph MDS and hard links
Thanks, that's useful to know. I've pasted the output you asked for
below, thanks for taking a look.
Here's the output of dump_mempools:
{
"mempool": {
"by_pool": {
"bloom_filter": {
"items": 4806709,
"bytes": 4806709
},
"bluestore_alloc": {
"items": 0,
"bytes": 0
},
"bluestore_cache_data": {
"items": 0,
"bytes": 0
},
"bluestore_cache_onode": {
"items": 0,
"bytes": 0
},
"bluestore_cache_other": {
"items": 0,
"bytes": 0
},
"bluestore_fsck": {
"items": 0,
"bytes": 0
},
"bluestore_txc": {
"items": 0,
"bytes": 0
},
"bluestore_writing_deferred": {
"items": 0,
"bytes": 0
},
"bluestore_writing": {
"items": 0,
"bytes": 0
},
"bluefs": {
"items": 0,
"bytes": 0
},
"buffer_anon": {
"items": 1303621,
"bytes": 6643324694
},
"buffer_meta": {
"items": 2397,
"bytes": 153408
},
"osd": {
"items": 0,
"bytes": 0
},
"osd_mapbl": {
"items": 0,
"bytes": 0
},
"osd_pglog": {
"items": 0,
"bytes": 0
},
"osdmap": {
"items": 8222,
"bytes": 185840
},
"osdmap_mapping": {
"items": 0,
"bytes": 0
},
"pgmap": {
"items": 0,
"bytes": 0
},
"mds_co": {
"items": 160660321,
"bytes": 4080240182
},
"unittest_1": {
"items": 0,
"bytes": 0
},
"unittest_2": {
"items": 0,
"bytes": 0
}
},
"total": {
"items": 166781270,
"bytes": 10728710833
}
}
}
and heap_stats:
MALLOC:12418630040 (11843.3 MiB) Bytes in use by application
MALLOC: + 1310720 (1.2 MiB) Bytes in page heap freelist
MALLOC: +378986760 ( 361.4 MiB) Bytes in central cache freelist
MALLOC: + 4713472 (4.5 MiB) Bytes in transfer cache freelist
MALLOC: + 20722016 ( 19.8 MiB) Bytes in thread cache freelists
MALLOC: + 62652416 ( 59.8 MiB) Bytes in malloc metadata
MALLOC:
MALLOC: = 12887015424 (12290.0 MiB) Actual memory used (physical + swap)
MALLOC: +309624832 ( 295.3 MiB) Bytes released to OS (aka unmapped)
MALLOC:
MALLOC: = 13196640256 (12585.3 MiB) Virtual address space used
MALLOC:
MALLOC: 921411 Spans in use
MALLOC: 20 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
On Wed, Aug 1, 2018 at 10:31 PM, Yan, Zheng wrote:
> On Thu, Aug 2, 2018 at 3:36 AM Benjeman Meekhof wrote:
>>
>> I've been encountering lately a much higher than expected memory usage
>> on our MDS which doesn't align with the cache_memory limit even
>> accounting for potential over-runs. Our memory limit is 4GB but the
>> MDS process is steadily at around 11GB used.
>>
>> Coincidentally we also have a new user heavily relying on hard links.
>> This led me to the following (old) document which says "Hard links are
>> also supported, although in their current implementation each link
>> requires a small bit of MDS memory and so there is an implied limit
>> based on your available memory. "
>> (https://ceph.com/geen-categorie/cephfs-mds-status-discussion/)
>>
>> Is that statement still correct, could it potentially explain why our
>> memory usage appears so high? As far as I know this is a recent
>> development and it does very closely correspond to a new user doing a
>> lot of hardlinking. Ceph Mimic 13.2.1, though we first saw the issue
>> while still running 13.2.0.
>>
>
> That statement is no longer correct. what are output of "ceph
> daemon mds.x dump_mempools" and "ceph tell mds.x heap stats"?
>
>
>> thanks,
>> Ben
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph MDS and hard links
On Thu, Aug 2, 2018 at 3:36 AM Benjeman Meekhof wrote: > > I've been encountering lately a much higher than expected memory usage > on our MDS which doesn't align with the cache_memory limit even > accounting for potential over-runs. Our memory limit is 4GB but the > MDS process is steadily at around 11GB used. > > Coincidentally we also have a new user heavily relying on hard links. > This led me to the following (old) document which says "Hard links are > also supported, although in their current implementation each link > requires a small bit of MDS memory and so there is an implied limit > based on your available memory. " > (https://ceph.com/geen-categorie/cephfs-mds-status-discussion/) > > Is that statement still correct, could it potentially explain why our > memory usage appears so high? As far as I know this is a recent > development and it does very closely correspond to a new user doing a > lot of hardlinking. Ceph Mimic 13.2.1, though we first saw the issue > while still running 13.2.0. > That statement is no longer correct. what are output of "ceph daemon mds.x dump_mempools" and "ceph tell mds.x heap stats"? > thanks, > Ben > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph MDS and hard links
I've been encountering lately a much higher than expected memory usage on our MDS which doesn't align with the cache_memory limit even accounting for potential over-runs. Our memory limit is 4GB but the MDS process is steadily at around 11GB used. Coincidentally we also have a new user heavily relying on hard links. This led me to the following (old) document which says "Hard links are also supported, although in their current implementation each link requires a small bit of MDS memory and so there is an implied limit based on your available memory. " (https://ceph.com/geen-categorie/cephfs-mds-status-discussion/) Is that statement still correct, could it potentially explain why our memory usage appears so high? As far as I know this is a recent development and it does very closely correspond to a new user doing a lot of hardlinking. Ceph Mimic 13.2.1, though we first saw the issue while still running 13.2.0. thanks, Ben ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
