You can sort of simulate it: >>>> * E.g. if you do something silly like "ceph osd crush reweight osd.1 >>>> 10000" you will see the RSS of osd.28 skyrocket. Reweighting it back down >>>> will not release the memory until you do "heap release".
But this is expected, methinks. Jan > On 09 Sep 2015, at 15:51, Mark Nelson <[email protected]> wrote: > > Yes, under no circumstances is it really ok for an OSD to consume 8GB of RSS! > :) It'd be really swell if we could replicate that kind of memory growth > in-house on demand. > > Mark > > On 09/09/2015 05:56 AM, Jan Schermer wrote: >> Sorry if I wasn't clear. >> Going from 2GB to 8GB is not normal, although some slight bloating is >> expected. In your case it just got much worse than usual for reasons yet >> unknown. >> >> Jan >> >> >>> On 09 Sep 2015, at 12:40, Mariusz Gronczewski >>> <[email protected]> wrote: >>> >>> >>> well I was going by >>> http://ceph.com/docs/master/start/hardware-recommendations/ and planning >>> for 2GB per OSD so that was a suprise.... maybe there should be warning >>> somewhere ? >>> >>> >>> On Wed, 9 Sep 2015 12:21:15 +0200, Jan Schermer <[email protected]> wrote: >>> >>>> The memory gets used for additional PGs on the OSD. >>>> If you were to "swap" PGs between two OSDs, you'll get memory wasted on >>>> both of them because tcmalloc doesn't release it.* >>>> It usually gets stable after few days even during backfills, so it does >>>> get reused if needed. >>>> If for some reason your OSDs get to 8GB RSS then I recommend you just get >>>> more memory, or try disabling tcmalloc which can either help or make it >>>> even worse :-) >>>> >>>> * E.g. if you do something silly like "ceph osd crush reweight osd.1 >>>> 10000" you will see the RSS of osd.28 skyrocket. Reweighting it back down >>>> will not release the memory until you do "heap release". >>>> >>>> Jan >>>> >>>> >>>>> On 09 Sep 2015, at 12:05, Mariusz Gronczewski >>>>> <[email protected]> wrote: >>>>> >>>>> On Tue, 08 Sep 2015 16:14:15 -0500, Chad William Seys >>>>> <[email protected]> wrote: >>>>> >>>>>> Does 'ceph tell osd.* heap release' help with OSD RAM usage? >>>>>> >>>>>> From >>>>>> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-August/003932.html >>>>>> >>>>>> Chad. >>>>> >>>>> it did help now, but cluster is in clean state at the moment. But I >>>>> didnt know that one, thanks. >>>>> >>>>> High memory usage stopped once cluster rebuilt, but I've planned >>>>> cluster to have 2GB per OSD so I needed to add ram to even get to the >>>>> point of ceph starting to rebuild, as some OSD ate up to 8 GBs during >>>>> recover >>>>> >>>>> -- >>>>> Mariusz Gronczewski, Administrator >>>>> >>>>> Efigence S. A. >>>>> ul. Wołoska 9a, 02-583 Warszawa >>>>> T: [+48] 22 380 13 13 >>>>> F: [+48] 22 380 13 14 >>>>> E: [email protected] >>>>> <mailto:[email protected]> >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> [email protected] >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>> >>> >>> -- >>> Mariusz Gronczewski, Administrator >>> >>> Efigence S. A. >>> ul. Wołoska 9a, 02-583 Warszawa >>> T: [+48] 22 380 13 13 >>> F: [+48] 22 380 13 14 >>> E: [email protected] >>> <mailto:[email protected]> >> >> _______________________________________________ >> ceph-users mailing list >> [email protected] >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
