Hi,

so.. +1

We don't run compression as far as I know, so that wouldn't be it. We do
actually run a mix of bluestore & filestore - due to the rest of the
cluster predating a stable bluestore by some amount.

The interesting part is - the behavior seems to be specific to our
bluestore nodes.

Below - yellow line, node with 10 x ~4TB SSDs, green line 8 x 800GB SSDs.
Blue line - dump_mempools total bytes for all the OSDs running on the
yellow line. The big dips - forced restarts after having suffered through
after effects of letting linux deal with it by OOM->SIGKILL previously.


‚Äč
A gross extrapolation - "right now" the "memory used" seems to be close
enough to "sum of RSS of ceph-osd processes" running on the machines.

-KJ

On Thu, Mar 1, 2018 at 7:18 PM, Alex Gorbachev <a...@iss-integration.com>
wrote:

> On Thu, Mar 1, 2018 at 5:37 PM, Subhachandra Chandra
> <schan...@grailbio.com> wrote:
> > Even with bluestore we saw memory usage plateau at 3-4GB with 8TB drives
> > filled to around 90%. One thing that does increase memory usage is the
> > number of clients simultaneously sending write requests to a particular
> > primary OSD if the write sizes are large.
>
> We have not seen a memory increase in Ubuntu 16.04, but I also
> observed repeatedly the following phenomenon:
>
> When doing a VMotion in ESXi of a large 3TB file (this generates a log
> of IO requests of small size) to a Ceph pool with compression set to
> force, after some time the Ceph cluster shows a large number of
> blocked requests and eventually timeouts become very large (to the
> point where ESXi aborts the IO due to timeouts).  After abort, the
> blocked/slow requests messages disappear.  There are no OSD errors.  I
> have OSD logs if anyone is interested.
>
> This does not occur when compression is unset.
>
> --
> Alex Gorbachev
> Storcium
>
> >
> > Subhachandra
> >
> > On Thu, Mar 1, 2018 at 6:18 AM, David Turner <drakonst...@gmail.com>
> wrote:
> >>
> >> With default memory settings, the general rule is 1GB ram/1TB OSD.  If
> you
> >> have a 4TB OSD, you should plan to have at least 4GB ram.  This was the
> >> recommendation for filestore OSDs, but it was a bit much memory for the
> >> OSDs.  From what I've seen, this rule is a little more appropriate with
> >> bluestore now and should still be observed.
> >>
> >> Please note that memory usage in a HEALTH_OK cluster is not the same
> >> amount of memory that the daemons will use during recovery.  I have seen
> >> deployments with 4x memory usage during recovery.
> >>
> >> On Thu, Mar 1, 2018 at 8:11 AM Stefan Kooman <ste...@bit.nl> wrote:
> >>>
> >>> Quoting Caspar Smit (caspars...@supernas.eu):
> >>> > Stefan,
> >>> >
> >>> > How many OSD's and how much RAM are in each server?
> >>>
> >>> Currently 7 OSDs, 128 GB RAM. Max wil be 10 OSDs in these servers. 12
> >>> cores (at least one core per OSD).
> >>>
> >>> > bluestore_cache_size=6G will not mean each OSD is using max 6GB RAM
> >>> > right?
> >>>
> >>> Apparently. Sure they will use more RAM than just cache to function
> >>> correctly. I figured 3 GB per OSD would be enough ...
> >>>
> >>> > Our bluestore hdd OSD's with bluestore_cache_size at 1G use ~4GB of
> >>> > total
> >>> > RAM. The cache is a part of the memory usage by bluestore OSD's.
> >>>
> >>> A factor 4 is quite high, isn't it? Where is all this RAM used for
> >>> besides cache? RocksDB?
> >>>
> >>> So how should I size the amount of RAM in a OSD server for 10 bluestore
> >>> SSDs in a
> >>> replicated setup?
> >>>
> >>> Thanks,
> >>>
> >>> Stefan
> >>>
> >>> --
> >>> | BIT BV  http://www.bit.nl/        Kamer van Koophandel 09090351
> >>> | GPG: 0xD14839C6                   +31 318 648 688 / i...@bit.nl
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Kjetil Joergensen <kje...@medallia.com>
SRE, Medallia Inc
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to