Hi,

addendum: We're running 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b).

The workload is a mix of 3xreplicated & ec-coded (rbd, cephfs, rgw).

-KJ

On Tue, Mar 6, 2018 at 3:53 PM, Kjetil Joergensen <kje...@medallia.com>
wrote:

> Hi,
>
> so.. +1
>
> We don't run compression as far as I know, so that wouldn't be it. We do
> actually run a mix of bluestore & filestore - due to the rest of the
> cluster predating a stable bluestore by some amount.
>
> The interesting part is - the behavior seems to be specific to our
> bluestore nodes.
>
> Below - yellow line, node with 10 x ~4TB SSDs, green line 8 x 800GB SSDs.
> Blue line - dump_mempools total bytes for all the OSDs running on the
> yellow line. The big dips - forced restarts after having suffered through
> after effects of letting linux deal with it by OOM->SIGKILL previously.
>
>
> ‚Äč
> A gross extrapolation - "right now" the "memory used" seems to be close
> enough to "sum of RSS of ceph-osd processes" running on the machines.
>
> -KJ
>
> On Thu, Mar 1, 2018 at 7:18 PM, Alex Gorbachev <a...@iss-integration.com>
> wrote:
>
>> On Thu, Mar 1, 2018 at 5:37 PM, Subhachandra Chandra
>> <schan...@grailbio.com> wrote:
>> > Even with bluestore we saw memory usage plateau at 3-4GB with 8TB drives
>> > filled to around 90%. One thing that does increase memory usage is the
>> > number of clients simultaneously sending write requests to a particular
>> > primary OSD if the write sizes are large.
>>
>> We have not seen a memory increase in Ubuntu 16.04, but I also
>> observed repeatedly the following phenomenon:
>>
>> When doing a VMotion in ESXi of a large 3TB file (this generates a log
>> of IO requests of small size) to a Ceph pool with compression set to
>> force, after some time the Ceph cluster shows a large number of
>> blocked requests and eventually timeouts become very large (to the
>> point where ESXi aborts the IO due to timeouts).  After abort, the
>> blocked/slow requests messages disappear.  There are no OSD errors.  I
>> have OSD logs if anyone is interested.
>>
>> This does not occur when compression is unset.
>>
>> --
>> Alex Gorbachev
>> Storcium
>>
>> >
>> > Subhachandra
>> >
>> > On Thu, Mar 1, 2018 at 6:18 AM, David Turner <drakonst...@gmail.com>
>> wrote:
>> >>
>> >> With default memory settings, the general rule is 1GB ram/1TB OSD.  If
>> you
>> >> have a 4TB OSD, you should plan to have at least 4GB ram.  This was the
>> >> recommendation for filestore OSDs, but it was a bit much memory for the
>> >> OSDs.  From what I've seen, this rule is a little more appropriate with
>> >> bluestore now and should still be observed.
>> >>
>> >> Please note that memory usage in a HEALTH_OK cluster is not the same
>> >> amount of memory that the daemons will use during recovery.  I have
>> seen
>> >> deployments with 4x memory usage during recovery.
>> >>
>> >> On Thu, Mar 1, 2018 at 8:11 AM Stefan Kooman <ste...@bit.nl> wrote:
>> >>>
>> >>> Quoting Caspar Smit (caspars...@supernas.eu):
>> >>> > Stefan,
>> >>> >
>> >>> > How many OSD's and how much RAM are in each server?
>> >>>
>> >>> Currently 7 OSDs, 128 GB RAM. Max wil be 10 OSDs in these servers. 12
>> >>> cores (at least one core per OSD).
>> >>>
>> >>> > bluestore_cache_size=6G will not mean each OSD is using max 6GB RAM
>> >>> > right?
>> >>>
>> >>> Apparently. Sure they will use more RAM than just cache to function
>> >>> correctly. I figured 3 GB per OSD would be enough ...
>> >>>
>> >>> > Our bluestore hdd OSD's with bluestore_cache_size at 1G use ~4GB of
>> >>> > total
>> >>> > RAM. The cache is a part of the memory usage by bluestore OSD's.
>> >>>
>> >>> A factor 4 is quite high, isn't it? Where is all this RAM used for
>> >>> besides cache? RocksDB?
>> >>>
>> >>> So how should I size the amount of RAM in a OSD server for 10
>> bluestore
>> >>> SSDs in a
>> >>> replicated setup?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Stefan
>> >>>
>> >>> --
>> >>> | BIT BV  http://www.bit.nl/        Kamer van Koophandel 09090351
>> >>> | GPG: 0xD14839C6                   +31 318 648 688 / i...@bit.nl
>> >>> _______________________________________________
>> >>> ceph-users mailing list
>> >>> ceph-users@lists.ceph.com
>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >>
>> >> _______________________________________________
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> >
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
> Kjetil Joergensen <kje...@medallia.com>
> SRE, Medallia Inc
>



-- 
Kjetil Joergensen <kje...@medallia.com>
SRE, Medallia Inc
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to