On Mon, Feb 26, 2018 at 8:25 AM Oliver Freyermuth < [email protected]> wrote:
> Am 26.02.2018 um 16:59 schrieb Patrick Donnelly: > > On Sun, Feb 25, 2018 at 10:26 AM, Oliver Freyermuth > > <[email protected]> wrote: > >> Looking with: > >> ceph daemon osd.2 perf dump > >> I get: > >> "bluefs": { > >> "gift_bytes": 0, > >> "reclaim_bytes": 0, > >> "db_total_bytes": 84760592384, > >> "db_used_bytes": 78920024064, > >> "wal_total_bytes": 0, > >> "wal_used_bytes": 0, > >> "slow_total_bytes": 0, > >> "slow_used_bytes": 0, > >> so it seems this is almost exclusively RocksDB usage. > >> > >> Is this expected? > > > > Yes. The directory entries are stored in the omap of the objects. This > > will be stored in the RocksDB backend of Bluestore. > > > >> Is there a recommendation on how much MDS storage is needed for a > CephFS with 450 TB? > > > > It seems in the above test you're using about 1KB per inode (file). > > Using that you can extrapolate how much space the data pool needs > > based on your file system usage. (If all you're doing is filling the > > file system with empty files, of course you're going to need an > > unusually large metadata pool.) > > > Many thanks, this helps! > We naturally hope our users will not do this, this stress test was a worst > case - > but the rough number (1 kB per inode) does indeed help a lot, and also the > increase with modifications > of the file as laid out by David. > > Is also the slow backfilling normal? > Will such increase in storage (by many file modifications) at some point > also be reduced, i.e. > is the database compacted / can one trigger that / is there something like > "SQL vacuum"? > > To also answer David's questions in parallel: > - Concerning the slow backfill, I am only talking about the "metadata > OSDs". > They are fully SSD backed, and have no separate device for block.db / > WAL. > - I adjusted backfills up to 128 for those metadata OSDs, the cluster is > currently fully empty, i.e. no client's are doing anything. > There are no slow requests. > Since no clients are doing anything and the rest of the cluster is now > clean (apart from the two backfilling OSDs), > right now there is also no memory pressure at all. > The "clean" OSDs are reading with 7 MB/s each, with 5 % CPU load each. > The OSDs being backfilled have 3.3 % CPU load, and have about 250 kB/s > of write throughput. > Network traffic between the node with the clean OSDs and the > "being-bbackfilled" OSDs is about 1.5 Mbit/s, while there is significantly > more bandwidth available... > - Checking sleeps with: > # ceph -n osd.1 --show-config | grep sleep > osd_recovery_sleep = 0.000000 > osd_recovery_sleep_hdd = 0.100000 > osd_recovery_sleep_hybrid = 0.025000 > osd_recovery_sleep_ssd = 0.000000 > shows there should be 0 sleep. Or is there another way to query? > Check if the OSDs are reporting their stores or their journals to be "rotational" via "ceph osd metadata"? If that's being detected wrong, that would cause them to be using those sleeps. -Greg > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
