[ceph-users] Re: undersized - really ?

[Anthony D'Atri via ceph-users]

> You have only 8 PGs in total? They take up several hundreds of GB, making one 
> PG quite large.


Agreed.  We don't have information about whether the autoscaler is enabled, but 
the docs do have guidance about pg_num for small clusters.

Fewer than 5 OSDs: set pg_num to 128.
Between 5 and 10 OSDs: set pg_num to 512.
Between 10 and 50 OSDs: set pg_num to 1024.

That simplistically describes a single pool but you get the idea.

A contributing factor is the very small OSDs. OSDs smaller than, say, half a 
GiB have a disproportionate fraction devoted to metadata and tend to not fill 
as one would like.

>> 1 pg backfill_toofull;

This is subtle: an OSD being over the backfill_toofull ratio is different from 
a PG being backfill_toofull, another case where we might have picked more 
distinct names but it's too late :-/


> That's why it's backfill_toofull, it can't store another 200 GB (or so) on 
> that OSD because it would breach the (default) limit of 90%. You need to 
> increase pg_num drastically to decrease your PG sizes.

For reasons that I can't explain, this calculation in some circumstances 
effectively double-counts the data in a PG:  once for (stale) data already on 
the destination, and once for the newer data on the source.  PGs can be 
undersized because CRUSH can't place all shards/replicas because of the space / 
granularity dynamic.  This will be especially pronounced if you're trying to do 
EC here.  I'll guess that this is a sandbox cobbled together from what was 
laying around? If so I suggest sticking to replicated pools with size=2 to 
partly mitigate the limitations.

> As a first shot I would aim for 100 PGs per OSD or so.

For these small and non-uniform OSDs, I might go ever smaller

_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io