Hi,
Start by setting:
ceph config set mgr mgr/balancer/upmap_max_deviation 1
This configures the balancer to squeeze the OSDs to within 1 PG of eachother.
I'm starting to think this should be the default.
Cheers, dan
On Wed, Jul 28, 2021 at 9:08 AM Manuel Holtgrewe <[email protected]> wrote:
>
> Dear all,
>
> I'm running Ceph 14.2.11. I have 140 HDDs in my cluster of 4 nodes, 35 HDDs
> per node. I am observing fill ratios of 66% to 70% of OSDs and then one
> with 82% (see attached ceph-osd-df.txt for output of "ceph osd df").
>
> Previously, I had problems with single OSDs filling up to 85% and then
> everything coming to a grinding halt. Ideally, I would like to have all OSD
> fill grade to be close to the mean of 67%... At the very least I need to
> get the 82% OSD back into the range.
>
> I have upmap balancing enabled and the balancer says:
>
> # ceph balancer status
> {
> "last_optimize_duration": "0:00:00.053686",
> "plans": [],
> "mode": "upmap",
> "active": true,
> "optimize_result": "Unable to find further optimization, or pool(s)
> pg_num is decreasing, or distribution is already perfect",
> "last_optimize_started": "Wed Jul 28 09:03:02 2021"
> }
>
> Creating an offline balancing plan looks like this:
>
> # ceph osd getmap -o om
> got osdmap epoch 281708
> # osdmaptool om --upmap out.txt --upmap-pool hdd_ec --upmap-deviation 1
> --upmap-active
> osdmaptool: osdmap file 'om'
> writing upmap command output to: out.txt
> checking for upmap cleanups
> upmap, max-count 10, max deviation 1
> limiting to pools hdd_ec ([3])
> pools hdd_ec
> prepared 0/10 changes
> Time elapsed 0.0275739 secs
> Unable to find further optimization, or distribution is already perfect
> osd.0 pgs 43
> [...]
> # wc -l out.txt
> 0 out.txt
>
> Does anyone have a suggestion how to proceed getting the 82% OSD closer to
> the mean fill ratio (and maybe the other OSDs as well?)
>
> Thanks,
> Manuel
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
