Re: [ceph-users] Ceph Balancer per Pool/Crush Unit

[Reed Dier]

I suppose I may have found the solution I was unaware existed.

> balancer optimize  { [...]} :  Run optimizer to create a 
> new plan

So apparently you can create a plan specific to a pool(s).
So just to double check this, I created two plans, plan1 with the hdd pool (and 
not the ssd pool); plan2 with no arguments.

I then ran ceph balancer show planN and also ceph osd crush weight-set dump.
Then compared the values in the weight-set dump against the values in the two 
plans, and concluded that plan1 did not adjust the values for ssd osd’s, which 
is exactly what I was looking for:

> ID  CLASS WEIGHTTYPE NAMESTATUS REWEIGHT PRI-AFF
> -1417.61093 host ceph00
>  24   ssd   1.76109 osd.24   up  1.0 1.0
>  25   ssd   1.76109 osd.25   up  1.0 1.0
>  26   ssd   1.76109 osd.26   up  1.0 1.0
>  27   ssd   1.76109 osd.27   up  1.0 1.0
>  28   ssd   1.76109 osd.28   up  1.0 1.0
>  29   ssd   1.76109 osd.29   up  1.0 1.0
>  30   ssd   1.76109 osd.30   up  1.0 1.0
>  31   ssd   1.76109 osd.31   up  1.0 1.0
>  32   ssd   1.76109 osd.32   up  1.0 1.0
>  33   ssd   1.76109 osd.33   up  1.0 1.0


ceph osd crush weight-set dump
>{
> "bucket_id": -14,
> "weight_set": [
> [
> 1.756317,
> 1.613647,
> 1.733200,
> 1.735825,
> 1.961304,
> 1.583069,
> 1.963791,
> 1.773041,
> 1.890228,
> 1.793457
> ]
> ]
> },


plan1 (no change)
> ceph osd crush weight-set reweight-compat 24 1.756317
> ceph osd crush weight-set reweight-compat 25 1.613647
> ceph osd crush weight-set reweight-compat 26 1.733200
> ceph osd crush weight-set reweight-compat 27 1.735825
> ceph osd crush weight-set reweight-compat 28 1.961304
> ceph osd crush weight-set reweight-compat 29 1.583069
> ceph osd crush weight-set reweight-compat 30 1.963791
> ceph osd crush weight-set reweight-compat 31 1.773041
> ceph osd crush weight-set reweight-compat 32 1.890228
> ceph osd crush weight-set reweight-compat 33 1.793457


plan2 (change)
> ceph osd crush weight-set reweight-compat 24 1.742185
> ceph osd crush weight-set reweight-compat 25 1.608330
> ceph osd crush weight-set reweight-compat 26 1.753393
> ceph osd crush weight-set reweight-compat 27 1.713531
> ceph osd crush weight-set reweight-compat 28 1.964446
> ceph osd crush weight-set reweight-compat 29 1.629001
> ceph osd crush weight-set reweight-compat 30 1.961968
> ceph osd crush weight-set reweight-compat 31 1.738253
> ceph osd crush weight-set reweight-compat 32 1.884098
> ceph osd crush weight-set reweight-compat 33 1.779180


Hopefully this will be helpful for someone else who overlooks this in the -h 
output.

Reed

> On Aug 1, 2018, at 6:05 PM, Reed Dier  wrote:
> 
> Hi Cephers,
> 
> I’m starting to play with the Ceph Balancer plugin after moving to straw2 and 
> running into something I’m surprised I haven’t seen posted here.
> 
> My cluster has two crush roots, one for HDD, one for SSD.
> 
> Right now, HDD’s are a single pool to themselves, SSD’s are a single pool to 
> themselves.
> 
> Using Ceph Balancer Eval, I can see the eval score for the hdd’s (worse), and 
> the ssd’s (better), and the blended score of the cluster overall.
> pool “hdd" score 0.012529 (lower is better)
> pool “ssd" score 0.004654 (lower is better)
> current cluster score 0.008484 (lower is better)
> 
> My problem is that I need to get my hdd’s better, and stop touching my ssd's, 
> because shuffling data wear’s the ssd's unnecessarily, and it has actually 
> gotten the distribution worse over time. https://imgur.com/RVh0jfH 
> 
> You can see that between 06:00 and 09:00 on the second day in the graph that 
> the spread was very tight, and then it expanded back.
> 
> So my question is, how can I run the balancer on just my hdd’s without 
> touching my ssd’s?
> 
> I removed about 15% of the PG’s living on the HDD’s because they were empty.
> I also have two tiers of HDD’s 8TB’s and 2TB’s, but they are roughly equally 
> weighted in crush at the chassis level where my failure domains are 
> configured.
> Hopefully this abbreviated ceph osd tree displays the hierarchy. Multipliers 
> for that bucket on right.
>> ID  CLASS WEIGHTTYPE NAME
>>  -1   218.49353 root default.hdd
>> -10   218.49353 rack default.rack-hdd
>> -7043.66553 chassis hdd-2tb-chassis1 *1
>> -6743.66553 host 

[ceph-users] Ceph Balancer per Pool/Crush Unit

[Reed Dier]

Hi Cephers,

I’m starting to play with the Ceph Balancer plugin after moving to straw2 and 
running into something I’m surprised I haven’t seen posted here.

My cluster has two crush roots, one for HDD, one for SSD.

Right now, HDD’s are a single pool to themselves, SSD’s are a single pool to 
themselves.

Using Ceph Balancer Eval, I can see the eval score for the hdd’s (worse), and 
the ssd’s (better), and the blended score of the cluster overall.
pool “hdd" score 0.012529 (lower is better)
pool “ssd" score 0.004654 (lower is better)
current cluster score 0.008484 (lower is better)

My problem is that I need to get my hdd’s better, and stop touching my ssd's, 
because shuffling data wear’s the ssd's unnecessarily, and it has actually 
gotten the distribution worse over time. https://imgur.com/RVh0jfH 

You can see that between 06:00 and 09:00 on the second day in the graph that 
the spread was very tight, and then it expanded back.

So my question is, how can I run the balancer on just my hdd’s without touching 
my ssd’s?

I removed about 15% of the PG’s living on the HDD’s because they were empty.
I also have two tiers of HDD’s 8TB’s and 2TB’s, but they are roughly equally 
weighted in crush at the chassis level where my failure domains are configured.
Hopefully this abbreviated ceph osd tree displays the hierarchy. Multipliers 
for that bucket on right.
> ID  CLASS WEIGHTTYPE NAME
>  -1   218.49353 root default.hdd
> -10   218.49353 rack default.rack-hdd
> -7043.66553 chassis hdd-2tb-chassis1  *1
> -6743.66553 host hdd-2tb-24-1   *1
>  74   hdd   1.81940 osd.74  *24
> -5543.70700 chassis hdd-8tb-chassis1*4
>  -221.85350 host hdd-8tb-3-1
>   0   hdd   7.28450 osd.0   *3
>  -321.85350 host hdd-8tb-3-1
>   1   hdd   7.28450 osd.1   *3


I assume this doesn’t complicate totally, but figured I would mention it, as I 
assume it is more difficult to equally distribute across OSD’s that are 4:1 
size delta.


If I create a plan plan1 with ceph balancer optimize plan1,
then do a show plan1, I see an entry:
ceph osd crush weight-set reweight-compat $OSD $ArbitraryNumberNearOsdSize

Could I then copy this output, remove entries for SSD OSD’s and then run the 
ceph osd crush weight-set reweight-compat commands in a script?

I and my SSD’s appreciate any insight.

Thanks,

Reed___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com