subject:"Re\: \[ceph\-users\] ceph balancer\: further optimizations\?"

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-22 Thread Stefan Priebe - Profihost AG

Am 21.08.2018 um 17:28 schrieb Gregory Farnum:
> You should be able to create issues now; we had a misconfiguration in
> the tracker following the recent spam attack.
> -Greg
> 
> On Tue, Aug 21, 2018 at 3:07 AM, Stefan Priebe - Profihost AG
>  wrote:
>>
>> Am 21.08.2018 um 12:03 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 21.08.2018 um 11:56 schrieb Dan van der Ster:
 On Tue, Aug 21, 2018 at 11:54 AM Stefan Priebe - Profihost AG
  wrote:
>
> Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
>> On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>>
>>> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
 On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
  wrote:
>
>
> Am 20.08.2018 um 21:52 schrieb Sage Weil:
>> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>>> Hello,
>>>
>>> since loic seems to have left ceph development and his wunderful 
>>> crush
>>> optimization tool isn'T working anymore i'm trying to get a good
>>> distribution with the ceph balancer.
>>>
>>> Sadly it does not work as good as i want.
>>>
>>> # ceph osd df | sort -k8
>>>
>>> show 75 to 83% Usage which is 8% difference which is too much for 
>>> me.
>>> I'm optimization by bytes.
>>>
>>> # ceph balancer eval
>>> current cluster score 0.005420 (lower is better)
>>>
>>> # ceph balancer eval $OPT_NAME
>>> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>>>
>>> I'm unable to optimize further ;-( Is there any chance to optimize
>>> further even in case of more rebelancing?
>>
>> The scoring that the balancer module is doing is currently a hybrid 
>> of pg
>> count, bytes, and object count.  Picking a single metric might help 
>> a bit
>> (as those 3 things are not always perfectly aligned).
>
> Hi,
>
> ok i found a bug in the balancer code which seems to be present in all
> releases.
>
>  861 best_ws = next_ws
>  862 best_ow = next_ow
>
>
> should be:
>
>  861 best_ws = copy.deepcopy(next_ws)
>  862 best_ow = copy.deepcopy(next_ow)
>
> otherwise it does not use the best but the last.

 Interesting... does that change improve things?
>>>
>>> It fixes the following (mgr debug output):
>>> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001180, misplacing 0.000912
>>> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
>>> worse, taking another step
>>> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
>>> default (pools ['cephstor2']) by bytes
>>> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001180, misplacing 0.000912
>>> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
>>> worse, taking another step
>>> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
>>> default (pools ['cephstor2']) by bytes
>>> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001180, misplacing 0.000912
>>> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
>>> worse, taking another step
>>> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
>>> default (pools ['cephstor2']) by bytes
>>> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001180, misplacing 0.000912
>>> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
>>> worse, trying smaller step 0.000244
>>> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
>>> default (pools ['cephstor2']) by bytes
>>> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001152, misplacing 0.001141
>>> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
>>> default (pools ['cephstor2']) by bytes
>>> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
>>> score 0.001152 -> 0.001180, misplacing 0.000912
>>> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
>>> worse, taking another step
>>> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
>>> 0.001155 -> 0.001152
>>>
>>> BUT:
>>> # ceph balancer eval myplan
>>> plan myplan final score 0.001180 (lower is better)
>>>
>>> So the final plan does NOT contain the expected

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Gregory Farnum

You should be able to create issues now; we had a misconfiguration in
the tracker following the recent spam attack.
-Greg

On Tue, Aug 21, 2018 at 3:07 AM, Stefan Priebe - Profihost AG
 wrote:
>
> Am 21.08.2018 um 12:03 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 21.08.2018 um 11:56 schrieb Dan van der Ster:
>>> On Tue, Aug 21, 2018 at 11:54 AM Stefan Priebe - Profihost AG
>>>  wrote:

 Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
> On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
>  wrote:
>>
>>
>> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
>>> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>>>  wrote:


 Am 20.08.2018 um 21:52 schrieb Sage Weil:
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> since loic seems to have left ceph development and his wunderful 
>> crush
>> optimization tool isn'T working anymore i'm trying to get a good
>> distribution with the ceph balancer.
>>
>> Sadly it does not work as good as i want.
>>
>> # ceph osd df | sort -k8
>>
>> show 75 to 83% Usage which is 8% difference which is too much for me.
>> I'm optimization by bytes.
>>
>> # ceph balancer eval
>> current cluster score 0.005420 (lower is better)
>>
>> # ceph balancer eval $OPT_NAME
>> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>>
>> I'm unable to optimize further ;-( Is there any chance to optimize
>> further even in case of more rebelancing?
>
> The scoring that the balancer module is doing is currently a hybrid 
> of pg
> count, bytes, and object count.  Picking a single metric might help a 
> bit
> (as those 3 things are not always perfectly aligned).

 Hi,

 ok i found a bug in the balancer code which seems to be present in all
 releases.

  861 best_ws = next_ws
  862 best_ow = next_ow


 should be:

  861 best_ws = copy.deepcopy(next_ws)
  862 best_ow = copy.deepcopy(next_ow)

 otherwise it does not use the best but the last.
>>>
>>> Interesting... does that change improve things?
>>
>> It fixes the following (mgr debug output):
>> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, trying smaller step 0.000244
>> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001152, misplacing 0.001141
>> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
>> 0.001155 -> 0.001152
>>
>> BUT:
>> # ceph balancer eval myplan
>> plan myplan final score 0.001180 (lower is better)
>>
>> So the final plan does NOT contain the expected optimization. The
>> deepcopy fixes it.
>>
>> After:
>> # ceph balancer eval myplan
>> plan myplan final score 0.001152 (lower is better)
>>
>
> OK that looks

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Stefan Priebe - Profihost AG



Am 21.08.2018 um 12:03 schrieb Stefan Priebe - Profihost AG:
> 
> Am 21.08.2018 um 11:56 schrieb Dan van der Ster:
>> On Tue, Aug 21, 2018 at 11:54 AM Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>> Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
 On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
  wrote:
>
>
> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
>> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>>  wrote:
>>>
>>>
>>> Am 20.08.2018 um 21:52 schrieb Sage Weil:
 On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> Hello,
>
> since loic seems to have left ceph development and his wunderful crush
> optimization tool isn'T working anymore i'm trying to get a good
> distribution with the ceph balancer.
>
> Sadly it does not work as good as i want.
>
> # ceph osd df | sort -k8
>
> show 75 to 83% Usage which is 8% difference which is too much for me.
> I'm optimization by bytes.
>
> # ceph balancer eval
> current cluster score 0.005420 (lower is better)
>
> # ceph balancer eval $OPT_NAME
> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>
> I'm unable to optimize further ;-( Is there any chance to optimize
> further even in case of more rebelancing?

 The scoring that the balancer module is doing is currently a hybrid of 
 pg
 count, bytes, and object count.  Picking a single metric might help a 
 bit
 (as those 3 things are not always perfectly aligned).
>>>
>>> Hi,
>>>
>>> ok i found a bug in the balancer code which seems to be present in all
>>> releases.
>>>
>>>  861 best_ws = next_ws
>>>  862 best_ow = next_ow
>>>
>>>
>>> should be:
>>>
>>>  861 best_ws = copy.deepcopy(next_ws)
>>>  862 best_ow = copy.deepcopy(next_ow)
>>>
>>> otherwise it does not use the best but the last.
>>
>> Interesting... does that change improve things?
>
> It fixes the following (mgr debug output):
> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, trying smaller step 0.000244
> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001152, misplacing 0.001141
> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
> 0.001155 -> 0.001152
>
> BUT:
> # ceph balancer eval myplan
> plan myplan final score 0.001180 (lower is better)
>
> So the final plan does NOT contain the expected optimization. The
> deepcopy fixes it.
>
> After:
> # ceph balancer eval myplan
> plan myplan final score 0.001152 (lower is better)
>

 OK that looks like a bug. Did you create a tracker or PR?
>>>
>>> No not yet. Should i create a PR on github with the fix?
>>
>> Yeah, probably tracker first (requesting luminous,mimic backports),
>> then a PR on master with "Fixes: tracker..."
> 
> Will do but can't find a create button in the tracker. I've opened
> several

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Stefan Priebe - Profihost AG



Am 21.08.2018 um 11:56 schrieb Dan van der Ster:
> On Tue, Aug 21, 2018 at 11:54 AM Stefan Priebe - Profihost AG
>  wrote:
>>
>> Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
>>> On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
>>>  wrote:


 Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>  wrote:
>>
>>
>> Am 20.08.2018 um 21:52 schrieb Sage Weil:
>>> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
 Hello,

 since loic seems to have left ceph development and his wunderful crush
 optimization tool isn'T working anymore i'm trying to get a good
 distribution with the ceph balancer.

 Sadly it does not work as good as i want.

 # ceph osd df | sort -k8

 show 75 to 83% Usage which is 8% difference which is too much for me.
 I'm optimization by bytes.

 # ceph balancer eval
 current cluster score 0.005420 (lower is better)

 # ceph balancer eval $OPT_NAME
 plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)

 I'm unable to optimize further ;-( Is there any chance to optimize
 further even in case of more rebelancing?
>>>
>>> The scoring that the balancer module is doing is currently a hybrid of 
>>> pg
>>> count, bytes, and object count.  Picking a single metric might help a 
>>> bit
>>> (as those 3 things are not always perfectly aligned).
>>
>> Hi,
>>
>> ok i found a bug in the balancer code which seems to be present in all
>> releases.
>>
>>  861 best_ws = next_ws
>>  862 best_ow = next_ow
>>
>>
>> should be:
>>
>>  861 best_ws = copy.deepcopy(next_ws)
>>  862 best_ow = copy.deepcopy(next_ow)
>>
>> otherwise it does not use the best but the last.
>
> Interesting... does that change improve things?

 It fixes the following (mgr debug output):
 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001180, misplacing 0.000912
 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
 worse, taking another step
 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
 default (pools ['cephstor2']) by bytes
 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001180, misplacing 0.000912
 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
 worse, taking another step
 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
 default (pools ['cephstor2']) by bytes
 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001180, misplacing 0.000912
 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
 worse, taking another step
 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
 default (pools ['cephstor2']) by bytes
 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001180, misplacing 0.000912
 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
 worse, trying smaller step 0.000244
 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
 default (pools ['cephstor2']) by bytes
 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001152, misplacing 0.001141
 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
 default (pools ['cephstor2']) by bytes
 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
 score 0.001152 -> 0.001180, misplacing 0.000912
 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
 worse, taking another step
 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
 0.001155 -> 0.001152

 BUT:
 # ceph balancer eval myplan
 plan myplan final score 0.001180 (lower is better)

 So the final plan does NOT contain the expected optimization. The
 deepcopy fixes it.

 After:
 # ceph balancer eval myplan
 plan myplan final score 0.001152 (lower is better)

>>>
>>> OK that looks like a bug. Did you create a tracker or PR?
>>
>> No not yet. Should i create a PR on github with the fix?
> 
> Yeah, probably tracker first (requesting luminous,mimic backports),
> then a PR on master with "Fixes: tracker..."

Will do but can't find a create button in the tracker. I've opened
several reports in the past but right now it seems a can't create a ticket.

Stefan

> 
> -- dan
> 
> 
>>
>>> -- Dan
>>>
>>>
>
> Also, if most of your data is in one pool you can try ceph

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Dan van der Ster

On Tue, Aug 21, 2018 at 11:54 AM Stefan Priebe - Profihost AG
 wrote:
>
> Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
> > On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
> >  wrote:
> >>
> >>
> >> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
> >>> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
> >>>  wrote:
> 
> 
>  Am 20.08.2018 um 21:52 schrieb Sage Weil:
> > On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> >> Hello,
> >>
> >> since loic seems to have left ceph development and his wunderful crush
> >> optimization tool isn'T working anymore i'm trying to get a good
> >> distribution with the ceph balancer.
> >>
> >> Sadly it does not work as good as i want.
> >>
> >> # ceph osd df | sort -k8
> >>
> >> show 75 to 83% Usage which is 8% difference which is too much for me.
> >> I'm optimization by bytes.
> >>
> >> # ceph balancer eval
> >> current cluster score 0.005420 (lower is better)
> >>
> >> # ceph balancer eval $OPT_NAME
> >> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> >>
> >> I'm unable to optimize further ;-( Is there any chance to optimize
> >> further even in case of more rebelancing?
> >
> > The scoring that the balancer module is doing is currently a hybrid of 
> > pg
> > count, bytes, and object count.  Picking a single metric might help a 
> > bit
> > (as those 3 things are not always perfectly aligned).
> 
>  Hi,
> 
>  ok i found a bug in the balancer code which seems to be present in all
>  releases.
> 
>   861 best_ws = next_ws
>   862 best_ow = next_ow
> 
> 
>  should be:
> 
>   861 best_ws = copy.deepcopy(next_ws)
>   862 best_ow = copy.deepcopy(next_ow)
> 
>  otherwise it does not use the best but the last.
> >>>
> >>> Interesting... does that change improve things?
> >>
> >> It fixes the following (mgr debug output):
> >> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001180, misplacing 0.000912
> >> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
> >> worse, taking another step
> >> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
> >> default (pools ['cephstor2']) by bytes
> >> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001180, misplacing 0.000912
> >> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
> >> worse, taking another step
> >> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
> >> default (pools ['cephstor2']) by bytes
> >> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001180, misplacing 0.000912
> >> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
> >> worse, taking another step
> >> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
> >> default (pools ['cephstor2']) by bytes
> >> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001180, misplacing 0.000912
> >> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
> >> worse, trying smaller step 0.000244
> >> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
> >> default (pools ['cephstor2']) by bytes
> >> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001152, misplacing 0.001141
> >> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
> >> default (pools ['cephstor2']) by bytes
> >> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
> >> score 0.001152 -> 0.001180, misplacing 0.000912
> >> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
> >> worse, taking another step
> >> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
> >> 0.001155 -> 0.001152
> >>
> >> BUT:
> >> # ceph balancer eval myplan
> >> plan myplan final score 0.001180 (lower is better)
> >>
> >> So the final plan does NOT contain the expected optimization. The
> >> deepcopy fixes it.
> >>
> >> After:
> >> # ceph balancer eval myplan
> >> plan myplan final score 0.001152 (lower is better)
> >>
> >
> > OK that looks like a bug. Did you create a tracker or PR?
>
> No not yet. Should i create a PR on github with the fix?

Yeah, probably tracker first (requesting luminous,mimic backports),
then a PR on master with "Fixes: tracker..."

-- dan


>
> > -- Dan
> >
> >
> >>>
> >>> Also, if most of your data is in one pool you can try ceph balancer
> >>> eval 
> >>
> >> Already tried this doesn't help much.
> >>
> >> Greets,
> >> Stefan
> >>
> >>
> >>> -- dan
> >>>
> 
>  I'm also using this one:
>

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Stefan Priebe - Profihost AG

Am 21.08.2018 um 11:47 schrieb Dan van der Ster:
> On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
>  wrote:
>>
>>
>> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
>>> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>>>  wrote:


 Am 20.08.2018 um 21:52 schrieb Sage Weil:
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> since loic seems to have left ceph development and his wunderful crush
>> optimization tool isn'T working anymore i'm trying to get a good
>> distribution with the ceph balancer.
>>
>> Sadly it does not work as good as i want.
>>
>> # ceph osd df | sort -k8
>>
>> show 75 to 83% Usage which is 8% difference which is too much for me.
>> I'm optimization by bytes.
>>
>> # ceph balancer eval
>> current cluster score 0.005420 (lower is better)
>>
>> # ceph balancer eval $OPT_NAME
>> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>>
>> I'm unable to optimize further ;-( Is there any chance to optimize
>> further even in case of more rebelancing?
>
> The scoring that the balancer module is doing is currently a hybrid of pg
> count, bytes, and object count.  Picking a single metric might help a bit
> (as those 3 things are not always perfectly aligned).

 Hi,

 ok i found a bug in the balancer code which seems to be present in all
 releases.

  861 best_ws = next_ws
  862 best_ow = next_ow


 should be:

  861 best_ws = copy.deepcopy(next_ws)
  862 best_ow = copy.deepcopy(next_ow)

 otherwise it does not use the best but the last.
>>>
>>> Interesting... does that change improve things?
>>
>> It fixes the following (mgr debug output):
>> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, trying smaller step 0.000244
>> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001152, misplacing 0.001141
>> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
>> default (pools ['cephstor2']) by bytes
>> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
>> score 0.001152 -> 0.001180, misplacing 0.000912
>> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
>> worse, taking another step
>> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
>> 0.001155 -> 0.001152
>>
>> BUT:
>> # ceph balancer eval myplan
>> plan myplan final score 0.001180 (lower is better)
>>
>> So the final plan does NOT contain the expected optimization. The
>> deepcopy fixes it.
>>
>> After:
>> # ceph balancer eval myplan
>> plan myplan final score 0.001152 (lower is better)
>>
> 
> OK that looks like a bug. Did you create a tracker or PR?

No not yet. Should i create a PR on github with the fix?

> -- Dan
> 
> 
>>>
>>> Also, if most of your data is in one pool you can try ceph balancer
>>> eval 
>>
>> Already tried this doesn't help much.
>>
>> Greets,
>> Stefan
>>
>>
>>> -- dan
>>>

 I'm also using this one:
 https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615

 to optimize by bytes only.

 Greets,
 Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-21 Thread Dan van der Ster

On Mon, Aug 20, 2018 at 10:45 PM Stefan Priebe - Profihost AG
 wrote:
>
>
> Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
> > On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
> >  wrote:
> >>
> >>
> >> Am 20.08.2018 um 21:52 schrieb Sage Weil:
> >>> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>  Hello,
> 
>  since loic seems to have left ceph development and his wunderful crush
>  optimization tool isn'T working anymore i'm trying to get a good
>  distribution with the ceph balancer.
> 
>  Sadly it does not work as good as i want.
> 
>  # ceph osd df | sort -k8
> 
>  show 75 to 83% Usage which is 8% difference which is too much for me.
>  I'm optimization by bytes.
> 
>  # ceph balancer eval
>  current cluster score 0.005420 (lower is better)
> 
>  # ceph balancer eval $OPT_NAME
>  plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> 
>  I'm unable to optimize further ;-( Is there any chance to optimize
>  further even in case of more rebelancing?
> >>>
> >>> The scoring that the balancer module is doing is currently a hybrid of pg
> >>> count, bytes, and object count.  Picking a single metric might help a bit
> >>> (as those 3 things are not always perfectly aligned).
> >>
> >> Hi,
> >>
> >> ok i found a bug in the balancer code which seems to be present in all
> >> releases.
> >>
> >>  861 best_ws = next_ws
> >>  862 best_ow = next_ow
> >>
> >>
> >> should be:
> >>
> >>  861 best_ws = copy.deepcopy(next_ws)
> >>  862 best_ow = copy.deepcopy(next_ow)
> >>
> >> otherwise it does not use the best but the last.
> >
> > Interesting... does that change improve things?
>
> It fixes the following (mgr debug output):
> 2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, trying smaller step 0.000244
> 2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001152, misplacing 0.001141
> 2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
> default (pools ['cephstor2']) by bytes
> 2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
> score 0.001152 -> 0.001180, misplacing 0.000912
> 2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
> worse, taking another step
> 2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
> 0.001155 -> 0.001152
>
> BUT:
> # ceph balancer eval myplan
> plan myplan final score 0.001180 (lower is better)
>
> So the final plan does NOT contain the expected optimization. The
> deepcopy fixes it.
>
> After:
> # ceph balancer eval myplan
> plan myplan final score 0.001152 (lower is better)
>

OK that looks like a bug. Did you create a tracker or PR?

-- Dan


> >
> > Also, if most of your data is in one pool you can try ceph balancer
> > eval 
>
> Already tried this doesn't help much.
>
> Greets,
> Stefan
>
>
> > -- dan
> >
> >>
> >> I'm also using this one:
> >> https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615
> >>
> >> to optimize by bytes only.
> >>
> >> Greets,
> >> Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread Stefan Priebe - Profihost AG



Am 20.08.2018 um 22:38 schrieb Dan van der Ster:
> On Mon, Aug 20, 2018 at 10:19 PM Stefan Priebe - Profihost AG
>  wrote:
>>
>>
>> Am 20.08.2018 um 21:52 schrieb Sage Weil:
>>> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
 Hello,

 since loic seems to have left ceph development and his wunderful crush
 optimization tool isn'T working anymore i'm trying to get a good
 distribution with the ceph balancer.

 Sadly it does not work as good as i want.

 # ceph osd df | sort -k8

 show 75 to 83% Usage which is 8% difference which is too much for me.
 I'm optimization by bytes.

 # ceph balancer eval
 current cluster score 0.005420 (lower is better)

 # ceph balancer eval $OPT_NAME
 plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)

 I'm unable to optimize further ;-( Is there any chance to optimize
 further even in case of more rebelancing?
>>>
>>> The scoring that the balancer module is doing is currently a hybrid of pg
>>> count, bytes, and object count.  Picking a single metric might help a bit
>>> (as those 3 things are not always perfectly aligned).
>>
>> Hi,
>>
>> ok i found a bug in the balancer code which seems to be present in all
>> releases.
>>
>>  861 best_ws = next_ws
>>  862 best_ow = next_ow
>>
>>
>> should be:
>>
>>  861 best_ws = copy.deepcopy(next_ws)
>>  862 best_ow = copy.deepcopy(next_ow)
>>
>> otherwise it does not use the best but the last.
> 
> Interesting... does that change improve things?

It fixes the following (mgr debug output):
2018-08-20 22:33:46.078525 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.078574 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.078770 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.156326 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.156374 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.156581 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.233818 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.233868 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.234043 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.313212 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.313714 7f2fbc3b6700  0 mgr[balancer] Score got
worse, trying smaller step 0.000244
2018-08-20 22:33:46.313887 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.391586 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001152, misplacing 0.001141
2018-08-20 22:33:46.393374 7f2fbc3b6700  0 mgr[balancer] Balancing root
default (pools ['cephstor2']) by bytes
2018-08-20 22:33:46.473956 7f2fbc3b6700  0 mgr[balancer] Step result
score 0.001152 -> 0.001180, misplacing 0.000912
2018-08-20 22:33:46.474001 7f2fbc3b6700  0 mgr[balancer] Score got
worse, taking another step
2018-08-20 22:33:46.474046 7f2fbc3b6700  0 mgr[balancer] Success, score
0.001155 -> 0.001152

BUT:
# ceph balancer eval myplan
plan myplan final score 0.001180 (lower is better)

So the final plan does NOT contain the expected optimization. The
deepcopy fixes it.

After:
# ceph balancer eval myplan
plan myplan final score 0.001152 (lower is better)

> 
> Also, if most of your data is in one pool you can try ceph balancer
> eval 

Already tried this doesn't help much.

Greets,
Stefan


> -- dan
> 
>>
>> I'm also using this one:
>> https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615
>>
>> to optimize by bytes only.
>>
>> Greets,
>> Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread David Turner

I didn't ask how many PGs per OSD, I asked how large are your PGs in
comparison to your OSDs.  For instance my primary data pool in my home
cluster has 10914GB of data in it and has 256 PGs.  That means that each PG
accounts for 42GB of data.  I'm using 5TB disks in this cluster.  Each PG
on an OSD accounts for 0.8% of the total disk capacity on each OSD.  With
that % size per PG, I can get a really good balance in that cluster since
even if an OSD had 5 more PGs than another, it's only different by 4% total.

Now let's change the amount of PGs in my cluster.  The same amount of data
and the same size of disk, but only 32 PGs in the pool.  Each PG in that
pool takes up 6.8% of an OSD's available space.  Now even if I only have 2
more PGs on one osd than another, it's off by 13.4%.

On Mon, Aug 20, 2018 at 4:29 PM Stefan Priebe - Profihost AG <
s.pri...@profihost.ag> wrote:

>
> Am 20.08.2018 um 22:13 schrieb David Turner:
> > You might just have too much data per PG.  If a single PG can account
> > for 4% of your OSD, then 9% difference in used space on your OSDs is
> > caused by an OSD having only 2 more PGs than another OSD.  If you do
> > have very large PGs, increasing your PG count in those pools should
> > improve your data distribution.
>
> 4384 pgs
> 91 osds
> 3x replication
>
> 4384 / 91 * 3 => ~ 150 pgs which seems to be even too much - at least
> the doc says around 100 pgs / osd.
>
> >
> > On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  > > wrote:
> >
> > On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > > Hello,
> > >
> > > since loic seems to have left ceph development and his wunderful
> crush
> > > optimization tool isn'T working anymore i'm trying to get a good
> > > distribution with the ceph balancer.
> > >
> > > Sadly it does not work as good as i want.
> > >
> > > # ceph osd df | sort -k8
> > >
> > > show 75 to 83% Usage which is 8% difference which is too much for
> me.
> > > I'm optimization by bytes.
> > >
> > > # ceph balancer eval
> > > current cluster score 0.005420 (lower is better)
> > >
> > > # ceph balancer eval $OPT_NAME
> > > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is
> better)
> > >
> > > I'm unable to optimize further ;-( Is there any chance to optimize
> > > further even in case of more rebelancing?
> >
> > The scoring that the balancer module is doing is currently a hybrid
> > of pg
> > count, bytes, and object count.  Picking a single metric might help
> > a bit
> > (as those 3 things are not always perfectly aligned).
> >
> > s
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread Stefan Priebe - Profihost AG


Am 20.08.2018 um 22:13 schrieb David Turner:
> You might just have too much data per PG.  If a single PG can account
> for 4% of your OSD, then 9% difference in used space on your OSDs is
> caused by an OSD having only 2 more PGs than another OSD.  If you do
> have very large PGs, increasing your PG count in those pools should
> improve your data distribution.

4384 pgs
91 osds
3x replication

4384 / 91 * 3 => ~ 150 pgs which seems to be even too much - at least
the doc says around 100 pgs / osd.

> 
> On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  > wrote:
> 
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > Hello,
> >
> > since loic seems to have left ceph development and his wunderful crush
> > optimization tool isn'T working anymore i'm trying to get a good
> > distribution with the ceph balancer.
> >
> > Sadly it does not work as good as i want.
> >
> > # ceph osd df | sort -k8
> >
> > show 75 to 83% Usage which is 8% difference which is too much for me.
> > I'm optimization by bytes.
> >
> > # ceph balancer eval
> > current cluster score 0.005420 (lower is better)
> >
> > # ceph balancer eval $OPT_NAME
> > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> >
> > I'm unable to optimize further ;-( Is there any chance to optimize
> > further even in case of more rebelancing?
> 
> The scoring that the balancer module is doing is currently a hybrid
> of pg
> count, bytes, and object count.  Picking a single metric might help
> a bit
> (as those 3 things are not always perfectly aligned).
> 
> s
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread Stefan Priebe - Profihost AG



Am 20.08.2018 um 21:52 schrieb Sage Weil:
> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
>> Hello,
>>
>> since loic seems to have left ceph development and his wunderful crush
>> optimization tool isn'T working anymore i'm trying to get a good
>> distribution with the ceph balancer.
>>
>> Sadly it does not work as good as i want.
>>
>> # ceph osd df | sort -k8
>>
>> show 75 to 83% Usage which is 8% difference which is too much for me.
>> I'm optimization by bytes.
>>
>> # ceph balancer eval
>> current cluster score 0.005420 (lower is better)
>>
>> # ceph balancer eval $OPT_NAME
>> plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
>>
>> I'm unable to optimize further ;-( Is there any chance to optimize
>> further even in case of more rebelancing?
> 
> The scoring that the balancer module is doing is currently a hybrid of pg 
> count, bytes, and object count.  Picking a single metric might help a bit 
> (as those 3 things are not always perfectly aligned).

Hi,

ok i found a bug in the balancer code which seems to be present in all
releases.

 861 best_ws = next_ws
 862 best_ow = next_ow


should be:

 861 best_ws = copy.deepcopy(next_ws)
 862 best_ow = copy.deepcopy(next_ow)

otherwise it does not use the best but the last.

I'm also using this one:
https://github.com/ceph/ceph/pull/20665/commits/c161a74ad6cf006cd9b33b40fd7705b67c170615

to optimize by bytes only.

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread David Turner

You might just have too much data per PG.  If a single PG can account for
4% of your OSD, then 9% difference in used space on your OSDs is caused by
an OSD having only 2 more PGs than another OSD.  If you do have very large
PGs, increasing your PG count in those pools should improve your data
distribution.

On Mon, Aug 20, 2018 at 3:59 PM Sage Weil  wrote:

> On Mon, 20 Aug 2018, Stefan Priebe - Profihost AG wrote:
> > Hello,
> >
> > since loic seems to have left ceph development and his wunderful crush
> > optimization tool isn'T working anymore i'm trying to get a good
> > distribution with the ceph balancer.
> >
> > Sadly it does not work as good as i want.
> >
> > # ceph osd df | sort -k8
> >
> > show 75 to 83% Usage which is 8% difference which is too much for me.
> > I'm optimization by bytes.
> >
> > # ceph balancer eval
> > current cluster score 0.005420 (lower is better)
> >
> > # ceph balancer eval $OPT_NAME
> > plan spriebe_2018-08-20_19:36 final score 0.005456 (lower is better)
> >
> > I'm unable to optimize further ;-( Is there any chance to optimize
> > further even in case of more rebelancing?
>
> The scoring that the balancer module is doing is currently a hybrid of pg
> count, bytes, and object count.  Picking a single metric might help a bit
> (as those 3 things are not always perfectly aligned).
>
> s
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

Re: [ceph-users] ceph balancer: further optimizations?

12 matches

Site Navigation

Mail list logo

Footer information