On 8/26/19 1:35 PM, Simon Oosthoek wrote:
> On 26-08-19 13:25, Simon Oosthoek wrote:
>> On 26-08-19 13:11, Wido den Hollander wrote:
>> <snip>
>>>
>>> The reweight might actually cause even more confusion for the balancer.
>>> The balancer uses upmap mode and that re-allocates PGs to different OSDs
>>> if needed.
>>>
>>> Looking at the output send earlier I have some replies. See below.
>>>
>> <snip>
>>>
>>> Looking at this output the balancing seems OK, but from a different
>>> perspective.
>>>
>>> PGs are allocated to OSDs and not Objects nor data. All OSDs have 95~97
>>> Placement Groups allocated.
>>>
>>> That's good! A almost perfect distribution.
>>>
>>> The problem that now rises is the difference in the size of these
>>> Placement Groups as they hold different objects.
>>>
>>> This is one of the side-effects of larger disks. The PGs on them will
>>> grow and this will lead to inbalance between the OSDs.
>>>
>>> I *think* that increasing the amount of PGs on this cluster would help,
>>> but only for the pools which will contain most of the data.
>>>
>>> This will consume a bit more CPU Power and Memory, but on modern systems
>>> this should be less of a problem.
>>>
>>> The good thing is that with Nautilus you can also scale down on the
>>> amount of PGs if things would become a problem.
>>>
>>> More PGs will mean smaller PGs and thus lead to a better data
>>> distribution.
>> <snip>
>>
>> That makes sense, dividing the data in smaller chunks makes it more
>> flexible. The osd nodes are quite underloaded, even with turbo
>> recovery mode on (10, not 32 ;-).
>>
>> When the cluster is in HEALTH_OK again, I'll increase the PGs for the
>> cephfs pools...
>
> On second thought, I reverted my reweight commands and adjusted the PGs,
> which were quite low for some of the pools. The reason they were low is
> that when we first created them, we expected them to be rarely used, but
> then we started filling them just for the filling, and these are
> probably the cause of the unbalance.
>
You should make sure that the pools which contain the most data have the
most PGs.
Although ~100 PGs per OSD is the recommendation it won't hurt to have
~200 PGs as long as you have enough CPU power and Memory. More PGs will
mean better data distribution with such large disks.
> The cluster now has over 8% misplaced objects, so that can take a while...
>
> Cheers
>
> /Simon
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
