On 7/24/19 7:15 PM, Kevin Hrpcek wrote:
> I often add 50+ OSDs at a time and my cluster is all NLSAS. Here is what
> I do, you can obviously change the weight increase steps to what you are
> comfortable with. This has worked well for me and my workloads. I've
> sometimes seen peering take longer if I do steps too quickly but I don't
> run any mission critical has to be up 100% stuff and I usually don't
> notice if a pg takes a while to peer.
>
> Add all OSDs with an initial weight of 0. (nothing gets remapped)
> Ensure cluster is healthy.
> Use a for loop to increase weight on all news OSDs to 0.5 with a
> generous sleep between each for peering.
> Let the cluster balance and get healthy or close to healthy.
> Then repeat the previous 2 steps increasing weight by +0.5 or +1.0 until
> I am at the desired weight.
That works, but with newer releases I've been doing this:
- Make sure cluster is HEALTH_OK
- Set the 'norebalance' flag (and usually nobackfill)
- Add all the OSDs
- Wait for the PGs to peer. I usually wait a few minutes
- Remove the norebalance and nobackfill flag
- Wait for HEALTH_OK
Wido
>
> Kevin
>
> On 7/24/19 11:44 AM, Xavier Trilla wrote:
>>
>> Hi,
>>
>>
>>
>> What would be the proper way to add 100 new OSDs to a cluster?
>>
>>
>>
>> I have to add 100 new OSDs to our actual > 300 OSDs cluster, and I
>> would like to know how you do it.
>>
>>
>>
>> Usually, we add them quite slowly. Our cluster is a pure SSD/NVMe one,
>> and it can handle plenty of load, but for the sake of safety -it hosts
>> thousands of VMs via RBD- we usually add them one by one, waiting for
>> a long time between adding each OSD.
>>
>>
>>
>> Obviously this leads to PLENTY of data movement, as each time the
>> cluster geometry changes, data is migrated among all the OSDs. But
>> with the kind of load we have, if we add several OSDs at the same
>> time, some PGs can get stuck for a while, while they peer to the new OSDs.
>>
>>
>>
>> Now that I have to add > 100 new OSDs I was wondering if somebody has
>> some suggestions.
>>
>>
>>
>> Thanks!
>>
>> Xavier.
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
