Dan, I set norebalance, do a bunch of reweights, then unset norebalance. Degraded PGs will still recover as long as they're not waiting on one of the PGs that is marked as backfilling (which does happen).
What I believe is happening is that when you change CRUSH weights while PGs are actively backfilling, sometimes the backfilling PGs will be remapped again and the peering process takes a bit longer (which blocks I/O on those PGs). However, when 'norebalance' is set, I believe the peering process is much faster which prevents the slow requests. This is just a guess, so I would love for a developer to chime in to confirm whether or not that's the case. Bryan From: Dan van der Ster <d...@vanderster.com> Date: Tuesday, August 8, 2017 at 2:06 AM To: Bryan Stillwell <bstillw...@godaddy.com> Cc: Laszlo Budai <las...@componentsoft.eu>, ceph-users <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] expanding cluster with minimal impact Hi Bryan, How does the norebalance procedure work? You set the flag, increase the weight, then I expect the PGs to stay in remapped unless they're degraded ... why would a PG be degraded just because of a weight change? And then what happens when you unset norebalance? Cheers, Dan On Mon, Aug 7, 2017 at 6:07 PM, Bryan Stillwell <bstillw...@godaddy.com<mailto:bstillw...@godaddy.com>> wrote: Dan, We recently went through an expansion of an RGW cluster and found that we needed 'norebalance' set whenever making CRUSH weight changes to avoid slow requests. We were also increasing the CRUSH weight by 1.0 each time which seemed to reduce the extra data movement we were seeing with smaller weight increases. Maybe something to try out next time? Bryan From: ceph-users <ceph-users-boun...@lists.ceph.com<mailto:ceph-users-boun...@lists.ceph.com>> on behalf of Dan van der Ster <d...@vanderster.com<mailto:d...@vanderster.com>> Date: Friday, August 4, 2017 at 1:59 AM To: Laszlo Budai <las...@componentsoft.eu<mailto:las...@componentsoft.eu>> Cc: ceph-users <ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com>> Subject: Re: [ceph-users] expanding cluster with minimal impact Hi Laszlo, The script defaults are what we used to do a large intervention (the default delta weight is 0.01). For our clusters going any faster becomes disruptive, but this really depends on your cluster size and activity. BTW, in case it wasn't clear, to use this script for adding capacity you need to create the new OSDs to your cluster with initial crush weight = 0.0 osd crush initial weight = 0 osd crush update on start = true -- Dan On Thu, Aug 3, 2017 at 8:12 PM, Laszlo Budai <las...@componentsoft.eu<mailto:las...@componentsoft.eu>> wrote: Dear all, I need to expand a ceph cluster with minimal impact. Reading previous threads on this topic from the list I've found the ceph-gentle-reweight script (https://github.com/cernceph/ceph-scripts/blob/master/tools/ceph-gentle-reweight) created by Dan van der Ster (Thank you Dan for sharing the script with us!). I've done some experiments, and it looks promising, but it is needed to properly set the parameters. Did any of you tested this script before? what is the recommended delta_weight to be used? From the default parameters of the script I can see that the default delta weight is .5% of the target weight that means 200 reweighting cycles. I have experimented with a reweight ratio of 5% while running a fio test on a client. The results were OK (I mean no slow requests), but my test cluster was a very small one. If any of you has done some larger experiments with this script I would be really interested to read about your results. Thank you! Laszlo _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com<mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
