Thanks to everyone who replied. After setting osd_recovery_sleep_hdd to 0 and changing osd-max-backfills to 16, my recovery throughput increased from < 1MBPS to 40-60MPBS and finished up late last night.
The cluster is mopping up a bunch of queued deep scrubs, but is otherwise now healthy. I do have one remaining question - the cluster now shows 81TB of free space, but the data pool only shows 22.8TB of free space. I was expecting/hoping to see the free space value for the pool grow more after doubling the capacity of the cluster (it previously had 21 OSDs w/ 2.7TB SAS drives; I just added 12 more OSDs w/ 5.5TB drives). Are my expectations flawed, or is there something I can do to prod Ceph into growing the data pool free space? [image: image.png] [image: image.png] On Fri, Aug 28, 2020 at 9:37 AM <[email protected]> wrote: > Dallas; > > I would expect so, yes. > > I wouldn't be surprised to see the used percentage slowly drop as the > recovery / rebalance progresses. I believe that the pool free space number > is based on the free space of the most filled OSD under any of the PGs, so > I expect the free space will go up as your near-full OSDs drain. > > I've added OSDs to one of our clusters, once, and the recovery / rebalance > completed fairly quickly. I don't remember how the pool sizes progressed. > I'm going to need to expand our other cluster in the next couple of months, > so follow up on how this proceeds would be appreciated. > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International, Inc. > [email protected] > www.PerformAir.com > > > From: Dallas Jones [mailto:[email protected]] > Sent: Friday, August 28, 2020 7:58 AM > To: Florian Pritz > Cc: [email protected]; Dominic Hilsbos > Subject: Re: [ceph-users] Re: Cluster degraded after adding OSDs to > increase capacity > > Thanks for the reply. I dialed up the value for max backfills yesterday, > which increased my recovery throughput from about 1mbps to 5ish. After > tweaking osd_recovery_sleep_hdd, I'm seeing 50-60MBPS - which is fairly > epic. No clients are currently using this cluster, so I'm not worried about > tanking client performance. > > One remaining question: Will the pool sizes begin to adjust once the > recovery process is complete? Per the following screenshot, my data pool is > ~94% full... > > > > On Fri, Aug 28, 2020 at 4:31 AM Florian Pritz < > [email protected]> wrote: > On Thu, Aug 27, 2020 at 05:56:22PM +0000, [email protected] wrote: > > 2) Adjust performance settings to allow the data movement to go > faster. Again, I don't have those setting immediately to hand, but > Googling something like 'ceph recovery tuning,' or searching this list, > should point you in the right direction. Notice that you only have 6 PGs > trying to move at a time, with 2 blocked on your near-full OSDs (8 & 19). > I believe; by default, each OSD daemon is only involved in 1 data movement > at a time. The tradeoff here is user activity suffers if you adjust to > favor recovery, however, with the cluster in ERROR status, I suspect user > activity is already suffering. > > We've set osd_max_backfills to 16 in the config and when necessary we > manually change the runtime value of osd_recovery_sleep_hdd. It defaults > to 0.1 seconds of wait time between objects (I think?). If you really > want fast recovery try this additional change: > > ceph tell osd.\* config set osd_recovery_sleep_hdd 0 > > Be warned though, this will seriously affect client performance. Then > again it can bump your recovery speed by multiple orders of magnitude. > If you want to go back to how things were, set it back to 0.1 instead of > 0. It may take a couple of seconds (maybe a minute) until performance > for clients starts to improve. I guess the OSDs are too busy with > recovery to instantly accept the changed value. > > Florian >
_______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
