[ceph-users] Re: Stop Rebalancing

that, I'll update with what happens overnight. Thanks everyone! > > > Thank you, > > Ray > > > From: Anthony D'Atri > Sent: Wednesday, April 13, 2022 4:49 PM > To: Ceph Users > Subject: [ceph-users] Re: Stop Rebalancin

[ceph-users] Re: Stop Rebalancing

We've done that, I'll update with what happens overnight. Thanks everyone! Thank you, Ray From: Anthony D'Atri Sent: Wednesday, April 13, 2022 4:49 PM To: Ceph Users Subject: [ceph-users] Re: Stop Rebalancing > In any case, isn

[ceph-users] Re: Stop Rebalancing

> In any case, isn't this still the best approach to make all PGs go > active+clean ASAP in this scenario? > > 1. turn off the autoscaler (for those pools, or fully) > 2. for any pool with pg_num_target or pgp_num_target values, get the > current pgp_num X and use it to `ceph osd pool set pg_nu

[ceph-users] Re: Stop Rebalancing

On Wed, Apr 13, 2022 at 7:07 PM Gregory Farnum wrote: > > On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote: > > > > I would set the pg_num, not pgp_num. In older versions of ceph you could > > manipulate these things separately, but in pacific I'm not confident about > > what setting pgp_n

[ceph-users] Re: Stop Rebalancing

ng for the moment.. Thank you, Ray -Original Message- From: Gregory Farnum Sent: Wednesday, April 13, 2022 12:07 PM To: Dan van der Ster Cc: Ray Cunningham ; Ceph Users Subject: Re: [ceph-users] Re: Stop Rebalancing On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote: > >

[ceph-users] Re: Stop Rebalancing

On Wed, Apr 13, 2022 at 10:01 AM Dan van der Ster wrote: > > I would set the pg_num, not pgp_num. In older versions of ceph you could > manipulate these things separately, but in pacific I'm not confident about > what setting pgp_num directly will do in this exact scenario. > > To understand, the

[ceph-users] Re: Stop Rebalancing

I would set the pg_num, not pgp_num. In older versions of ceph you could manipulate these things separately, but in pacific I'm not confident about what setting pgp_num directly will do in this exact scenario. To understand, the difference between these two depends on if you're splitting or mergin

[ceph-users] Re: Stop Rebalancing

Thank you so much, Dan! Can you confirm for me that for pool7, which has 2048/2048 for pg_num and 883/2048 for pgp_num, we should change pg_num or pgp_num? And can they be different for a single pool, or does pg_num and pgp_num have to always be the same? IF we just set pgp_num to 890 we wil

[ceph-users] Re: Stop Rebalancing

Hi, Thanks. norebalance/nobackfill are useful to pause ongoing backfilling, but aren't the best option now to get the PGs to go active+clean and let the mon db come back under control. Unset those before continuing. I think you need to set the pg_num for pool1 to something close to but less than

[ceph-users] Re: Stop Rebalancing

No repair IO and misplaced objects increasing with norebalance and nobackfill set. Thank you, Ray From: Ray Cunningham Sent: Wednesday, April 13, 2022 10:38:29 AM To: Dan van der Ster Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Stop Rebalancing All poo

[ceph-users] Re: Stop Rebalancing

All pools have gone backfillfull. Thank you, Ray Cunningham Systems Engineering and Services Manager keepertechnology (571) 223-7242 From: Ray Cunningham Sent: Wednesday, April 13, 2022 10:15:56 AM To: Dan van der Ster Cc: ceph-

[ceph-users] Re: Stop Rebalancing

Perfect timing, I was just about to reply. We have disabled autoscaler on all pools now. Unfortunately, I can't just copy and paste from this system... `ceph osd pool ls detail` only 2 pools have any difference. pool1: pgnum 940, pgnum target 256, pgpnum 926 pgpnum target 256 pool7: pgnum 2

[ceph-users] Re: Stop Rebalancing

One more thing, could you please also share the `ceph osd pool autoscale-status` ? On Tue, Apr 12, 2022 at 9:50 PM Ray Cunningham wrote: > > Thank you Dan! I will definitely disable autoscaler on the rest of our pools. > I can't get the PG numbers today, but I will try to get them tomorrow. We

[ceph-users] Re: Stop Rebalancing

OK -- here's the tracker for what I mentioned: https://tracker.ceph.com/issues/55303 On Tue, Apr 12, 2022 at 9:50 PM Ray Cunningham wrote: > > Thank you Dan! I will definitely disable autoscaler on the rest of our pools. > I can't get the PG numbers today, but I will try to get them tomorrow. We

[ceph-users] Re: Stop Rebalancing

Thank you Dan! I will definitely disable autoscaler on the rest of our pools. I can't get the PG numbers today, but I will try to get them tomorrow. We definitely want to get this under control. Thank you, Ray   -Original Message- From: Dan van der Ster Sent: Tuesday, April 12, 2022

[ceph-users] Re: Stop Rebalancing

Thanks Matt! I didn't know about nobackfill and norebalance! That could be a good stop gap, as long as there's no issue having it set for weeks. We estimate our legacy bluestore cleanup to take about 3-4 weeks. You are correct, I don't want to cancel it we just need to catch up on other mainte

[ceph-users] Re: Stop Rebalancing

Hi Ray, Disabling the autoscaler on all pools is probably a good idea. At least until https://tracker.ceph.com/issues/53729 is fixed. (You are likely not susceptible to that -- but better safe than sorry). To pause the ongoing PG merges, you can indeed set the pg_num to the current value. This wi

[ceph-users] Re: Stop Rebalancing

It sounds like this is from a PG merge, so I'm going to _guess_ that you don't want to straight up cancel the current backfill and instead pause it to catch your breath. You can set `nobackfill` and/or `norebalance` which should pause the backfill. Alternatively, use `ceph config set osd.* os