With 5x existing hosts and Proxmox, I suspect that Robert's cluster is using RBD with 3x replication and is okay in this respect. There were a couple of bugs wrt uneven failure domain weights back around Firefly, but not recently.
Is it possible that any of your OSDs have old-style override reweights? The REWEIGHT column of `ceph osd tree` ? That can confuse a cluster using pg-upmap. Please follow up with a copy of `ceph osd df tree` Another approach is https://www.clyso.com/us/adding-capacity-to-ceph-the-clyso-way/ upmap-remapped -- be sure to have the latest version -- is a terrific tool. With this strategy you avoid the thundering herd of backfill, and since the balancer will only move a smaller bite at a time, you avoid the dynamic of intermediate states filling up OSDs. A certain Ceph installation added 40 hosts (yes, you read that right) at once to a cluster and ran into this, though with a smaller upgrade I wouldn't expect to see the problem. With this approach most or all of the backfill gets cancelled before it starts, and the balancer moves it very incrementally. So if you start seeing a problem, you can just turn off the balancer while you investigate. You might also crank down how big a bite the balancer will take in one shot. ceph config set mgr target_max_misplaced_ratio .01 > On Jul 1, 2026, at 9:03 AM, Eugen Block via ceph-users <[email protected]> > wrote: > > Hi, > > I would try to avoid adding a host with only few OSDs if other hosts are > equipped differently. Although I haven't checked in a while I don't expect > the behavior to have changed in recent versions. > > So here's my theory: If you add a new host with only one or two OSDs, Ceph > can assign many more PGs than average to the first up OSD in order to comply > with failure domain (probably host). Same goes for the last OSD during shut > down. We had a related case four years ago when we tried to rebuild OSD nodes > in a customer cluster. Since the customer cluster was still on default values > for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) > assigned PGs led to inactive PGs: > >> maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 > > When more OSDs were created on that host (ceph-volume does that > sequentially), the issue resolved eventually, except for some inactive PGs, > failing the MGR resolved those as well. Since you have around 360 PGs per OSD > in average, I assume you increased mon_max_pg_per_osd (default 250), maybe > also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the > "inactive PG" bullet. > > So my recommendation would be to rather add the hosts with all OSDs at the > same time but with crush weight 0. That way you have time to inspect the > outcome (for example with dedicated DB/WAL devices you can check if they were > created according to the specs). Then reweight the OSDs to the correct value, > some users do that gradually to prevent an overload. > > Note the recent discussion about osd heartbeats when adding multiple hosts at > once: > > https://lists.ceph.io/hyperkitty/list/[email protected]/thread/RHVPS247M4E4CQAQI2XRXDCVGFYBNSKN/ > > Regards, > Eugen > > Zitat von Robert Lukan via ceph-users <[email protected]>: > >> Hi everyone, >> >> We are currently in the middle of migrating our infrastructure to a >> Proxmox/Ceph stack and ran into a concerning edge case while expanding the >> cluster. I am hoping to get some advice on the safest way to handle this >> without risking a nearfull OSD. >> >> *The Topology & Situation:* >> >> *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. >> >> *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and >> osd.41). >> >> *Cluster Average:* ~360 PGs per OSD, ~60% utilization. >> >> *Ceph Version:* Squid 19.2.3 >> >> The last time we were expanding the cluster, we had issues when expanding >> it with many OSDs at the same time(it was a memory problem on a host), so >> we decided to be more careful and add 2 disks at the time. >> >> During the backfill phase, we noticed osd.40 catching significantly more >> PGs than osd.41. We paused the migration by setting the nobackfill flag to >> prevent a potential nearfull situation. >> >> *The Current State (ceph osd df):* >> >> osd.40: 423 PGs, 72.56% used, >> >> osd.41: 307 PGs, 51.87% used, >> >> We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we >> have paused the backfill. >> >> If we were to unset nobackfill, the incoming data for those 218 PGs would >> risk pushing osd.40 past the 85% nearfull threshold. However, our >> balancer refused to generate any optimization plans. It reports: >> "optimize_result": >> "Unable to find further optimization... or distribution is already perfect" >> >> We "understand" why this is happening: the balancer is evaluating only when >> backfilling is done. However, it is completely ignoring the transient acting >> state, which was heavily overloading osd.40 >> >> *Our Temporary Workaround:* To prevent the potential issue, we have >> temporarily set the reweight of osd.40 to 0 (effectively draining it out) >> so we can safely lift the nobackfill flag and let the cluster settle >> without hitting capacity limits. >> >> *My Questions for the Community:* >> >> What is the standard best practice for handling this "small bucket" >> transient imbalance when adding a small number of OSDs to a new host? Or is >> this just a bad idea in general ? >> >> Is there a way to force the built-in upmap balancer to respect the >> transient/acting capacity of an OSD so it doesn't allow a drive to go >> towards 85% during a migration? I understand we were not there, but being >> 12 % away was "close" enough. >> >> One of the ideas was to reweight the offending OSD, to remove a few PGs, so >> it would not be so close to nearfull situation. But this approach is >> tedious and requires "baby-sitting". >> >> I guess this might be a good candidate for upmap remap script, but we were >> not "brave" enough to run in it a production environment the first time. >> >> Thanks, >> >> Robert >> _______________________________________________ >> ceph-users mailing list -- [email protected] >> To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
