Thank you Eugen and Anthony for your comments. Expanding a cluster with one host with the same amount of disks as in other nodes worked perfectly fine. We also (unwantingly) made a test with expanding with one disk(osd) and we have crossed the near-full state. Also thank you for sharing some more details, it is really informative to get those insights into more corner case scenarios.
I wish that Ceph documentation would have a few more guidelines, something similar to what Clyso has put together. Regards, Robert On Wed, 1 Jul 2026 at 15:07, Eugen Block via ceph-users <[email protected]> wrote: > Hi, > > I would try to avoid adding a host with only few OSDs if other hosts > are equipped differently. Although I haven't checked in a while I > don't expect the behavior to have changed in recent versions. > > So here's my theory: If you add a new host with only one or two OSDs, > Ceph can assign many more PGs than average to the first up OSD in > order to comply with failure domain (probably host). Same goes for the > last OSD during shut down. We had a related case four years ago when > we tried to rebuild OSD nodes in a customer cluster. Since the > customer cluster was still on default values for mon_max_pg_per_osd > and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to > inactive PGs: > > > maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 > > When more OSDs were created on that host (ceph-volume does that > sequentially), the issue resolved eventually, except for some inactive > PGs, failing the MGR resolved those as well. Since you have around 360 > PGs per OSD in average, I assume you increased mon_max_pg_per_osd > (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), > you most likely dodged the "inactive PG" bullet. > > So my recommendation would be to rather add the hosts with all OSDs at > the same time but with crush weight 0. That way you have time to > inspect the outcome (for example with dedicated DB/WAL devices you can > check if they were created according to the specs). Then reweight the > OSDs to the correct value, some users do that gradually to prevent an > overload. > > Note the recent discussion about osd heartbeats when adding multiple > hosts at once: > > > https://lists.ceph.io/hyperkitty/list/[email protected]/thread/RHVPS247M4E4CQAQI2XRXDCVGFYBNSKN/ > > Regards, > Eugen > > Zitat von Robert Lukan via ceph-users <[email protected]>: > > > Hi everyone, > > > > We are currently in the middle of migrating our infrastructure to a > > Proxmox/Ceph stack and ran into a concerning edge case while expanding > the > > cluster. I am hoping to get some advice on the safest way to handle this > > without risking a nearfull OSD. > > > > *The Topology & Situation:* > > > > *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. > > > > *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and > > osd.41). > > > > *Cluster Average:* ~360 PGs per OSD, ~60% utilization. > > > > *Ceph Version:* Squid 19.2.3 > > > > The last time we were expanding the cluster, we had issues when expanding > > it with many OSDs at the same time(it was a memory problem on a host), so > > we decided to be more careful and add 2 disks at the time. > > > > During the backfill phase, we noticed osd.40 catching significantly more > > PGs than osd.41. We paused the migration by setting the nobackfill flag > to > > prevent a potential nearfull situation. > > > > *The Current State (ceph osd df):* > > > > osd.40: 423 PGs, 72.56% used, > > > > osd.41: 307 PGs, 51.87% used, > > > > We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we > > have paused the backfill. > > > > If we were to unset nobackfill, the incoming data for those 218 PGs would > > risk pushing osd.40 past the 85% nearfull threshold. However, our > > balancer refused to generate any optimization plans. It reports: > > "optimize_result": > > "Unable to find further optimization... or distribution is already > perfect" > > > > We "understand" why this is happening: the balancer is evaluating only > when > > backfilling is done. However, it is completely ignoring the transient > acting > > state, which was heavily overloading osd.40 > > > > *Our Temporary Workaround:* To prevent the potential issue, we have > > temporarily set the reweight of osd.40 to 0 (effectively draining it out) > > so we can safely lift the nobackfill flag and let the cluster settle > > without hitting capacity limits. > > > > *My Questions for the Community:* > > > > What is the standard best practice for handling this "small bucket" > > transient imbalance when adding a small number of OSDs to a new host? Or > is > > this just a bad idea in general ? > > > > Is there a way to force the built-in upmap balancer to respect the > > transient/acting capacity of an OSD so it doesn't allow a drive to go > > towards 85% during a migration? I understand we were not there, but being > > 12 % away was "close" enough. > > > > One of the ideas was to reweight the offending OSD, to remove a few PGs, > so > > it would not be so close to nearfull situation. But this approach is > > tedious and requires "baby-sitting". > > > > I guess this might be a good candidate for upmap remap script, but we > were > > not "brave" enough to run in it a production environment the first time. > > > > Thanks, > > > > Robert > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
