Hi everyone, We are currently in the middle of migrating our infrastructure to a Proxmox/Ceph stack and ran into a concerning edge case while expanding the cluster. I am hoping to get some advice on the safest way to handle this without risking a nearfull OSD.
*The Topology & Situation:* *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and osd.41). *Cluster Average:* ~360 PGs per OSD, ~60% utilization. *Ceph Version:* Squid 19.2.3 The last time we were expanding the cluster, we had issues when expanding it with many OSDs at the same time(it was a memory problem on a host), so we decided to be more careful and add 2 disks at the time. During the backfill phase, we noticed osd.40 catching significantly more PGs than osd.41. We paused the migration by setting the nobackfill flag to prevent a potential nearfull situation. *The Current State (ceph osd df):* osd.40: 423 PGs, 72.56% used, osd.41: 307 PGs, 51.87% used, We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we have paused the backfill. If we were to unset nobackfill, the incoming data for those 218 PGs would risk pushing osd.40 past the 85% nearfull threshold. However, our balancer refused to generate any optimization plans. It reports: "optimize_result": "Unable to find further optimization... or distribution is already perfect" We "understand" why this is happening: the balancer is evaluating only when backfilling is done. However, it is completely ignoring the transient acting state, which was heavily overloading osd.40 *Our Temporary Workaround:* To prevent the potential issue, we have temporarily set the reweight of osd.40 to 0 (effectively draining it out) so we can safely lift the nobackfill flag and let the cluster settle without hitting capacity limits. *My Questions for the Community:* What is the standard best practice for handling this "small bucket" transient imbalance when adding a small number of OSDs to a new host? Or is this just a bad idea in general ? Is there a way to force the built-in upmap balancer to respect the transient/acting capacity of an OSD so it doesn't allow a drive to go towards 85% during a migration? I understand we were not there, but being 12 % away was "close" enough. One of the ideas was to reweight the offending OSD, to remove a few PGs, so it would not be so close to nearfull situation. But this approach is tedious and requires "baby-sitting". I guess this might be a good candidate for upmap remap script, but we were not "brave" enough to run in it a production environment the first time. Thanks, Robert _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
