Hi,
thanks for the confirmation, glad that it was helpful. I still have
the upmap-remapped procedure on my to-do list, I just played with it
in a lab cluster a while ago but not yet on a real production cluster.
Regards
Eugen
Zitat von Robert Lukan <[email protected]>:
Thank you Eugen and Anthony for your comments. Expanding a cluster with one
host with the same amount of disks as in other nodes worked perfectly fine.
We also (unwantingly) made a test with expanding with one disk(osd) and we
have crossed the near-full state. Also thank you for sharing some more
details, it is really informative to get those insights into more corner
case scenarios.
I wish that Ceph documentation would have a few more guidelines, something
similar to what Clyso has put together.
Regards,
Robert
On Wed, 1 Jul 2026 at 15:07, Eugen Block via ceph-users <[email protected]>
wrote:
Hi,
I would try to avoid adding a host with only few OSDs if other hosts
are equipped differently. Although I haven't checked in a while I
don't expect the behavior to have changed in recent versions.
So here's my theory: If you add a new host with only one or two OSDs,
Ceph can assign many more PGs than average to the first up OSD in
order to comply with failure domain (probably host). Same goes for the
last OSD during shut down. We had a related case four years ago when
we tried to rebuild OSD nodes in a customer cluster. Since the
customer cluster was still on default values for mon_max_pg_per_osd
and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to
inactive PGs:
> maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750
When more OSDs were created on that host (ceph-volume does that
sequentially), the issue resolved eventually, except for some inactive
PGs, failing the MGR resolved those as well. Since you have around 360
PGs per OSD in average, I assume you increased mon_max_pg_per_osd
(default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3),
you most likely dodged the "inactive PG" bullet.
So my recommendation would be to rather add the hosts with all OSDs at
the same time but with crush weight 0. That way you have time to
inspect the outcome (for example with dedicated DB/WAL devices you can
check if they were created according to the specs). Then reweight the
OSDs to the correct value, some users do that gradually to prevent an
overload.
Note the recent discussion about osd heartbeats when adding multiple
hosts at once:
https://lists.ceph.io/hyperkitty/list/[email protected]/thread/RHVPS247M4E4CQAQI2XRXDCVGFYBNSKN/
Regards,
Eugen
Zitat von Robert Lukan via ceph-users <[email protected]>:
> Hi everyone,
>
> We are currently in the middle of migrating our infrastructure to a
> Proxmox/Ceph stack and ran into a concerning edge case while expanding
the
> cluster. I am hoping to get some advice on the safest way to handle this
> without risking a nearfull OSD.
>
> *The Topology & Situation:*
>
> *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs.
>
> *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and
> osd.41).
>
> *Cluster Average:* ~360 PGs per OSD, ~60% utilization.
>
> *Ceph Version:* Squid 19.2.3
>
> The last time we were expanding the cluster, we had issues when expanding
> it with many OSDs at the same time(it was a memory problem on a host), so
> we decided to be more careful and add 2 disks at the time.
>
> During the backfill phase, we noticed osd.40 catching significantly more
> PGs than osd.41. We paused the migration by setting the nobackfill flag
to
> prevent a potential nearfull situation.
>
> *The Current State (ceph osd df):*
>
> osd.40: 423 PGs, 72.56% used,
>
> osd.41: 307 PGs, 51.87% used,
>
> We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we
> have paused the backfill.
>
> If we were to unset nobackfill, the incoming data for those 218 PGs would
> risk pushing osd.40 past the 85% nearfull threshold. However, our
> balancer refused to generate any optimization plans. It reports:
> "optimize_result":
> "Unable to find further optimization... or distribution is already
perfect"
>
> We "understand" why this is happening: the balancer is evaluating only
when
> backfilling is done. However, it is completely ignoring the transient
acting
> state, which was heavily overloading osd.40
>
> *Our Temporary Workaround:* To prevent the potential issue, we have
> temporarily set the reweight of osd.40 to 0 (effectively draining it out)
> so we can safely lift the nobackfill flag and let the cluster settle
> without hitting capacity limits.
>
> *My Questions for the Community:*
>
> What is the standard best practice for handling this "small bucket"
> transient imbalance when adding a small number of OSDs to a new host? Or
is
> this just a bad idea in general ?
>
> Is there a way to force the built-in upmap balancer to respect the
> transient/acting capacity of an OSD so it doesn't allow a drive to go
> towards 85% during a migration? I understand we were not there, but being
> 12 % away was "close" enough.
>
> One of the ideas was to reweight the offending OSD, to remove a few PGs,
so
> it would not be so close to nearfull situation. But this approach is
> tedious and requires "baby-sitting".
>
> I guess this might be a good candidate for upmap remap script, but we
were
> not "brave" enough to run in it a production environment the first time.
>
> Thanks,
>
> Robert
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
