Hi,
I would try to avoid adding a host with only few OSDs if other hosts
are equipped differently. Although I haven't checked in a while I
don't expect the behavior to have changed in recent versions.
So here's my theory: If you add a new host with only one or two OSDs,
Ceph can assign many more PGs than average to the first up OSD in
order to comply with failure domain (probably host). Same goes for the
last OSD during shut down. We had a related case four years ago when
we tried to rebuild OSD nodes in a customer cluster. Since the
customer cluster was still on default values for mon_max_pg_per_osd
and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to
inactive PGs:
maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750
When more OSDs were created on that host (ceph-volume does that
sequentially), the issue resolved eventually, except for some inactive
PGs, failing the MGR resolved those as well. Since you have around 360
PGs per OSD in average, I assume you increased mon_max_pg_per_osd
(default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3),
you most likely dodged the "inactive PG" bullet.
So my recommendation would be to rather add the hosts with all OSDs at
the same time but with crush weight 0. That way you have time to
inspect the outcome (for example with dedicated DB/WAL devices you can
check if they were created according to the specs). Then reweight the
OSDs to the correct value, some users do that gradually to prevent an
overload.
Note the recent discussion about osd heartbeats when adding multiple
hosts at once:
https://lists.ceph.io/hyperkitty/list/[email protected]/thread/RHVPS247M4E4CQAQI2XRXDCVGFYBNSKN/
Regards,
Eugen
Zitat von Robert Lukan via ceph-users <[email protected]>:
Hi everyone,
We are currently in the middle of migrating our infrastructure to a
Proxmox/Ceph stack and ran into a concerning edge case while expanding the
cluster. I am hoping to get some advice on the safest way to handle this
without risking a nearfull OSD.
*The Topology & Situation:*
*Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs.
*Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and
osd.41).
*Cluster Average:* ~360 PGs per OSD, ~60% utilization.
*Ceph Version:* Squid 19.2.3
The last time we were expanding the cluster, we had issues when expanding
it with many OSDs at the same time(it was a memory problem on a host), so
we decided to be more careful and add 2 disks at the time.
During the backfill phase, we noticed osd.40 catching significantly more
PGs than osd.41. We paused the migration by setting the nobackfill flag to
prevent a potential nearfull situation.
*The Current State (ceph osd df):*
osd.40: 423 PGs, 72.56% used,
osd.41: 307 PGs, 51.87% used,
We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we
have paused the backfill.
If we were to unset nobackfill, the incoming data for those 218 PGs would
risk pushing osd.40 past the 85% nearfull threshold. However, our
balancer refused to generate any optimization plans. It reports:
"optimize_result":
"Unable to find further optimization... or distribution is already perfect"
We "understand" why this is happening: the balancer is evaluating only when
backfilling is done. However, it is completely ignoring the transient acting
state, which was heavily overloading osd.40
*Our Temporary Workaround:* To prevent the potential issue, we have
temporarily set the reweight of osd.40 to 0 (effectively draining it out)
so we can safely lift the nobackfill flag and let the cluster settle
without hitting capacity limits.
*My Questions for the Community:*
What is the standard best practice for handling this "small bucket"
transient imbalance when adding a small number of OSDs to a new host? Or is
this just a bad idea in general ?
Is there a way to force the built-in upmap balancer to respect the
transient/acting capacity of an OSD so it doesn't allow a drive to go
towards 85% during a migration? I understand we were not there, but being
12 % away was "close" enough.
One of the ideas was to reweight the offending OSD, to remove a few PGs, so
it would not be so close to nearfull situation. But this approach is
tedious and requires "baby-sitting".
I guess this might be a good candidate for upmap remap script, but we were
not "brave" enough to run in it a production environment the first time.
Thanks,
Robert
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
