Are you using upmap-remapped to mitigate the thundering herd? And, if you're using the PG autoscaler, to mitigate the thundering herd of backfill that it adds?
> On Apr 6, 2026, at 7:33 AM, Michal Strnad via ceph-users <[email protected]> > wrote: > > Hi. > > We operate a two‑site stretch cluster (version 20.2.0) with cca 2000 OSDs and > another 1000 OSDs to be added. We are concerned with issues related to a > large number of peering connections and remappings when a node with 38 OSDs > is added at once. Adding 38 OSDs triggers remapping of roughly 2000 PGs and > can leave tens of minutes of PGs inactive, which we see reflected in the ceph > status output before the cluster finally stabilizes. > > In a classic (non‑stretch) Ceph cluster the convergence is noticeably quicker > (inactive PGs appear only for a very short interval). In our stretch > deployment, however, the window when PGs are inactive is considerably longer, > consistent with our observations. The network latency between the two sites > is consistently below 10 ms with no packet loss. > > We are currently trying to determine whether this impacts client‑side > read/write simulations via S3, but this has not yet been verified. Our next > step is to test with RBD, which should be more resilient and demonstrable. > Naturally, we would like to control the process in a way that minimizes the > impact on clients. Could you please confirm that the periods of PG inactivity > are short enough to prevent client visible glitches, or even better, could > you recommend an optimal protocol for adding new OSDs to a cluster of this > size to ensure a smooth convergence? Do you have any tested or recommended > timeout adjustments for the stretch configuration? > > Thank you for any feedback > -- > Michal > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
