Hi Wannes, The 33% threshold is not applied against the raw total number of OSDs in the cluster. It is applied against the OSD’s heartbeat peer set. With the cluster network VLAN missing, public-network connectivity to MONs could still work, while OSD heartbeats, replication, recovery, and peering over the cluster network failed. -- Alex Gorbachev ISS/Storcium
On Wed, Jun 24, 2026 at 1:25 AM Wannes Smet <[email protected]> wrote: > Hi Alex, > > Lessons learned are always valuable information to share! Thanks. > > Pretty sure I'm missing out on a subtlety here. I can't really put my > finger on it why only osd_heartbeat_min_healthy_ratio caused this, so some > questions from my side. > > At first glance I'd think 50% is still larger than 33%? I also assume OSDs > can be peers even if they never were able to heartbeat each other before > due to crush. > > Maybe this happened because that peering sets aren't perfectly balanced? > So you might have gotten slightly below that 50%, but still above 33%. > Meaning you're <17% away from 33%, "critical mass". But then some > additional peering/activity or the like happened due to the "shock change", > and the whole effectively dropped below 33% cluster wide due to some > cascading effects? > > Thanks, > > Wannes > ------------------------------ > *From:* Alex Gorbachev via ceph-users <[email protected]> > *Sent:* Tuesday, June 23, 2026 18:10 > *To:* ceph-users <[email protected]> > *Subject:* [ceph-users] PSA: be aware of osd_heartbeat_min_healthy_ratio > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers > an OSD must be able to reach before it can accurately mark itself up or > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > heartbeat decisions are made independently of monitor quorum health. An OSD > can begin reporting peers as down based on heartbeat failures even when > monitor quorum remains healthy. > I was adding a second rack of OSD servers to an existing installation, > assuming that the worst that could happen is that those will be > unreachable. I also missed setting up the cluster_network VLAN on one side > (in the 4x100GbE bond). > > I added all the new OSDs at once. The moment that happened, the existing > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > The take away in my recommendation is to add OSDs in such a way that > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to > fix connectivity issues before dealing with down OSDs, as I experienced. > -- > Alex Gorbachev > ISS/Storcium > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
