> > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs?
It seems that it's not enabled by default. This plausibly could have warned after hosts were added, before OSD deployment, assuming the hosts had already been added to cephadm and the check was enabled. And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. I doubt this would have helped because the CRUSH weight just controls data placement, but osd heartbeat comes at a lower level than that, and that would have fired anyway. -- Alex Gorbachev ISS/Storcium On Wed, Jun 24, 2026 at 5:50 AM Eugen Block via ceph-users < [email protected]> wrote: > Hi, > > since I just stumbled across 'ceph cephadm config-check', is it > enabled in this cluster? Could the MGR have warned you with the health > check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before > deploying OSDs? > > It seems that it's not enabled by default. > > And one more question: could 'osd_crush_initial_weight = 0' have > prevented this from happening? We have that set in our own cluster to > carefully check if all OSDs are deployed correctly before reweighting. > > Thanks, > Eugen > > Zitat von Alex Gorbachev via ceph-users <[email protected]>: > > > osd_heartbeat_min_healthy_ratio determines the minimum percentage of > peers > > an OSD must be able to reach before it can accurately mark itself up or > > avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD > > heartbeat decisions are made independently of monitor quorum health. An > OSD > > can begin reporting peers as down based on heartbeat failures even when > > monitor quorum remains healthy. > > I was adding a second rack of OSD servers to an existing installation, > > assuming that the worst that could happen is that those will be > > unreachable. I also missed setting up the cluster_network VLAN on one > side > > (in the 4x100GbE bond). > > > > I added all the new OSDs at once. The moment that happened, the existing > > OSDs saw 50% unreachable, and all production OSDs promptly shut down. > > > > The take away in my recommendation is to add OSDs in such a way that > > they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time > to > > fix connectivity issues before dealing with down OSDs, as I experienced. > > -- > > Alex Gorbachev > > ISS/Storcium > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
