Hi,
since I just stumbled across 'ceph cephadm config-check', is it
enabled in this cluster? Could the MGR have warned you with the health
check "CEPHADM_CHECK_NETWORK_MISSING" after adding hosts but before
deploying OSDs?
It seems that it's not enabled by default.
And one more question: could 'osd_crush_initial_weight = 0' have
prevented this from happening? We have that set in our own cluster to
carefully check if all OSDs are deployed correctly before reweighting.
Thanks,
Eugen
Zitat von Alex Gorbachev via ceph-users <[email protected]>:
osd_heartbeat_min_healthy_ratio determines the minimum percentage of peers
an OSD must be able to reach before it can accurately mark itself up or
avoid marking itself down in the cluster. Default is 1/3 or 33%. OSD
heartbeat decisions are made independently of monitor quorum health. An OSD
can begin reporting peers as down based on heartbeat failures even when
monitor quorum remains healthy.
I was adding a second rack of OSD servers to an existing installation,
assuming that the worst that could happen is that those will be
unreachable. I also missed setting up the cluster_network VLAN on one side
(in the 4x100GbE bond).
I added all the new OSDs at once. The moment that happened, the existing
OSDs saw 50% unreachable, and all production OSDs promptly shut down.
The take away in my recommendation is to add OSDs in such a way that
they're always below osd_heartbeat_min_healthy_ratio, i.e. you have time to
fix connectivity issues before dealing with down OSDs, as I experienced.
--
Alex Gorbachev
ISS/Storcium
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
