Siyao Meng created HDDS-17122:
---------------------------------
Summary: Support and test OM and SCM HA with more than 3 nodes
Key: HDDS-17122
URL: https://issues.apache.org/jira/browse/HDDS-17122
Project: Apache Ozone
Issue Type: Improvement
Components: OM HA, SCM HA
Reporter: Siyao Meng
OM HA and SCM HA have no hard limit on the number of nodes:
{{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
number of node IDs, and every node becomes a Ratis peer. In practice only
3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
while one node is down for maintenance (see HDDS-3099).
This umbrella tracks making 5+ node OM and SCM rings a tested, documented
configuration.
h3. Findings so far
New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (sub-task "Add
integration tests for 5-node OM HA and SCM HA") show that the core Ratis
behaviour works: a single Raft group of 5, replication to all nodes, leader
transfer, surviving 2 failures including the leader, no leader with 3 of 5
down, catch-up through checkpoint install, bootstrap 3 -> 5 and decommission 5
-> 3, secure 5-SCM failover.
The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
serve checkpoints once it is leader, so lagging followers never catch up. With
3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and
a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under
client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.
Each item is a sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
