[
https://issues.apache.org/jira/browse/HDDS-17122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17122:
------------------------------
Description:
OM HA and SCM HA have no hard limit on the number of nodes:
{{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
number of node IDs, and every node becomes a Ratis peer. In practice only
3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
while one node is down for maintenance (see HDDS-3099).
This umbrella tracks making 5+ node OM and SCM rings a tested, documented
configuration.
h3. Ratis background
* *Static rings:* Ratis has supported Raft groups of any size from the start.
Vote and commit majorities are computed for N peers ({{num > size()/2}} and
{{indices[(n-1)/2]}}, unchanged since Ratis 0.2.0), and Ratis' core test suites
(RaftBasicTests, LeaderElectionTests) have run 5-server clusters since 2016. I
found no RATIS bug specific to a group that is configured with more than 3
peers from the start.
* *Growing a ring at runtime* ({{ozone om --bootstrap}}, {{ozone scm
--bootstrap}}, OM/SCM decommission) is a Raft membership change:
OzoneManagerRatisServer#addOMToRatisRing / removeOMFromRatisRing and
SCMRatisServerImpl#addSCM / removeSCM call {{setConfiguration}}. Ratis had bugs
there, several of which bite exactly when 2 peers fail, which only a 5+ node
ring is meant to survive:
** RATIS-97 (fixed in 0.2.0): after setConfiguration adds servers, the old
leader may keep running without a majority.
** RATIS-1824 (2.5.0 / 3.0.0): membership change fails when a listener is
present (relevant to Listener OMs).
** RATIS-1912 (3.0.0): infinite election after replacing 2 of 3 peers.
** RATIS-2274 (3.2.0): a newly added peer may keep an outdated configuration;
if the leader and one more peer then fail, elections deadlock.
** RATIS-2283 (3.2.0): reconfiguration can block forever after a log appender
thread restart.
* *Still open:* RATIS-2657, the leader re-initiates InstallSnapshot against a
not-ready follower with no backoff (measured on a 5-node group). Related to
HDDS-17127.
* *Ratis version per Ozone release:* 1.0.0 -> 1.0.0, 1.1.0 -> 2.0.0, 1.2.x ->
2.2.0, 1.3.0 -> 2.4.0, 1.4.0 -> 3.0.1, 1.4.1 -> 3.1.1, 2.0.0 -> 3.1.2, 2.1.0 ->
3.2.0, 2.2.0 -> 3.2.1, master -> 3.3.1.
So, as far as known Ratis bugs go: a ring configured with 5+ nodes from the
start needs no particular Ratis release, while growing or shrinking a ring past
3 nodes by bootstrap/decommission needs Ozone 2.1.0+ (Ratis 3.2.0+). The
remaining known problems are on the Ozone side (the sub-tasks here) plus
RATIS-2657.
h3. Findings so far
New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123) show
that the core Ratis behaviour works: a single Raft group of 5, replication to
all nodes, leader transfer, surviving 2 failures including the leader, no
leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5
and decommission 5 -> 3, secure 5-SCM failover.
The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
serve checkpoints once it is leader, so lagging followers never catch up. With
3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and
a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under
client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.
Each item is a sub-task.
was:
OM HA and SCM HA have no hard limit on the number of nodes:
{{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
number of node IDs, and every node becomes a Ratis peer. In practice only
3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
while one node is down for maintenance (see HDDS-3099).
This umbrella tracks making 5+ node OM and SCM rings a tested, documented
configuration.
h3. Findings so far
New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123) show
that the core Ratis behaviour works: a single Raft group of 5, replication to
all nodes, leader transfer, surviving 2 failures including the leader, no
leader with 3 of 5 down, catch-up through checkpoint install, bootstrap 3 -> 5
and decommission 5 -> 3, secure 5-SCM failover.
The problems are around it:
* *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
serve checkpoints once it is leader, so lagging followers never catch up. With
3 SCMs this is rare (the primordial SCM usually stays leader); with 5 SCMs and
a few failovers the 5-node test hits it reliably.
* Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent under
client retry.
* OM clients cannot reach a leader that is not in their configured OM list.
* SCM checkpoint download is retried without backoff.
* Test infrastructure bugs in MiniOzoneHAClusterImpl.
* Missing acceptance coverage and documentation.
Each item is a sub-task.
> Support and test OM and SCM HA with more than 3 nodes
> -----------------------------------------------------
>
> Key: HDDS-17122
> URL: https://issues.apache.org/jira/browse/HDDS-17122
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM HA, SCM HA
> Reporter: Siyao Meng
> Priority: Major
>
> OM HA and SCM HA have no hard limit on the number of nodes:
> {{ozone.om.nodes.<serviceId>}} / {{ozone.scm.nodes.<serviceId>}} accept any
> number of node IDs, and every node becomes a Ratis peer. In practice only
> 3-node rings are tested and documented (SCM-HA.md: "For reliable HA support
> choose 3 independent nodes"). Operators ask for 5 nodes so a ring keeps HA
> while one node is down for maintenance (see HDDS-3099).
> This umbrella tracks making 5+ node OM and SCM rings a tested, documented
> configuration.
> h3. Ratis background
> * *Static rings:* Ratis has supported Raft groups of any size from the start.
> Vote and commit majorities are computed for N peers ({{num > size()/2}} and
> {{indices[(n-1)/2]}}, unchanged since Ratis 0.2.0), and Ratis' core test
> suites (RaftBasicTests, LeaderElectionTests) have run 5-server clusters since
> 2016. I found no RATIS bug specific to a group that is configured with more
> than 3 peers from the start.
> * *Growing a ring at runtime* ({{ozone om --bootstrap}}, {{ozone scm
> --bootstrap}}, OM/SCM decommission) is a Raft membership change:
> OzoneManagerRatisServer#addOMToRatisRing / removeOMFromRatisRing and
> SCMRatisServerImpl#addSCM / removeSCM call {{setConfiguration}}. Ratis had
> bugs there, several of which bite exactly when 2 peers fail, which only a 5+
> node ring is meant to survive:
> ** RATIS-97 (fixed in 0.2.0): after setConfiguration adds servers, the old
> leader may keep running without a majority.
> ** RATIS-1824 (2.5.0 / 3.0.0): membership change fails when a listener is
> present (relevant to Listener OMs).
> ** RATIS-1912 (3.0.0): infinite election after replacing 2 of 3 peers.
> ** RATIS-2274 (3.2.0): a newly added peer may keep an outdated configuration;
> if the leader and one more peer then fail, elections deadlock.
> ** RATIS-2283 (3.2.0): reconfiguration can block forever after a log appender
> thread restart.
> * *Still open:* RATIS-2657, the leader re-initiates InstallSnapshot against a
> not-ready follower with no backoff (measured on a 5-node group). Related to
> HDDS-17127.
> * *Ratis version per Ozone release:* 1.0.0 -> 1.0.0, 1.1.0 -> 2.0.0, 1.2.x ->
> 2.2.0, 1.3.0 -> 2.4.0, 1.4.0 -> 3.0.1, 1.4.1 -> 3.1.1, 2.0.0 -> 3.1.2, 2.1.0
> -> 3.2.0, 2.2.0 -> 3.2.1, master -> 3.3.1.
> So, as far as known Ratis bugs go: a ring configured with 5+ nodes from the
> start needs no particular Ratis release, while growing or shrinking a ring
> past 3 nodes by bootstrap/decommission needs Ozone 2.1.0+ (Ratis 3.2.0+). The
> remaining known problems are on the Ozone side (the sub-tasks here) plus
> RATIS-2657.
> h3. Findings so far
> New MiniOzoneCluster integration tests with 5 OMs and 5 SCMs (HDDS-17123)
> show that the core Ratis behaviour works: a single Raft group of 5,
> replication to all nodes, leader transfer, surviving 2 failures including the
> leader, no leader with 3 of 5 down, catch-up through checkpoint install,
> bootstrap 3 -> 5 and decommission 5 -> 3, secure 5-SCM failover.
> The problems are around it:
> * *Blocker:* HDDS-16924. An SCM that has installed a checkpoint can no longer
> serve checkpoints once it is leader, so lagging followers never catch up.
> With 3 SCMs this is rare (the primordial SCM usually stays leader); with 5
> SCMs and a few failovers the 5-node test hits it reliably.
> * Leadership transfer takes 4-13 s and the OM admin RPC is not idempotent
> under client retry.
> * OM clients cannot reach a leader that is not in their configured OM list.
> * SCM checkpoint download is retried without backoff.
> * Test infrastructure bugs in MiniOzoneHAClusterImpl.
> * Missing acceptance coverage and documentation.
> Each item is a sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
