[
https://issues.apache.org/jira/browse/HDDS-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17218:
------------------------------
Attachment: TestBugMC6StalePeerMaintenance.java
> Datanode can enter maintenance below the replica minimum because a replica on
> a STALE datanode still counts as an available copy
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17218
> URL: https://issues.apache.org/jira/browse/HDDS-17218
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-6-wait-for-stale-replica-node-before-admin-completion.patch,
> TestBugMC6StalePeerMaintenance.java
>
>
> h3. Mechanism
> {{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} relies on
> {{ReplicationManager.checkContainerStatus}} to decide whether enough copies
> remain elsewhere. {{RatisContainerReplicaCount.countReplicas}} classifies
> each replica only by the persisted operational state of its datanode, so a
> replica on an IN_SERVICE datanode that SCM has already marked STALE is
> counted as a healthy copy until the datanode is DEAD and {{DeadNodeHandler}}
> removes its replicas. The monitor does not look at the health of the other
> replica holders either; after the check it only re-reads the health of the
> node it is processing.
> Replication Manager counts a replica on a STALE datanode until the datanode
> is DEAD. The over replication handlers ({{RatisOverReplicationHandler}},
> {{ECOverReplicationHandler}}, {{QuasiClosedStuckOverReplicationHandler}})
> however leave STALE replicas out before they choose a replica to delete,
> because the STALE node may be about to die. Taking a node offline for
> maintenance or decommission removes a reachable copy in the same way, but the
> admin check has no such guard.
> h3. Trigger
> # A closed RATIS/THREE container has replicas on A, B and C. D is another
> datanode.
> # B stops. SCM marks it STALE (after {{ozone.scm.stale.node.interval}}, 5m)
> but not yet DEAD ({{ozone.scm.dead.node.interval}}, 10m).
> # Within that window an admin puts A into maintenance. The admission check
> passes because A, C and D are IN_SERVICE and HEALTHY, which leaves two after
> A, meeting the minimum of 2.
> # The check counts B and C as two healthy copies, which meets
> {{hdds.scm.replication.maintenance.replica.minimum}} (2). A becomes
> IN_MAINTENANCE and no copy is made on D.
> Decommission uses the same check, so with a STALE B it completes with only
> two reachable in service copies (argued from the code and covered by the unit
> test, not run on a cluster).
> h3. Impact
> A is in maintenance with one reachable copy (C), one below the configured
> minimum. If C fails before B is declared DEAD and Replication Manager has
> made a new copy on D, the data is unreadable until A leaves maintenance or B
> comes back. No data is lost, A keeps its copy. The window is bounded by the
> STALE period; once B is DEAD, Replication Manager makes the second copy on D
> from C (or from A while it is still running). The same check is in
> ozone-2.1.2, ozone-2.2.0 and ozone-2.2.1 (read from the tagged source, not
> run on a release).
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC6StalePeerMaintenance.java]
> ({{hadoop-ozone/integration-test}}) runs a mini cluster with four datanodes
> and the settings of {{TestDecommissionAndMaintenance}}, except that the dead
> interval is 60s so that the STALE window is wide enough for the 1s monitor
> interval. It writes one RATIS/THREE key, closes its container, stops B, waits
> for STALE, calls {{startMaintenanceNodes}} for A without force through
> {{ContainerOperationClient}}, then stops A and C and reads the key. Run on
> the unmodified source: A reached IN_MAINTENANCE 7.8s after B became STALE,
> with B still STALE and one IN_SERVICE HEALTHY replica, and the read failed
> with GetBlock errors on all replicas. A passing test means the defect is
> present.
> h3. Patch
> [^MC-6-wait-for-stale-replica-node-before-admin-completion.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 20d95ed37fe.
> The monitor treats a container as not yet replicated while any of its
> replicas is on an IN_SERVICE datanode that is not HEALTHY (STALE, or DEAD
> before {{DeadNodeHandler}} has removed its replicas). The node then waits
> until that datanode is HEALTHY again, or DEAD with its replicas removed,
> after which the normal count applies and Replication Manager makes the
> missing copy. Each wait ends within the dead interval: the node completes at
> the first monitor run in which no IN_SERVICE datanode holding one of its
> replicas is STALE or DEAD. Replicas on datanodes in maintenance or
> decommission are ignored: they are not counted as available copies, and a
> DEAD node in maintenance keeps its replicas (HDDS-2607), so waiting for it
> would not end. The check is in the monitor, not in the replica counts, so
> Replication Manager's own handling of STALE replicas does not change. It
> applies to EC replicas as well (not tested with an EC container). With a
> STALE replica holder the node now waits even if the container would have
> enough copies without that replica, for example with surplus replicas, or
> when the replica on the STALE datanode is one that is not counted anyway,
> such as an UNHEALTHY one. While it waits, the container is counted as under
> replicated in the decommission metrics and listed as under replicated by
> {{ozone admin datanode status decommission}}, although Replication Manager
> does not report it so. The monitor log names the datanode it waits for
> ("Container ... waits for its replica on ..., which is IN_SERVICE but not
> HEALTHY") in place of the usual under replicated message, within the existing
> limit on logged containers.
> Covered in the existing {{TestDatanodeAdminMonitor}} by
> {{testAdminWaitsWhileReplicaIsOnStaleNode}} (decommission and maintenance,
> the other replica holders registered as IN_SERVICE and HEALTHY),
> {{testDecommissionWaitsForReplicaOnDeadInServiceNodeToBeRemoved}} (a replica
> on a DEAD IN_SERVICE datanode holds the node until it is removed) and the
> negative case {{testMaintenanceNotBlockedByReplicaOnDeadMaintenanceNode}}.
> Without the change the first two fail with "expected: <ENTERING_MAINTENANCE>
> but was: <IN_MAINTENANCE>" and "expected: <DECOMMISSIONING> but was:
> <DECOMMISSIONED>". With it {{TestDatanodeAdminMonitor}},
> {{TestNodeDecommissionMetrics}} and {{TestNodeDecommissionManager}} pass, as
> does the mini cluster suite {{TestDecommissionAndMaintenance}} (9 tests),
> checkstyle is clean, and in the reproduction A stays ENTERING_MAINTENANCE
> until B is DEAD and enters maintenance with two reachable copies, 61s after B
> became STALE.
> Found by TLA+ model checking and code review of the SCM datanode decommission
> and maintenance admin paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
