Siyao Meng created HDDS-17218:
---------------------------------
Summary: Datanode can enter maintenance below the replica minimum
because a replica on a STALE datanode still counts as an available copy
Key: HDDS-17218
URL: https://issues.apache.org/jira/browse/HDDS-17218
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-wait-for-stale-replica-node-before-admin-completion.patch,
TestBugMC6StalePeerMaintenance.java
h3. Mechanism
{{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} relies on
{{ReplicationManager.checkContainerStatus}} to decide whether enough copies
remain elsewhere. {{RatisContainerReplicaCount.countReplicas}} classifies each
replica only by the persisted operational state of its datanode, so a replica
on an IN_SERVICE datanode that SCM has already marked STALE is counted as a
healthy copy until the datanode is DEAD and {{DeadNodeHandler}} removes its
replicas. The monitor does not look at the health of the other replica holders
either; after the check it only re-reads the health of the node it is
processing.
Replication Manager counts a replica on a STALE datanode until the datanode is
DEAD. The over replication handlers ({{RatisOverReplicationHandler}},
{{ECOverReplicationHandler}}, {{QuasiClosedStuckOverReplicationHandler}})
however leave STALE replicas out before they choose a replica to delete,
because the STALE node may be about to die. Taking a node offline for
maintenance or decommission removes a reachable copy in the same way, but the
admin check has no such guard.
h3. Trigger
# A closed RATIS/THREE container has replicas on A, B and C. D is another
datanode.
# B stops. SCM marks it STALE (after {{ozone.scm.stale.node.interval}}, 5m) but
not yet DEAD ({{ozone.scm.dead.node.interval}}, 10m).
# Within that window an admin puts A into maintenance. The admission check
passes because A, C and D are IN_SERVICE and HEALTHY, which leaves two after A,
meeting the minimum of 2.
# The check counts B and C as two healthy copies, which meets
{{hdds.scm.replication.maintenance.replica.minimum}} (2). A becomes
IN_MAINTENANCE and no copy is made on D.
Decommission uses the same check, so with a STALE B it completes with only two
reachable in service copies (argued from the code and covered by the unit test,
not run on a cluster).
h3. Impact
A is in maintenance with one reachable copy (C), one below the configured
minimum. If C fails before B is declared DEAD and Replication Manager has made
a new copy on D, the data is unreadable until A leaves maintenance or B comes
back. No data is lost, A keeps its copy. The window is bounded by the STALE
period; once B is DEAD, Replication Manager makes the second copy on D from C
(or from A while it is still running). The same check is in ozone-2.1.2,
ozone-2.2.0 and ozone-2.2.1 (read from the tagged source, not run on a release).
h3. Reproduction
PASS on unmodified source. [^TestBugMC6StalePeerMaintenance.java]
({{hadoop-ozone/integration-test}}) runs a mini cluster with four datanodes and
the settings of {{TestDecommissionAndMaintenance}}, except that the dead
interval is 60s so that the STALE window is wide enough for the 1s monitor
interval. It writes one RATIS/THREE key, closes its container, stops B, waits
for STALE, calls {{startMaintenanceNodes}} for A without force through
{{ContainerOperationClient}}, then stops A and C and reads the key. Run on the
unmodified source: A reached IN_MAINTENANCE 7.8s after B became STALE, with B
still STALE and one IN_SERVICE HEALTHY replica, and the read failed with
GetBlock errors on all replicas. A passing test means the defect is present.
h3. Patch
[^MC-6-wait-for-stale-replica-node-before-admin-completion.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
20d95ed37fe.
The monitor treats a container as not yet replicated while any of its replicas
is on an IN_SERVICE datanode that is not HEALTHY (STALE, or DEAD before
{{DeadNodeHandler}} has removed its replicas). The node then waits until that
datanode is HEALTHY again, or DEAD with its replicas removed, after which the
normal count applies and Replication Manager makes the missing copy. Each wait
ends within the dead interval: the node completes at the first monitor run in
which no IN_SERVICE datanode holding one of its replicas is STALE or DEAD.
Replicas on datanodes in maintenance or decommission are ignored: they are not
counted as available copies, and a DEAD node in maintenance keeps its replicas
(HDDS-2607), so waiting for it would not end. The check is in the monitor, not
in the replica counts, so Replication Manager's own handling of STALE replicas
does not change. It applies to EC replicas as well (not tested with an EC
container). With a STALE replica holder the node now waits even if the
container would have enough copies without that replica, for example with
surplus replicas, or when the replica on the STALE datanode is one that is not
counted anyway, such as an UNHEALTHY one. While it waits, the container is
counted as under replicated in the decommission metrics and listed as under
replicated by {{ozone admin datanode status decommission}}, although
Replication Manager does not report it so. The monitor log names the datanode
it waits for ("Container ... waits for its replica on ..., which is IN_SERVICE
but not HEALTHY") in place of the usual under replicated message, within the
existing limit on logged containers.
Covered in the existing {{TestDatanodeAdminMonitor}} by
{{testAdminWaitsWhileReplicaIsOnStaleNode}} (decommission and maintenance, the
other replica holders registered as IN_SERVICE and HEALTHY),
{{testDecommissionWaitsForReplicaOnDeadInServiceNodeToBeRemoved}} (a replica on
a DEAD IN_SERVICE datanode holds the node until it is removed) and the negative
case {{testMaintenanceNotBlockedByReplicaOnDeadMaintenanceNode}}. Without the
change the first two fail with "expected: <ENTERING_MAINTENANCE> but was:
<IN_MAINTENANCE>" and "expected: <DECOMMISSIONING> but was: <DECOMMISSIONED>".
With it {{TestDatanodeAdminMonitor}}, {{TestNodeDecommissionMetrics}} and
{{TestNodeDecommissionManager}} pass, as does the mini cluster suite
{{TestDecommissionAndMaintenance}} (9 tests), checkstyle is clean, and in the
reproduction A stays ENTERING_MAINTENANCE until B is DEAD and enters
maintenance with two reachable copies, 61s after B became STALE.
Found by TLA+ model checking and code review of the SCM datanode decommission
and maintenance admin paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
