Siyao Meng created HDDS-17224:
---------------------------------
Summary: Decommission or maintenance completes on an empty
container set after SCM restart or datanode re-registration, before the full
container report
Key: HDDS-17224
URL: https://issues.apache.org/jira/browse/HDDS-17224
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-wait-for-full-container-report-before-admin-completion.patch,
TestBugMC2EmptyContainerSetPassesGate.java
h3. Mechanism
{{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} checks only the
containers in {{NodeManager.getContainers}} for the node and passes when that
set is empty. The set is filled only by container reports, and it starts empty
whenever the node gets a new {{DatanodeEntry}}:
* {{NodeStateMap.addNode}}, when a datanode registers with an SCM that has
restarted;
* {{NodeStateMap.updateNode}}, when a datanode registers again with a new
address, ports or software version (the container set loss described in
HDDS-16632).
{{SCMDatanodeProtocolServer.register}} fires the full container report as an
event only after {{SCMNodeManager.register}} has returned, and the report is
processed on the container report executor. The monitor runs on its own
executor and does not check safe mode or whether a report has been processed.
After an SCM restart, the admin workflow is resumed for a node that registers
with a persisted DECOMMISSIONING or ENTERING_MAINTENANCE state, by the NEW_NODE
handler ({{NodeDecommissionManager.continueAdminForNode}}) or, if the node
registered before this SCM became leader, by
{{NodeDecommissionManager.onBecomeLeader}}. The other gate conditions do not
hold the node: its pipelines were closed when the workflow started, and its
persisted state matches. If a monitor run lands before the report is processed,
the node completes the workflow. The report that arrives later does not reopen
it.
h3. Trigger
Re-registration:
# A datanode is being decommissioned and the monitor is waiting for an under
replicated container on it.
# The datanode restarts and registers again with a new software version
(rolling upgrade) or a changed port set (seen on a plain datanode restart in a
mini cluster in HDDS-17190). Its container set in SCM is now empty.
# A monitor run happens before the full container report from the registration
has been processed.
# The node becomes DECOMMISSIONED.
SCM restart:
# A datanode is DECOMMISSIONING or ENTERING_MAINTENANCE and holds an under
replicated container. SCM restarts.
# The datanode registers with its persisted state, and the leader starts
tracking it again.
# A monitor run happens before its full container report has been processed
(after an SCM restart many datanodes register and their reports queue up).
# The node becomes DECOMMISSIONED or IN_MAINTENANCE.
h3. Impact
The node is reported DECOMMISSIONED (safe to remove) or IN_MAINTENANCE while
containers on it are still under replicated, and it is no longer tracked. This
can hold for every container on the node at once. Once the report is processed,
Replication Manager sees the containers as under replicated and copies them
from the node while it is still up, so redundancy is lost only if the operator
removes or stops the node first, and data only with a further failure. The
window is from registration until the report is processed; how long it is on a
real cluster was not measured.
h3. Reproduction
PASS on unmodified source, deterministic.
[^TestBugMC2EmptyContainerSetPassesGate.java] ({{hadoop-hdds/server-scm}}) uses
the real {{SCMNodeManager}} for registration, heartbeats and operational state,
the real {{ContainerReportHandler}} for the full container report (with a stub
{{ContainerManager}} that knows one CLOSED RATIS/THREE container) and the real
{{DatanodeAdminMonitorImpl}}. Replication Manager is mocked as in
{{TestDatanodeAdminMonitor}}: the container has the replica on the node and one
other copy. The only thing the test controls is whether the monitor runs before
or after the report is processed.
* Re-registration: the node is DECOMMISSIONING and held by the container,
registers again with version 2.0.0 instead of 1.0.0 (container set empty), the
monitor runs, then the report is processed. The node is DECOMMISSIONED, and
stays so with the container back in its set.
* SCM restart, persisted DECOMMISSIONING and ENTERING_MAINTENANCE: with the
report processed first the node stays in its admin state; with the monitor
first it becomes DECOMMISSIONED or IN_MAINTENANCE.
A passing test means the defect is present.
h3. Patch
[^MC-2-wait-for-full-container-report-before-admin-completion.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
20d95ed37fe.
{{DatanodeInfo}} gets a flag that {{ContainerReportHandler}} sets once a full
container report has been applied to the node's container set, and
{{checkContainersReplicatedOnNode}} does not pass while the node's current
{{DatanodeInfo}} lacks it. The flag needs no reset: SCM creates a new
{{DatanodeInfo}} in exactly the two cases where the container set starts empty.
Incremental reports do not set it. The change is on the gate side, so it is
independent of how HDDS-16632 is fixed; carrying the container set over in
{{NodeStateMap.updateNode}} alone would not cover the SCM restart case. This
patch is meant to cover the admin monitor symptom that HDDS-16632 describes, so
a fix for HDDS-16632 does not need to change the monitor. The flag is read
after the container set, so a registration between the two reads counts as a
node still waiting for its report. Behaviour change: after an SCM restart or a
re-registration, a tracked node waits for its first full container report to be
processed before it can complete. Normally that is the report sent with the
registration; if the handler fails on that one, the node waits for the next
full report ({{hdds.container.report.interval}}, 60 minutes by default) or
until the datanode registers again, which sends a new report.
Covered by {{testAdminWaitsForContainerReportAfterRegistration}} (decommission
and maintenance) in the existing {{TestDatanodeAdminMonitor}}, and
{{testFullReportMarksContainerReportProcessed}} in the existing
{{TestIncrementalContainerReportHandler}} (an incremental report does not set
the flag, a full report does, after re-registering with a new version the node
starts without it). Without the gate change the first fails with "expected:
<DECOMMISSIONING> but was: <DECOMMISSIONED>" and "expected:
<ENTERING_MAINTENANCE> but was: <IN_MAINTENANCE>". With it
{{TestDatanodeAdminMonitor}}, {{TestIncrementalContainerReportHandler}},
{{TestContainerReportHandler}}, {{TestSCMNodeManager}},
{{TestNodeDecommissionManager}}, {{TestNodeDecommissionMetrics}} and
{{TestDeadNodeHandler}} pass, the mini cluster suite
{{TestDecommissionAndMaintenance}} (9 tests, including its SCM restart cases)
passes, checkstyle is clean, and the three reproduction tests fail because the
node waits for the report and is then held by the under replicated container.
Found by TLA+ model checking and code review of the SCM datanode decommission
and maintenance admin paths under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-16632 (container set lost on re-registration, mentions this
effect without a reproduction), HDDS-16925 (ContainerManager replicas lost on
SCM snapshot install). The attached patch is a proposal for review. Generated
with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
