Siyao Meng created HDDS-16905:
---------------------------------
Summary: SCM keeps deleted pipelines in its datanode to pipeline
map after a Ratis snapshot install, so decommission and maintenance stall while
it is leader
Key: HDDS-16905
URL: https://issues.apache.org/jira/browse/HDDS-16905
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-remove-stale-node-pipelines-on-scm-reinitialize.patch,
TestBugMC6StaleNode2PipelineAfterSnapshotInstall.java
h3. Mechanism
When a follower SCM installs a snapshot from the leader,
{{SCMHAManagerImpl.startServices}} calls
{{PipelineStateManagerImpl.reinitialize}}. It replaces the {{PipelineStateMap}}
with an empty one and {{initialize}} adds every pipeline of the new pipeline
table to the state map and to the NodeManager ({{nodeManager.addPipeline}}).
The pipelines of the old map are never removed from the NodeManager. The
datanode to pipeline map ({{Node2PipelineMap}}) is only updated by
{{PipelineStateManagerImpl}}, and its only removal path,
{{PipelineStateManagerImpl.removePipeline}}, needs the pipeline to be in the
state map. So a pipeline that was closed and deleted while the follower was
behind stays listed for its datanodes until that SCM process restarts.
{{DatanodeAdminMonitorImpl.checkPipelinesClosedOnNode}}, the first gate for
decommission and for entering maintenance, waits until
{{nodeManager.getPipelines(dn)}} is empty, and decommission has no expiry.
{{StartDatanodeAdminHandler}} cannot close the stale pipeline either, because
the pipeline manager does not know it.
h3. Trigger
# {{ozone.scm.ha.ratis.log.purge.enabled=true}} (default false). Without log
purge the leader keeps its whole log and sends a lagging follower entries
instead of a snapshot (read from code).
# A follower SCM knows pipeline P. It is stopped (or falls behind) while the
leader closes and deletes P and purges its Raft log past the follower's last
index.
# The follower catches up by installing a snapshot
({{SCMStateMachine.reinitialize}}). Its NodeManager still lists P for P's
datanodes.
# That SCM becomes leader and an operator decommissions a datanode of P.
Only the restart variant of step 2 was run. A live follower that is sent a
snapshot takes the same path (argued from code).
h3. Impact
* The decommission of every datanode that was in such a pipeline stays in
DECOMMISSIONING without an error while that SCM is leader. The monitor logs
"Waiting for pipelines to close ... There are 1 pipelines" on every run.
Restarting that SCM clears the stale entries (read from code, not run). Moving
leadership to another SCM should also let the decommission continue, since the
other SCMs have a correct map (argued, not run).
* A datanode entering maintenance waits on the stale entry the same way, until
its maintenance end time passes, or forever if no end time was given (argued,
not run).
* {{NodeManager.getPipelinesCount}} and the pipeline count in datanode usage
info include the stale entries (argued, not run). Pipeline placement is not
affected, because it ignores pipeline IDs the state manager does not know.
h3. Reproduction
PASS, unmodified source.
[^TestBugMC6StaleNode2PipelineAfterSnapshotInstall.java]
({{hadoop-ozone/integration-test}}) starts 3 SCMs, 1 OM and 3 datanodes with
log purge enabled and the small purge gap and snapshot threshold that
{{TestSCMInstallSnapshotWithHA}} uses. It stops a follower, closes an open
RATIS/THREE pipeline with the admin close pipeline RPC and lets the leader's
scrubber delete it, advances the Raft log by 200 entries with in process
{{ContainerManager.allocateContainer}} calls on the leader, restarts the
follower, which installs a snapshot, transfers leadership to it and
decommissions one datanode of the deleted pipeline. Output on the pin: after
the snapshot install the restarted SCM does not know the pipeline but lists it
for the datanode, and after 8 monitor runs the datanode is still
DECOMMISSIONING with exactly that pipeline left. A passing test means the
defect is present. With the patch it fails at the first check after the
snapshot install, because the datanode then lists only its live pipelines.
h3. Patch
[^MC-6-remove-stale-node-pipelines-on-scm-reinitialize.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
b9b33851b20.
{{PipelineStateManagerImpl.reinitialize}} removes the pipelines of the old
state map from the NodeManager before it rebuilds the map; {{initialize}} adds
the pipelines of the new table back as before. The NodeManager then lists
exactly the pipelines of the new state map. Recon's SCM DB snapshot sync
({{ReconStorageContainerManagerFacade.initializeNewRdbStore}}) calls the same
{{reinitialize}} and gets the same fix (read from code, not run). Covered by
{{testReinitializeRemovesPipelinesMissingFromNewStore}} in the existing
{{TestPipelineStateManagerImpl}} (two pipelines on one datanode, reinitialize
with a table that keeps one), which fails without the change. With it the
pipeline and node unit suites touched by this path (132 tests) and
{{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} pass,
checkstyle is clean. Entries left on an SCM that already installed a snapshot
are only cleared by a restart.
Found by TLA+ model checking and code review of the SCM pipeline lifecycle
paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
