Siyao Meng created HDDS-16804:
---------------------------------
Summary: ReplicationSupervisor loses a command of the new SCM
leader when an equal task of the previous leader is still queued
Key: HDDS-16804
URL: https://issues.apache.org/jira/browse/HDDS-16804
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-5-run-new-scm-leader-command-when-queued-duplicate-is-dropped.patch,
TestBugMC5NewLeaderCommandLost.java
h3. Mechanism
{{ReplicationSupervisor.addToQueue}} drops a task when an equal one is already
in {{inFlight}}. Task equality does not include the SCM term:
{{ReconcileContainerTask}} compares the container ID (HDDS-13086),
{{ECReconstructionCoordinatorTask}} the container ID, {{ReplicationTask}} the
container ID and the target. When a queued task gets a thread,
{{TaskRunner.run}} drops it if its term is older than the datanode's current
SCM term ("Ignoring ... since SCM leader has new term", HDDS-7620). The two
rules do not compose: a command of the new leader that arrives while the equal
task of the old leader is still queued is dropped as a duplicate, and the
queued task is then dropped for its term. Neither runs. The datanode term is
already advanced in {{StateContext.addCommand}} when the new command reaches
the supervisor.
h3. Trigger
# All threads of the supervisor are busy
({{hdds.datanode.replication.streams.limit}}, 10 by default, shared by the
three task types unless per volume pools are enabled) and a task sent by the
SCM leader of term N waits in its queue.
# SCM leadership changes.
# The new leader sends an equal command (same container for reconcile and EC
reconstruction, same container and target for replication) before the queued
task gets a thread.
# The queued task gets a thread and is dropped for its term.
For reconcile, step 3 means the request is issued again through the new leader
({{ozone admin container reconcile}}), because SCM sends reconcile commands
only on request. For replication and EC reconstruction, step 3 means the old
task is still queued when the new leader's ReplicationManager starts
({{hdds.scm.wait.time.after.safemode.exit}}, default 5m, after it becomes ready
as leader) and the new leader chooses this datanode again with the same source
and target or as coordinator of the same container (read from the code, not
run).
h3. Impact
* Reconcile: the container is not reconciled on that datanode although both
requests were accepted. SCM does not send the command again, so it stays that
way until somebody repeats the request. The only trace is the INFO line on the
datanode.
* Replication and EC reconstruction: the new leader's command is lost on the
datanode in the same way (run). The new leader's pending operation expires
after {{hdds.scm.replication.event.timeout}} (default 12m) and
ReplicationManager sends the command again on a later pass
({{hdds.scm.replication.thread.interval}}, default 300s), so the effect is a
delayed repair (read from the code, not run).
* No data is affected. A task that is already running when the new command
arrives is not affected.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugMC5NewLeaderCommandLost.java]
({{hadoop-hdds/container-service}}) sends 11 commands of term 1 and then 11
commands of term 2 for the same containers through {{StateContext.addCommand}},
{{StateContext.getNextCommand}} and the real command handler into a
{{ReplicationSupervisor}} with default settings. Only the component that does
the work (container controller, push replicator, EC reconstruction coordinator)
is a stand in that counts its calls and keeps running tasks open until
released. For each of the three command types the 11th container is processed 0
times after two commands, and the supervisor logs "Ignoring QUEUED ... since
SCM leader has new term (1 < 2)". A control with duplicates in the same term
processes every container once. A passing test means the defect is present. Not
run on a cluster with a real SCM failover.
h3. Patch
[^MC-5-run-new-scm-leader-command-when-queued-duplicate-is-dropped.patch],
against 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
8ca44914771.
{{ReplicationSupervisor}} keeps, for each task in flight, the equal command it
received last. When a task is dropped without running and a command of a newer
term was received for it meanwhile, that command is queued. Tasks of an
obsolete term are still dropped and a duplicate of a running task is still
dropped, so there is at most one of any set of equal tasks, as before. Not
changed by this patch: a later command of the same term that is deduplicated
against a queued task which then passes its deadline is still dropped
(replication and EC reconstruction only, reconcile commands carry no deadline).
Adding the term to the task equality is not an alternative: with the term in
{{ReconcileContainerCommand.equals}} the supervisor accepts the new leader's
command next to a reconciliation of the same container that is still running
(run), which is what HDDS-13086 removed.
Covered by two tests in the existing {{TestReplicationSupervisor}}.
{{commandOfNewLeaderRunsWhenQueuedTaskOfOldLeaderIsDropped}} fails without the
change because the new leader's command never runs.
{{commandOfNewLeaderIsDeduplicatedAgainstRunningTask}} passes with and without
it and pins that a running task gets no second run. With the patch
{{TestReplicationSupervisor}} (56 tests) and the command handler,
{{StateContext}}, {{DatanodeStateMachine}} and replication unit suites (130
tests in total) pass, checkstyle is clean. With the patch the reproduction
fails for all three command types because every container is processed.
{{TestContainerReplication}} (16 tests on a mini cluster) passes with the patch.
Found by TLA+ model checking and code review of the container reconciliation
paths in the datanode and SCM under HDDS-15926, on commit
1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
