[
https://issues.apache.org/jira/browse/HDDS-16804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16804:
------------------------------
Attachment: TestBugMC5NewLeaderCommandLost.java
> ReplicationSupervisor loses a command of the new SCM leader when an equal
> task of the previous leader is still queued
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16804
> URL: https://issues.apache.org/jira/browse/HDDS-16804
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-5-run-new-scm-leader-command-when-queued-duplicate-is-dropped.patch,
> TestBugMC5NewLeaderCommandLost.java
>
>
> h3. Mechanism
> {{ReplicationSupervisor.addToQueue}} drops a task when an equal one is
> already in {{inFlight}}. Task equality does not include the SCM term:
> {{ReconcileContainerTask}} compares the container ID (HDDS-13086),
> {{ECReconstructionCoordinatorTask}} the container ID, {{ReplicationTask}} the
> container ID and the target. When a queued task gets a thread,
> {{TaskRunner.run}} drops it if its term is older than the datanode's current
> SCM term ("Ignoring ... since SCM leader has new term", HDDS-7620). The two
> rules do not compose: a command of the new leader that arrives while the
> equal task of the old leader is still queued is dropped as a duplicate, and
> the queued task is then dropped for its term. Neither runs. The datanode term
> is already advanced in {{StateContext.addCommand}} when the new command
> reaches the supervisor.
> h3. Trigger
> # All threads of the supervisor are busy
> ({{hdds.datanode.replication.streams.limit}}, 10 by default, shared by the
> three task types unless per volume pools are enabled) and a task sent by the
> SCM leader of term N waits in its queue.
> # SCM leadership changes.
> # The new leader sends an equal command (same container for reconcile and EC
> reconstruction, same container and target for replication) before the queued
> task gets a thread.
> # The queued task gets a thread and is dropped for its term.
> For reconcile, step 3 means the request is issued again through the new
> leader ({{ozone admin container reconcile}}), because SCM sends reconcile
> commands only on request. For replication and EC reconstruction, step 3 means
> the old task is still queued when the new leader's ReplicationManager starts
> ({{hdds.scm.wait.time.after.safemode.exit}}, default 5m, after it becomes
> ready as leader) and the new leader chooses this datanode again with the same
> source and target or as coordinator of the same container (read from the
> code, not run).
> h3. Impact
> * Reconcile: the container is not reconciled on that datanode although both
> requests were accepted. SCM does not send the command again, so it stays that
> way until somebody repeats the request. The only trace is the INFO line on
> the datanode.
> * Replication and EC reconstruction: the new leader's command is lost on the
> datanode in the same way (run). The new leader's pending operation expires
> after {{hdds.scm.replication.event.timeout}} (default 12m) and
> ReplicationManager sends the command again on a later pass
> ({{hdds.scm.replication.thread.interval}}, default 300s), so the effect is a
> delayed repair (read from the code, not run).
> * No data is affected. A task that is already running when the new command
> arrives is not affected.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC5NewLeaderCommandLost.java] ({{hadoop-hdds/container-service}})
> sends 11 commands of term 1 and then 11 commands of term 2 for the same
> containers through {{StateContext.addCommand}},
> {{StateContext.getNextCommand}} and the real command handler into a
> {{ReplicationSupervisor}} with default settings. Only the component that does
> the work (container controller, push replicator, EC reconstruction
> coordinator) is a stand in that counts its calls and keeps running tasks open
> until released. For each of the three command types the 11th container is
> processed 0 times after two commands, and the supervisor logs "Ignoring
> QUEUED ... since SCM leader has new term (1 < 2)". A control with duplicates
> in the same term processes every container once. A passing test means the
> defect is present. Not run on a cluster with a real SCM failover.
> h3. Patch
> [^MC-5-run-new-scm-leader-command-when-queued-duplicate-is-dropped.patch],
> against 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master
> at 8ca44914771.
> {{ReplicationSupervisor}} keeps, for each task in flight, the equal command
> it received last. When a task is dropped without running and a command of a
> newer term was received for it meanwhile, that command is queued. Tasks of an
> obsolete term are still dropped and a duplicate of a running task is still
> dropped, so there is at most one of any set of equal tasks, as before. Not
> changed by this patch: a later command of the same term that is deduplicated
> against a queued task which then passes its deadline is still dropped
> (replication and EC reconstruction only, reconcile commands carry no
> deadline). Adding the term to the task equality is not an alternative: with
> the term in {{ReconcileContainerCommand.equals}} the supervisor accepts the
> new leader's command next to a reconciliation of the same container that is
> still running (run), which is what HDDS-13086 removed.
> Covered by two tests in the existing {{TestReplicationSupervisor}}.
> {{commandOfNewLeaderRunsWhenQueuedTaskOfOldLeaderIsDropped}} fails without
> the change because the new leader's command never runs.
> {{commandOfNewLeaderIsDeduplicatedAgainstRunningTask}} passes with and
> without it and pins that a running task gets no second run. With the patch
> {{TestReplicationSupervisor}} (56 tests) and the command handler,
> {{StateContext}}, {{DatanodeStateMachine}} and replication unit suites (130
> tests in total) pass, checkstyle is clean. With the patch the reproduction
> fails for all three command types because every container is processed.
> {{TestContainerReplication}} (16 tests on a mini cluster) passes with the
> patch.
> Found by TLA+ model checking and code review of the container reconciliation
> paths in the datanode and SCM under HDDS-15926, on commit
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
