[
https://issues.apache.org/jira/browse/HDDS-17266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17266:
------------------------------
Attachment: TestBugMC3LostHeartbeatReplyDropsCommand.java
> Reconcile container command is not resent when it is lost before the datanode
> runs it (lost heartbeat reply, SCM leader change, datanode restart)
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17266
> URL: https://issues.apache.org/jira/browse/HDDS-17266
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: TestBugMC3LostHeartbeatReplyDropsCommand.java
>
>
> h3. Mechanism
> {{CommandQueue.getCommand}} removes the commands queued for a datanode when
> {{SCMNodeManager.processHeartbeat}} builds the heartbeat response, and
> {{SCMDatanodeProtocolServer.sendHeartbeat}} returns them in the reply.
> Nothing acknowledges the reply and SCM keeps no copy. If the reply is lost
> after SCM processed the heartbeat, the datanode gets an {{IOException}}, does
> not resend that heartbeat, and the commands exist nowhere. Hadoop RPC does
> not resend the call either.
> Delivery is therefore at most once, and the other command types are built for
> that: their issuers repair a loss on their own. From reading the code, not
> tested:
> * DeleteBlocks: the SENT timeout in
> {{SCMDeletedBlockTransactionStatusManager}}.
> * Replicate, delete container, EC reconstruct: ReplicationManager pending op
> expiry and re-evaluation.
> * Close container: sent again while the container is CLOSING.
> * Pipelines: {{BackgroundPipelineScrubber}} and {{PipelineReportHandler}}.
> * Operational state, finalize and reregister: regenerated from every
> heartbeat. Refresh volume usage: sent on every balancer iteration.
> {{ReconcileContainerCommand}} is the exception.
> {{SCMClientProtocolServer.reconcileContainer}} fires the event,
> {{ReconcileContainerEventHandler}} queues one command per replica, and
> nothing tracks or resends them. This is unchanged on master at e0de543cbe9.
> h3. Trigger
> An admin runs {{ozone admin container reconcile}}, and the command for one
> replica is lost before that datanode runs it:
> * the heartbeat reply carrying it is lost after SCM built it: the connection
> is reset, or SCM answers after the heartbeat RPC timeout
> ({{ozone.scm.heartbeat.rpc-timeout}}, default 5s).
> * the SCM leader changes first. The per datanode {{CommandQueue}} is in the
> old leader's memory only, so the new leader has no copy; if the old leader
> hands the command out on a later heartbeat, it carries the old term and the
> datanode drops it once it knows the new term. A reconcile that already
> reached the datanode runs through {{ReplicationSupervisor}}
> ({{ReconcileContainerTask}} takes the command's term), which skips queued
> tasks whose term is lower than the current leader term ("Ignoring ... since
> SCM leader has new term"). With a replication backlog on the datanode, a
> failover long after the admin command still discards it.
> * the datanode restarts before it runs the command, or its command queue is
> full.
> For every other command type the new leader's ReplicationManager or report
> handlers send the command again after a failover.
> h3. Impact
> Reconciliation runs per replica: each replica repairs itself from its peers.
> If the replica that lost its command is one that needed repair, it stays
> unrepaired. The admin command reports that reconciliation was triggered and
> points to {{ozone admin container reconcile --status}}, which then keeps
> showing that replica's checksum differing from its peers; re-running the
> command repairs it. If the replica that lost its command was a healthy one,
> the others still reconcile against it and the result converges. No data is
> lost or changed. Not exercised: the full admin command on a cluster, the
> failover routes (code reading only), and the datanode not reconciling
> afterwards (it learns commands only from heartbeat replies).
> h3. Reproduction
> [^TestBugMC3LostHeartbeatReplyDropsCommand.java] ({{hadoop-hdds/server-scm}},
> target path and Maven command in the class comment) passes on unmodified
> source when the bug is present. It starts a real SCM with
> {{HddsTestUtils.getScm}} and a datanode client built by the real
> {{SCMConnectionManager}}, connected through a TCP relay in the test. A
> reconcile command is queued with the {{DATANODE_COMMAND}} event that
> {{ReconcileContainerEventHandler}} fires, and the relay loses the reply to
> the next heartbeat, by resetting the connection or by swallowing it until the
> client times out. In both cases SCM holds no copy afterwards and five later
> heartbeats carry no command. The datanode RPC server is started explicitly
> because the HA stub of the test has no Ratis log whose catch up starts it.
> The failover routes are not reproduced.
> h3. Suggested fix
> The fix belongs with the issuer, as for the other command types. The
> container reconciliation design doc already plans this for Phase II
> (HDDS-12987): "Automate container reconciliation requests as part of SCM's
> replication manager". If ReplicationManager triggers reconciliation on a
> checksum mismatch, a lost command gets the same repair as every other command
> type. Any issuer side tracking has to survive an SCM failover, so it must be
> re-derivable on the new leader from replica state (as ReplicationManager
> state is) or persisted; tracking kept only in the leader's memory would not
> cover the failover routes. There is no completion signal to stop on today
> (checksums that still differ can also mean that reconciliation ran and could
> not repair), so this needs a decision on what counts as done and how often to
> retry. Acknowledging heartbeat replies and redelivering in SCM would be a
> protocol change for every command type and would make non-idempotent commands
> at least once. Documenting that a reconcile may need to be re-run is the
> minimal alternative.
> Found by TLA+ model checking and code review of the datanode registration and
> heartbeat paths under HDDS-15926, on commit
> b10a7d427bdf83807a7e48a1be507d3b02d5a503. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-10372 (added the reconcile command), HDDS-13086 (blocks
> duplicate reconcile requests on the datanode), HDDS-10714 (restrict
> reconciliation requests by datanode status), HDDS-8882 (status tracking for
> DeleteBlocks) and HDDS-17064 (a full container report lost on a failed
> heartbeat, the other direction). Only the reproduction is attached, there is
> no fix patch. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
