[
https://issues.apache.org/jira/browse/HDDS-17620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17620:
------------------------------
Attachment:
MC-1-skip-repeat-delete-of-unhealthy-ec-replica-while-pending.patch
> SCM sends another DELETE for an UNHEALTHY EC replica while one is pending,
> and the queued copy can delete a replica later rebuilt on that datanode
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17620
> URL: https://issues.apache.org/jira/browse/HDDS-17620
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1,
> 2.2.2
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-1-skip-repeat-delete-of-unhealthy-ec-replica-while-pending.patch
>
>
> h3. Mechanism
> {{ClosedWithUnhealthyReplicasHandler.handle}} sends a force DELETE for every
> UNHEALTHY replica of a CLOSED EC container whose index also has a CLOSED
> copy, on every ReplicationManager check, without looking at the container's
> pending ops. {{ECContainerReplicaCount}} ignores pending deletes of UNHEALTHY
> replicas, so the container stays HEALTHY in {{ECReplicationCheckHandler}} and
> reaches this handler again on the next pass. While the first DELETE is still
> queued on the datanode, a second one is sent. SCM keeps a single pending op
> per target and replica index ({{ContainerReplicaPendingOps.addReplica}}
> replaces the earlier one), so the report of the first delete completes the
> only op and the datanode no longer counts as holding the container
> ({{ReplicationManagerUtil.getExcludedAndUsedNodes}}).
> On the datanode, {{DeleteContainerCommandHandler.handleInternal}} checks only
> the command deadline and the SCM term, then
> {{ContainerController.deleteContainer}} deletes the replica with that
> container ID, whatever its replica index. If another replica of the same
> container is placed on that datanode before the second DELETE runs (EC
> reconstruction of another index, or a balancer move, since
> {{MoveManager.move}} also checks only the replicas and pending ops SCM knows
> of), the second DELETE removes it.
> {{ECUnderReplicationHandler.checkAndRemoveUnhealthyReplica}} and
> {{DeletingContainerHandler}} already skip replicas with a delete in flight;
> this handler does not.
> h3. Trigger
> Default settings: monitor interval 300s,
> {{hdds.scm.replication.event.timeout}} 12m, datanode offset 6m, 2 delete
> container threads per datanode.
> # A CLOSED EC container has a CLOSED and an UNHEALTHY replica of index 2; the
> UNHEALTHY one is on datanode V. A check sends DELETE D1 to V.
> # D1 waits in V's delete queue behind other delete work. The next check, 5
> minutes later, sends D2 for the same replica; it reaches V while the replica
> still exists.
> # D1 runs before its datanode deadline (6 minutes after it was sent) and V's
> report completes the pending op.
> # Another index of the same container (index 1) is lost, and reconstruction
> rebuilds it on V while D2 is still queued.
> # D2 runs and deletes the rebuilt index 1 replica.
> This needs a delete backlog of minutes on V, the loss of another index of the
> same container within the same few minutes, and V chosen as the
> reconstruction target, so it is unlikely.
> h3. Impact
> A replica that SCM just placed and recorded as present is deleted, for
> example an index rebuilt by reconstruction or the target copy of a balancer
> move. The container loses redundancy again until ReplicationManager repairs
> it. No data is lost on its own.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC1RepeatedUnhealthyDeleteRemovesRebuiltReplica.java]
> ({{hadoop-hdds/server-scm}}) runs the real ReplicationManager check chain and
> {{ContainerReplicaPendingOps}} with mocked node and container managers, and
> gives V a real {{DeleteContainerCommandHandler}}, {{ContainerController}},
> {{KeyValueHandler}} and {{ContainerSet}} on a temporary volume. Commands are
> handed to V by the test, V's delete thread (one in the test) is held with
> latches, and V's report is applied to the replica map by hand. Reconstruction
> target selection is not run: the test checks that V is neither excluded nor
> used for it, then creates index 1 on V with the CreateContainer (RECOVERING)
> and CloseContainer requests that reconstruction sends. Output:
> {noformat}
> after second check: DELETE commands sent to V = 2, pending DELETE ops = 1
> rebuilt on V: replicaIndex = 1, state = CLOSED
> after the second DELETE (replicaIndex 2) ran on V: container present = false,
> expired = 0
> {noformat}
> A passing test means the defect is present. No mini cluster was run.
> h3. Suggested fix
> Skip the delete in SCM rather than on the datanode. A datanode check of the
> command's replica index would leave a replica of the same index placed later
> exposed, and {{AbstractContainerReportHandler.deleteReplica}} sends deletes
> without a replica index (0) for EC replicas, so such a check would have to
> treat 0 as matching any index.
> The expiry path of the same datanode behaviour (a DELETE resent after its op
> expired, HDDS-12127) is HDDS-16897, whose attached patch covers it. It does
> not cover this case, because here the replica is still present when both
> commands arrive.
> h3. Patch
> [^MC-1-skip-repeat-delete-of-unhealthy-ec-replica-while-pending.patch],
> against e4ae41fc4047c99af983020b4f33bd297c9d0ac5. It also applies to master
> at 3fc2387040f.
> {{ClosedWithUnhealthyReplicasHandler.handle}} does not send a DELETE for an
> UNHEALTHY replica whose datanode already has a pending DELETE for the
> container. The container is still reported {{UNHEALTHY_OVER_REPLICATED}}. A
> pending DELETE that expires is still resent by
> {{ReplicationManager.opCompleted}}, so the replica is retried every 12
> minutes by the existing resend instead of every monitor pass.
> Covered by {{testUnhealthyECReplicaWithPendingDeleteIsNotDeletedAgain}} in
> the existing {{TestReplicationManager}}: a second check sends no DELETE for
> the replica with a pending delete and still deletes another UNHEALTHY replica
> without one. Without the change it fails (two commands sent). With it the
> replication manager, scenario, EC handler, health check and pending ops
> suites pass (213 tests) and checkstyle is clean. With the patch the
> reproduction fails, because the second check sends no DELETE.
> Found by TLA+ model checking and code review of the SCM EC container
> replication and placement repair paths under HDDS-15926, on commit
> e4ae41fc4047c99af983020b4f33bd297c9d0ac5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
