[
https://issues.apache.org/jira/browse/HDDS-16809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16809:
------------------------------
Attachment: TestBugMC6EmptyQuasiClosedContainerDeletion.java
> Empty QUASI_CLOSED container is deleted although no known replica has its
> sequence ID, and the returning data replica is force deleted
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16809
> URL: https://issues.apache.org/jira/browse/HDDS-16809
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-6-keep-empty-quasi-closed-container-without-replica-at-container-bcsid.patch,
> TestBugMC6EmptyQuasiClosedContainerDeletion.java
>
>
> h3. Mechanism
> {{EmptyContainerHandler.handle}} moves a QUASI_CLOSED container to DELETING
> as soon as every replica SCM currently knows is empty
> ({{isContainerEmptyAndQuasiClosed}}). It does not require that any of those
> replicas has reached the container's sequence ID. The CLOSED branch of the
> same method has that check since HDDS-14551 (code comment: "don't update
> container state if replica seqid don't match with container seq id"). The
> QUASI_CLOSED branch was added afterwards by HDDS-14684 without it.
> {{DeletingContainerHandler}} then moves the container to DELETED once no
> replica is known. When a non-empty CLOSED or QUASI_CLOSED replica of a
> DELETING or DELETED RATIS container is reported,
> {{AbstractContainerReportHandler}} answers with a forced
> {{DeleteContainerCommand}}, whatever the replica's BCSID. The code
> (HDDS-14416, extended to QUASI_CLOSED replicas by HDDS-14684) compares the
> replica's BCSID with the container's sequence ID and means to keep a replica
> that is above it. But {{processContainerReplica}} calls
> {{updateContainerStats}} before {{updateContainerState}}, and
> {{updateContainerStats}} raises the container's sequence ID to the replica's
> BCSID for every healthy replica without looking at the container state.
> {{ContainerInfo.updateSequenceId}} excludes DELETING and DELETED only with a
> Java {{assert}}, and the launch scripts do not enable assertions. So when the
> comparison runs, the replica is never above the container. Only OPEN, CLOSING
> and UNHEALTHY replicas move the container back.
> This was discussed before. HDDS-12305 (resolved as duplicate of HDDS-12421)
> asked not to mark a container DELETED on empty replicas at BCSID 0. The
> answer there was that SCM cannot decide this at deletion time, so a replica
> with data that reports later moves the container back and no forced delete is
> used, at the price of orphan containers. HDDS-14416 then introduced the
> forced delete for returning CLOSED replicas at or below the container's
> sequence ID, and HDDS-14684 allowed the deletion of empty QUASI_CLOSED
> containers on the premise that a later replica with the missing blocks "would
> be tracked and replicated again". For a CLOSED or QUASI_CLOSED replica that
> premise does not hold.
> h3. Trigger
> RATIS/THREE container on d1, d2, d3. No operator action, only datanode
> outages:
> # d1 stops after the first chunk write has created the container on it and
> before the first block commit is applied there. The block is committed on d2
> and d3 (BCSID 4). d1's replica stays at BCSID 0 with no block. If the
> container is created while d1 is down, d1 has no replica and nothing is
> deleted.
> # d2 goes down before the container is closed. d3 quasi closes, the container
> becomes QUASI_CLOSED with sequence ID 4.
> # d1 returns, its replica is quasi closed, and d1 restarts once more. A
> datanode marks a non-open replica without blocks as empty only when it loads
> the container, so d1 now reports an empty replica at BCSID 0.
> # d3 is down longer than the dead node interval. {{DeadNodeHandler}} removes
> its replica record, SCM knows only d1's empty replica.
> # The replication manager runs {{EmptyContainerHandler}}: "Deleting empty
> QUASI_CLOSED container #1 (container BCSID=4, max replica BCSID=0)". The
> container goes to DELETING, then DELETED.
> # d3 returns and reports QUASI_CLOSED, BCSID 4, non-empty. SCM sends a forced
> delete and d3 deletes the replica.
> d2 must not come back. If it returns, its OPEN replica resurrects the
> container (HDDS-12421) and the key is readable again from d2. The
> resurrection by an OPEN replica was run at the SCM level, the read from d2
> was not run. In total: three datanode outages on the same pipeline in a fixed
> order, one of them permanent, plus one more restart of d1.
> With more than three datanodes SCM would copy d3's replica while d3 is up, so
> every copy would have to be unknown to SCM at the same replication manager
> pass, or d3 stops before its replica has been copied (argued, not run).
> Two variants, read from the code and not run: an SCM restart while d3 is down
> gives SCM the same view without the dead node interval. And on a cluster with
> spare datanodes a lagging replica can become empty through ordinary key
> deletion without a restart ({{BlockDeletingTask}} marks it empty, and
> {{DeletedBlockLogImpl.checkInadequateReplica}} lets block deletions through
> once the replication manager reports the container healthy); the deletion
> then needs all copies with the higher BCSID to be dead at the same time, and
> all of them are force deleted when they return, with no permanent node loss.
> h3. Impact
> SCM orders the deletion of the only replica that is still reachable. The data
> survives only on d2. If d2 is lost, the key is lost although d3 came back
> intact, where without the deletion one permanent node loss would have been
> survived. In the mini cluster run the read of the key failed with
> {{NO_REPLICA_FOUND}} after d3 had deleted its replica. The code is in 2.2.0
> and 2.2.1 (HDDS-14684 and HDDS-14416 are in both release tags).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC6EmptyQuasiClosedContainerDeletion.java]
> ({{hadoop-hdds/server-scm}}, about 2 seconds) runs the SCM side with the real
> container manager, container report handler, {{DeadNodeHandler}},
> {{EmptyContainerHandler}} and {{DeletingContainerHandler}}. The node manager
> and pipeline manager are the SCM test mocks and the datanodes are represented
> by the container reports they would send. d2 does not appear in it. It shows
> the container going from QUASI_CLOSED (sequence ID 4) to DELETED on one empty
> replica at BCSID 0, and a forced delete command for the returning non-empty
> replica at BCSID 4. A passing test means the defect is present.
> The full sequence above was also run once on a three datanode mini cluster
> with stale and dead node intervals shortened to 40s and 60s. The client watch
> type was set to {{MAJORITY_COMMITTED}} to avoid waiting for the default
> {{ALL_COMMITTED}} watch to time out and fall back to it (the fallback is read
> in {{XceiverClientRatis.watchForCommit}}, the default was not run). The run
> ended with the forced delete executed on d3 ("Marked container DELETED from
> QUASI_CLOSED ... non-empty") and the read of the committed key failing with
> {{NO_REPLICA_FOUND}}. That test takes about 3.5 minutes and is not attached.
> h3. Patch
> [^MC-6-keep-empty-quasi-closed-container-without-replica-at-container-bcsid.patch],
> against 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master
> at 8ca44914771.
> In the QUASI_CLOSED branch, when the highest sequence ID among the known
> replicas is below the container's sequence ID, the handler reports the
> container as EMPTY and returns without changing its state and without
> deleting replicas. This is the HDDS-14551 rule applied to the QUASI_CLOSED
> branch. Unlike the CLOSED branch it also keeps the empty replicas, because a
> later force close needs replicas from all origins.
> Trade-off: a QUASI_CLOSED container whose known replicas are all empty and
> all below the container's sequence ID is no longer deleted. It stays
> QUASI_CLOSED and is counted as EMPTY until a replica at the container's
> sequence ID is reported. This includes the last step of the manual test in
> the HDDS-14684 pull request, where a container that was deleted once is
> resurrected by a stale replica that later becomes empty ("container BCSID=14,
> max replica BCSID=8"). The case HDDS-14684 was written for is not affected:
> containers that were created on the followers and never written (HDDS-14083)
> have sequence ID 0 and replicas at 0, so they are still deleted (read from
> the code). What stays behind is only a container whose replica with the
> highest BCSID SCM has seen and no longer knows, with every remaining replica
> empty. Nothing else cleans those up, which is the state before 2.2.0 for them.
> Limit: the patch is the guard alone. It compares against the sequence ID SCM
> currently holds, and SCM persists that value only together with a state
> transition. After an SCM restart, or for a replica whose BCSID SCM never
> recorded, the container is still deleted and the returning replica still gets
> the forced delete. Run at the SCM level with the patch and JVM assertions
> off: d1 (BCSID 0) quasi closes first, so QUASI_CLOSED is persisted with
> sequence ID 0. d3 reports BCSID 4. A second container manager over the same
> database, which stands in for an SCM restart while d3 is down, holds sequence
> ID 0 and no replicas. d1 reports empty, the container goes to DELETING and
> DELETED, and d3's returning replica at BCSID 4 gets the forced delete.
> Not fixed here: the order of {{updateContainerStats}} and
> {{updateContainerState}} described under Mechanism. Not raising the sequence
> ID of a DELETING or DELETED container makes a QUASI_CLOSED replica above it
> resurrect the container, which also closes the restart case above. A CLOSED
> replica above it then resurrects the container to CLOSED with the old
> sequence ID, and its next report reaches the same {{assert}} (both run in a
> probe, not part of the patch). That needs a decision on the resurrect path.
> Opinions on which way to go are welcome.
> Covered by
> {{testEmptyQuasiClosedContainerNotDeletedWhenNoReplicaHasContainerSequenceId}}
> in the existing {{TestEmptyContainerHandler}}, which also checks that the
> container is deleted once an empty replica at the container's sequence ID is
> known. Without the change it fails because the DELETE event is fired. With it
> the empty, deleting and quasi closed handler suites, the replication manager
> suites, the container report handler suites, the container manager and state
> manager suites and {{TestDeadNodeHandler}} (296 tests) and
> {{TestReplicationManagerIntegration}} on a mini cluster (8 tests) pass,
> checkstyle is clean. With the patch the reproduction fails because the
> container stays QUASI_CLOSED, and in the mini cluster run the key stays
> readable after d3 returns. Containers that already went to DELETING or
> DELETED this way are not repaired.
> Found by TLA+ model checking and code review of the container reconciliation
> paths in the datanode and SCM under HDDS-15926, on commit
> 1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
