Siyao Meng created HDDS-16809:
---------------------------------
Summary: Empty QUASI_CLOSED container is deleted although no known
replica has its sequence ID, and the returning data replica is force deleted
Key: HDDS-16809
URL: https://issues.apache.org/jira/browse/HDDS-16809
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-keep-empty-quasi-closed-container-without-replica-at-container-bcsid.patch,
TestBugMC6EmptyQuasiClosedContainerDeletion.java
h3. Mechanism
{{EmptyContainerHandler.handle}} moves a QUASI_CLOSED container to DELETING as
soon as every replica SCM currently knows is empty
({{isContainerEmptyAndQuasiClosed}}). It does not require that any of those
replicas has reached the container's sequence ID. The CLOSED branch of the same
method has that check since HDDS-14551 (code comment: "don't update container
state if replica seqid don't match with container seq id"). The QUASI_CLOSED
branch was added afterwards by HDDS-14684 without it.
{{DeletingContainerHandler}} then moves the container to DELETED once no
replica is known. When a non-empty CLOSED or QUASI_CLOSED replica of a DELETING
or DELETED RATIS container is reported, {{AbstractContainerReportHandler}}
answers with a forced {{DeleteContainerCommand}}, whatever the replica's BCSID.
The code (HDDS-14416, extended to QUASI_CLOSED replicas by HDDS-14684) compares
the replica's BCSID with the container's sequence ID and means to keep a
replica that is above it. But {{processContainerReplica}} calls
{{updateContainerStats}} before {{updateContainerState}}, and
{{updateContainerStats}} raises the container's sequence ID to the replica's
BCSID for every healthy replica without looking at the container state.
{{ContainerInfo.updateSequenceId}} excludes DELETING and DELETED only with a
Java {{assert}}, and the launch scripts do not enable assertions. So when the
comparison runs, the replica is never above the container. Only OPEN, CLOSING
and UNHEALTHY replicas move the container back.
This was discussed before. HDDS-12305 (resolved as duplicate of HDDS-12421)
asked not to mark a container DELETED on empty replicas at BCSID 0. The answer
there was that SCM cannot decide this at deletion time, so a replica with data
that reports later moves the container back and no forced delete is used, at
the price of orphan containers. HDDS-14416 then introduced the forced delete
for returning CLOSED replicas at or below the container's sequence ID, and
HDDS-14684 allowed the deletion of empty QUASI_CLOSED containers on the premise
that a later replica with the missing blocks "would be tracked and replicated
again". For a CLOSED or QUASI_CLOSED replica that premise does not hold.
h3. Trigger
RATIS/THREE container on d1, d2, d3. No operator action, only datanode outages:
# d1 stops after the first chunk write has created the container on it and
before the first block commit is applied there. The block is committed on d2
and d3 (BCSID 4). d1's replica stays at BCSID 0 with no block. If the container
is created while d1 is down, d1 has no replica and nothing is deleted.
# d2 goes down before the container is closed. d3 quasi closes, the container
becomes QUASI_CLOSED with sequence ID 4.
# d1 returns, its replica is quasi closed, and d1 restarts once more. A
datanode marks a non-open replica without blocks as empty only when it loads
the container, so d1 now reports an empty replica at BCSID 0.
# d3 is down longer than the dead node interval. {{DeadNodeHandler}} removes
its replica record, SCM knows only d1's empty replica.
# The replication manager runs {{EmptyContainerHandler}}: "Deleting empty
QUASI_CLOSED container #1 (container BCSID=4, max replica BCSID=0)". The
container goes to DELETING, then DELETED.
# d3 returns and reports QUASI_CLOSED, BCSID 4, non-empty. SCM sends a forced
delete and d3 deletes the replica.
d2 must not come back. If it returns, its OPEN replica resurrects the container
(HDDS-12421) and the key is readable again from d2. The resurrection by an OPEN
replica was run at the SCM level, the read from d2 was not run. In total: three
datanode outages on the same pipeline in a fixed order, one of them permanent,
plus one more restart of d1.
With more than three datanodes SCM would copy d3's replica while d3 is up, so
every copy would have to be unknown to SCM at the same replication manager
pass, or d3 stops before its replica has been copied (argued, not run).
Two variants, read from the code and not run: an SCM restart while d3 is down
gives SCM the same view without the dead node interval. And on a cluster with
spare datanodes a lagging replica can become empty through ordinary key
deletion without a restart ({{BlockDeletingTask}} marks it empty, and
{{DeletedBlockLogImpl.checkInadequateReplica}} lets block deletions through
once the replication manager reports the container healthy); the deletion then
needs all copies with the higher BCSID to be dead at the same time, and all of
them are force deleted when they return, with no permanent node loss.
h3. Impact
SCM orders the deletion of the only replica that is still reachable. The data
survives only on d2. If d2 is lost, the key is lost although d3 came back
intact, where without the deletion one permanent node loss would have been
survived. In the mini cluster run the read of the key failed with
{{NO_REPLICA_FOUND}} after d3 had deleted its replica. The code is in 2.2.0 and
2.2.1 (HDDS-14684 and HDDS-14416 are in both release tags).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC6EmptyQuasiClosedContainerDeletion.java]
({{hadoop-hdds/server-scm}}, about 2 seconds) runs the SCM side with the real
container manager, container report handler, {{DeadNodeHandler}},
{{EmptyContainerHandler}} and {{DeletingContainerHandler}}. The node manager
and pipeline manager are the SCM test mocks and the datanodes are represented
by the container reports they would send. d2 does not appear in it. It shows
the container going from QUASI_CLOSED (sequence ID 4) to DELETED on one empty
replica at BCSID 0, and a forced delete command for the returning non-empty
replica at BCSID 4. A passing test means the defect is present.
The full sequence above was also run once on a three datanode mini cluster with
stale and dead node intervals shortened to 40s and 60s. The client watch type
was set to {{MAJORITY_COMMITTED}} to avoid waiting for the default
{{ALL_COMMITTED}} watch to time out and fall back to it (the fallback is read
in {{XceiverClientRatis.watchForCommit}}, the default was not run). The run
ended with the forced delete executed on d3 ("Marked container DELETED from
QUASI_CLOSED ... non-empty") and the read of the committed key failing with
{{NO_REPLICA_FOUND}}. That test takes about 3.5 minutes and is not attached.
h3. Patch
[^MC-6-keep-empty-quasi-closed-container-without-replica-at-container-bcsid.patch],
against 1e528b97aed3eef1be6846f1f0767c1ef98a883a. It also applies to master at
8ca44914771.
In the QUASI_CLOSED branch, when the highest sequence ID among the known
replicas is below the container's sequence ID, the handler reports the
container as EMPTY and returns without changing its state and without deleting
replicas. This is the HDDS-14551 rule applied to the QUASI_CLOSED branch.
Unlike the CLOSED branch it also keeps the empty replicas, because a later
force close needs replicas from all origins.
Trade-off: a QUASI_CLOSED container whose known replicas are all empty and all
below the container's sequence ID is no longer deleted. It stays QUASI_CLOSED
and is counted as EMPTY until a replica at the container's sequence ID is
reported. This includes the last step of the manual test in the HDDS-14684 pull
request, where a container that was deleted once is resurrected by a stale
replica that later becomes empty ("container BCSID=14, max replica BCSID=8").
The case HDDS-14684 was written for is not affected: containers that were
created on the followers and never written (HDDS-14083) have sequence ID 0 and
replicas at 0, so they are still deleted (read from the code). What stays
behind is only a container whose replica with the highest BCSID SCM has seen
and no longer knows, with every remaining replica empty. Nothing else cleans
those up, which is the state before 2.2.0 for them.
Limit: the patch is the guard alone. It compares against the sequence ID SCM
currently holds, and SCM persists that value only together with a state
transition. After an SCM restart, or for a replica whose BCSID SCM never
recorded, the container is still deleted and the returning replica still gets
the forced delete. Run at the SCM level with the patch and JVM assertions off:
d1 (BCSID 0) quasi closes first, so QUASI_CLOSED is persisted with sequence ID
0. d3 reports BCSID 4. A second container manager over the same database, which
stands in for an SCM restart while d3 is down, holds sequence ID 0 and no
replicas. d1 reports empty, the container goes to DELETING and DELETED, and
d3's returning replica at BCSID 4 gets the forced delete.
Not fixed here: the order of {{updateContainerStats}} and
{{updateContainerState}} described under Mechanism. Not raising the sequence ID
of a DELETING or DELETED container makes a QUASI_CLOSED replica above it
resurrect the container, which also closes the restart case above. A CLOSED
replica above it then resurrects the container to CLOSED with the old sequence
ID, and its next report reaches the same {{assert}} (both run in a probe, not
part of the patch). That needs a decision on the resurrect path. Opinions on
which way to go are welcome.
Covered by
{{testEmptyQuasiClosedContainerNotDeletedWhenNoReplicaHasContainerSequenceId}}
in the existing {{TestEmptyContainerHandler}}, which also checks that the
container is deleted once an empty replica at the container's sequence ID is
known. Without the change it fails because the DELETE event is fired. With it
the empty, deleting and quasi closed handler suites, the replication manager
suites, the container report handler suites, the container manager and state
manager suites and {{TestDeadNodeHandler}} (296 tests) and
{{TestReplicationManagerIntegration}} on a mini cluster (8 tests) pass,
checkstyle is clean. With the patch the reproduction fails because the
container stays QUASI_CLOSED, and in the mini cluster run the key stays
readable after d3 returns. Containers that already went to DELETING or DELETED
this way are not repaired.
Found by TLA+ model checking and code review of the container reconciliation
paths in the datanode and SCM under HDDS-15926, on commit
1e528b97aed3eef1be6846f1f0767c1ef98a883a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
