Siyao Meng created HDDS-17060:
---------------------------------
Summary: Recon keeps a container CLOSED after SCM moves it from
DELETING or DELETED back to QUASI_CLOSED
Key: HDDS-17060
URL: https://issues.apache.org/jira/browse/HDDS-17060
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-8-follow-scm-from-closed-back-to-quasi-closed.patch,
TestBugMC8ResurrectionNotMirrored.java
h3. Mechanism
SCM's replication manager moves an empty CLOSED container to DELETING and, once
no replica is left, to DELETED. If a datanode then reports a non-empty Ratis
replica that is OPEN, CLOSING, UNHEALTHY, or QUASI_CLOSED with a bcsId above
the container's, {{AbstractContainerReportHandler.updateContainerState}} moves
the container back to QUASI_CLOSED (HDDS-14684, which extended HDDS-12421).
Recon does not follow when it still has the container CLOSED at that point:
* {{ReconStorageContainerSyncHelper}} never copies DELETING, and copies DELETED
only from SCM's current DELETED list, which no longer contains the container.
* Recon's report handler runs the same {{updateContainerState}}, which does
nothing for a CLOSED container.
* {{ReconStorageContainerSyncHelper.reconcileToQuasiClosed}} handles a
container that SCM lists as QUASI_CLOSED only when Recon has it DELETED, OPEN
or CLOSING. For CLOSED it logs at debug level and skips it, on every cycle.
The skip is deliberate
({{TestReconSCMContainerSyncIntegration.doesNotOverwriteContainerAlreadyClosedInRecon}}):
Recon can be ahead of SCM when it processed a CLOSED replica that SCM has not
processed yet. But SCM never moves a CLOSED container to QUASI_CLOSED directly,
only through DELETING or DELETED, so in this case SCM is ahead.
h3. Trigger
# An empty CLOSED container; a datanode that holds a stale non-empty copy is
away.
# SCM's replication manager deletes the empty replicas and moves the container
to DELETING (then DELETED once the deletes are confirmed).
# The datanode comes back and reports its replica (UNHEALTHY in the
reproduction) before Recon's next sync has run its DELETED pass. SCM moves the
container back to QUASI_CLOSED.
The window runs from SCM's DELETE transition until Recon's next sync cycle has
run its DELETED pass ({{ozone.recon.scm.container.sync.task.interval.delay}},
default 6h); whether SCM is in DELETING or already in DELETED does not matter.
If Recon has already synced DELETED, its own report handler moves the container
back to QUASI_CLOSED together with SCM. If SCM later closes the container (a
CLOSED replica appears) or deletes it again, the CLOSED or DELETED pass catches
up. With only an UNHEALTHY replica left, SCM keeps it QUASI_CLOSED.
h3. Impact
Recon shows the container as CLOSED and leaves it out of its QUASI_CLOSED list
and count ({{ContainerEndpoint.getQuasiClosedContainers}}) for as long as SCM
keeps it QUASI_CLOSED. Recon's QUASI_CLOSED specific health states cannot apply
to it (argued from the code, not run). SCM and datanode state are not affected.
The same code is in 2.2.0 and 2.2.1. Recon replaces its SCM DB from a snapshot
only at startup with a large count difference or on an admin request.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC8ResurrectionNotMirrored.java] ({{hadoop-ozone/recon}}) runs a real
SCM {{ContainerManagerImpl}} and SCM {{IncrementalContainerReportHandler}} next
to the real {{ReconContainerManager}},
{{ReconIncrementalContainerReportHandler}} and
{{ReconStorageContainerSyncHelper}}. The SCM client that Recon calls answers
from the SCM container manager. Every datanode report goes to both sides; the
only step applied directly is the {{DELETE}} (and {{CLEANUP}}) transition the
replication manager makes. Two tests end, after six sync cycles and a repeated
report, with SCM QUASI_CLOSED and Recon CLOSED: one with the report arriving
while SCM is DELETING, one while SCM is already DELETED but before Recon's next
sync. A control test where Recon synced DELETED first shows Recon following
SCM. A passing test means the defect is present.
h3. Suggested fix
A rule that waits for two consecutive sync cycles is not needed. When Recon is
ahead, it got there by closing the container on a CLOSED replica, which it
records. So Recon can follow SCM back to QUASI_CLOSED whenever SCM lists the
container QUASI_CLOSED and Recon knows no CLOSED replica of it. This is a
heuristic with bounded, self healing errors: if Recon was ahead and has lost
its CLOSED replica record (for example the node died), it follows SCM to
QUASI_CLOSED and closes again through the CLOSED pass or the next CLOSED
replica report once SCM closes the container; a stale CLOSED replica record
delays the move until a report removes it.
h3. Patch
[^MC-8-follow-scm-from-closed-back-to-quasi-closed.patch], against
916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
62d982ff2d9.
In {{reconcileToQuasiClosed}}, a container that Recon has CLOSED and SCM lists
as QUASI_CLOSED is moved to QUASI_CLOSED when none of its replicas in Recon is
CLOSED. It uses the same transitions SCM went through ({{DELETE}}, then
{{transitionDeletingOrDeletedToTargetState}}), so the record and its replicas
are kept. A container with a CLOSED replica is still skipped as before.
Covered by the new {{followsScmBackToQuasiClosedWhenReconHasNoClosedReplica}}
in {{TestReconSCMContainerSyncIntegration}};
{{doesNotOverwriteContainerAlreadyClosedInRecon}} now gives Recon the CLOSED
replica that makes it ahead of SCM, and still expects CLOSED. Without the
change the new test fails with Recon CLOSED; with it the Recon {{scm}} package
tests pass and checkstyle is clean. With the patch the two defect tests of the
reproduction fail because Recon ends QUASI_CLOSED, and the control still passes.
Found by TLA+ model checking and code review of the datanode container report
and Recon container tracking paths under HDDS-15926, on commit
916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
