[
https://issues.apache.org/jira/browse/HDDS-17060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17060:
------------------------------
Attachment: TestBugMC8ResurrectionNotMirrored.java
> Recon keeps a container CLOSED after SCM moves it from DELETING or DELETED
> back to QUASI_CLOSED
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-17060
> URL: https://issues.apache.org/jira/browse/HDDS-17060
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-8-follow-scm-from-closed-back-to-quasi-closed.patch,
> TestBugMC8ResurrectionNotMirrored.java
>
>
> h3. Mechanism
> SCM's replication manager moves an empty CLOSED container to DELETING and,
> once no replica is left, to DELETED. If a datanode then reports a non-empty
> Ratis replica that is OPEN, CLOSING, UNHEALTHY, or QUASI_CLOSED with a bcsId
> above the container's,
> {{AbstractContainerReportHandler.updateContainerState}} moves the container
> back to QUASI_CLOSED (HDDS-14684, which extended HDDS-12421). Recon does not
> follow when it still has the container CLOSED at that point:
> * {{ReconStorageContainerSyncHelper}} never copies DELETING, and copies
> DELETED only from SCM's current DELETED list, which no longer contains the
> container.
> * Recon's report handler runs the same {{updateContainerState}}, which does
> nothing for a CLOSED container.
> * {{ReconStorageContainerSyncHelper.reconcileToQuasiClosed}} handles a
> container that SCM lists as QUASI_CLOSED only when Recon has it DELETED, OPEN
> or CLOSING. For CLOSED it logs at debug level and skips it, on every cycle.
> The skip is deliberate
> ({{TestReconSCMContainerSyncIntegration.doesNotOverwriteContainerAlreadyClosedInRecon}}):
> Recon can be ahead of SCM when it processed a CLOSED replica that SCM has
> not processed yet. But SCM never moves a CLOSED container to QUASI_CLOSED
> directly, only through DELETING or DELETED, so in this case SCM is ahead.
> h3. Trigger
> # An empty CLOSED container; a datanode that holds a stale non-empty copy is
> away.
> # SCM's replication manager deletes the empty replicas and moves the
> container to DELETING (then DELETED once the deletes are confirmed).
> # The datanode comes back and reports its replica (UNHEALTHY in the
> reproduction) before Recon's next sync has run its DELETED pass. SCM moves
> the container back to QUASI_CLOSED.
> The window runs from SCM's DELETE transition until Recon's next sync cycle
> has run its DELETED pass
> ({{ozone.recon.scm.container.sync.task.interval.delay}}, default 6h); whether
> SCM is in DELETING or already in DELETED does not matter. If Recon has
> already synced DELETED, its own report handler moves the container back to
> QUASI_CLOSED together with SCM. If SCM later closes the container (a CLOSED
> replica appears) or deletes it again, the CLOSED or DELETED pass catches up.
> With only an UNHEALTHY replica left, SCM keeps it QUASI_CLOSED.
> h3. Impact
> Recon shows the container as CLOSED and leaves it out of its QUASI_CLOSED
> list and count ({{ContainerEndpoint.getQuasiClosedContainers}}) for as long
> as SCM keeps it QUASI_CLOSED. Recon's QUASI_CLOSED specific health states
> cannot apply to it (argued from the code, not run). SCM and datanode state
> are not affected. The same code is in 2.2.0 and 2.2.1. Recon replaces its SCM
> DB from a snapshot only at startup with a large count difference or on an
> admin request.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC8ResurrectionNotMirrored.java] ({{hadoop-ozone/recon}}) runs a
> real SCM {{ContainerManagerImpl}} and SCM
> {{IncrementalContainerReportHandler}} next to the real
> {{ReconContainerManager}}, {{ReconIncrementalContainerReportHandler}} and
> {{ReconStorageContainerSyncHelper}}. The SCM client that Recon calls answers
> from the SCM container manager. Every datanode report goes to both sides; the
> only step applied directly is the {{DELETE}} (and {{CLEANUP}}) transition the
> replication manager makes. Two tests end, after six sync cycles and a
> repeated report, with SCM QUASI_CLOSED and Recon CLOSED: one with the report
> arriving while SCM is DELETING, one while SCM is already DELETED but before
> Recon's next sync. A control test where Recon synced DELETED first shows
> Recon following SCM. A passing test means the defect is present.
> h3. Suggested fix
> A rule that waits for two consecutive sync cycles is not needed. When Recon
> is ahead, it got there by closing the container on a CLOSED replica, which it
> records. So Recon can follow SCM back to QUASI_CLOSED whenever SCM lists the
> container QUASI_CLOSED and Recon knows no CLOSED replica of it. This is a
> heuristic with bounded, self healing errors: if Recon was ahead and has lost
> its CLOSED replica record (for example the node died), it follows SCM to
> QUASI_CLOSED and closes again through the CLOSED pass or the next CLOSED
> replica report once SCM closes the container; a stale CLOSED replica record
> delays the move until a report removes it.
> h3. Patch
> [^MC-8-follow-scm-from-closed-back-to-quasi-closed.patch], against
> 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
> 62d982ff2d9.
> In {{reconcileToQuasiClosed}}, a container that Recon has CLOSED and SCM
> lists as QUASI_CLOSED is moved to QUASI_CLOSED when none of its replicas in
> Recon is CLOSED. It uses the same transitions SCM went through ({{DELETE}},
> then {{transitionDeletingOrDeletedToTargetState}}), so the record and its
> replicas are kept. A container with a CLOSED replica is still skipped as
> before.
> Covered by the new {{followsScmBackToQuasiClosedWhenReconHasNoClosedReplica}}
> in {{TestReconSCMContainerSyncIntegration}};
> {{doesNotOverwriteContainerAlreadyClosedInRecon}} now gives Recon the CLOSED
> replica that makes it ahead of SCM, and still expects CLOSED. Without the
> change the new test fails with Recon CLOSED; with it the Recon {{scm}}
> package tests pass and checkstyle is clean. With the patch the two defect
> tests of the reproduction fail because Recon ends QUASI_CLOSED, and the
> control still passes.
> Found by TLA+ model checking and code review of the datanode container report
> and Recon container tracking paths under HDDS-15926, on commit
> 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
