[
https://issues.apache.org/jira/browse/HDDS-16925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16925:
------------------------------
Attachment: TestBugCR8ReplicaWipeOnReinitialize.java
> SCM forgets all container replicas when it installs a Ratis snapshot, until
> the next full container reports
> -----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16925
> URL: https://issues.apache.org/jira/browse/HDDS-16925
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-8-keep-container-replicas-across-reinitialize.patch,
> TestBugCR8ReplicaWipeOnReinitialize.java
>
>
> h3. Mechanism
> {{ContainerStateManagerImpl.reinitialize}}, which runs when an SCM installs a
> checkpoint from the leader ({{SCMHAManagerImpl.startServices}}), replaces its
> {{ContainerStateMap}} with an empty one and reloads only the
> {{ContainerInfo}} rows from the installed DB. Replicas are not in the DB;
> they come from datanode reports. So after the install the SCM knows no
> replica of any container, although the datanodes still hold them all. They
> come back with incremental reports for changed containers and with each
> datanode's next full container report ({{hdds.container.report.interval}}, 60
> minutes by default). Nothing asks for an earlier full report, and safe mode
> is not entered again on an install or on a leader change. Replica updates
> take only their stripe lock, not the global lock that {{reinitialize}} holds,
> so updates that run during the swap are lost as well.
> Recon's SCM DB snapshot sync
> ({{ReconStorageContainerManagerFacade.initializeNewRdbStore}}) calls the same
> method and wipes Recon's own replica view the same way; the sync is a Recon
> admin operation and Recon sends no commands (by code reading, not run).
> h3. Trigger
> # A follower SCM that is already processing datanode reports installs a
> checkpoint from the leader. With the default
> {{ozone.scm.ha.ratis.log.purge.enabled}}=false the leader purges its log only
> up to the lowest commit index of its peers, so a lagging follower normally
> catches up from the log. It installs a checkpoint when log purge is enabled
> and it falls behind the purged index, for example after a long pause. A newly
> bootstrapped or restarted SCM is not affected, because it starts its datanode
> server only after it has caught up.
> # Within the next report interval, that SCM becomes leader (failover or
> leadership transfer).
> h3. Impact
> While the new leader has not received the full reports (argued from the code
> unless stated otherwise):
> * {{getContainerWithPipeline}} for a closed container returns an empty
> pipeline, because the pipeline of a closed container is built from its
> replicas. Reads that need a fresh lookup fail.
> * After {{hdds.scm.wait.time.after.safemode.exit}} (5 minutes),
> ReplicationManager sees the containers as MISSING (no replica known) or
> UNDER_REPLICATED (some added back by incremental reports), replicates or
> reconstructs them and later deletes the extra copies as over replicated. A
> DELETING container with no known replica is marked DELETED while datanodes
> still hold it.
> * Decommission and maintenance: the monitor takes the node's containers from
> NodeManager (not reset) and their replicas from ContainerManager (reset).
> This monitor is not delayed by {{hdds.scm.wait.time.after.safemode.exit}}:
> the new leader resumes tracking the nodes in
> {{NodeDecommissionManager.onBecomeLeader}} and checks them every
> {{ozone.scm.datanode.admin.monitor.interval}} (30 seconds), so it acts on the
> wiped view right after the failover (by code reading). A Ratis container then
> holds the decommission until the reports return. An EC container with no
> known replica is classified MISSING, not UNDER_REPLICATED, so it counts as
> sufficiently replicated, and the node can finish decommission or enter
> maintenance without its EC replicas being copied. Shown in the reproduction
> with the real check handler; the monitor and the failover were not run.
> * Everything returns to normal after the full reports. I found no path to
> data loss on its own.
> h3. Reproduction
> PASS on the unmodified source. [^TestBugCR8ReplicaWipeOnReinitialize.java]
> ({{hadoop-hdds/server-scm}}) builds a real {{ContainerStateManagerImpl}} on a
> real SCM DB with a CLOSED Ratis THREE container and a CLOSED EC 3+2
> container, each with a replica on a DECOMMISSIONING datanode, and calls
> {{reinitialize}} with the same container table. It then applies the per
> container decision of
> {{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} with the real
> {{RatisContainerReplicaCount}}, {{ECContainerReplicaCount}} and
> {{ECReplicationCheckHandler}}. Before: the EC container is UNDER_REPLICATED
> and holds the decommission. After: {{Ratis replicas 0 (was 3), EC replicas 0
> (was 5)}}, the EC container is MISSING and passes the decommission check. A
> passing test means the defect is present.
> h3. Not covered by the patch
> Containers created on the leader while the follower lagged still have no
> replicas on the new SCM until the datanodes report them. A leader readiness
> gate after an install (wait for full reports, or ask the datanodes for one
> before acting as leader) would cover those, as a follow up.
> h3. Patch
> [^CR-8-keep-container-replicas-across-reinitialize.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> The patch keeps the replica sets of the containers that are still in the
> installed table. Inside the existing global write lock it also takes the
> write lock of every stripe, so a replica update cannot run during the swap
> and be overwritten by the copied set. Containers that are no longer in the
> table lose their replicas with them. Lock order is unchanged (global, then
> stripe). Report processing on that SCM waits while the container table
> reloads, which already blocks container reads. With the patch the
> reproduction fails because the replicas are still there.
> Covered by {{testReinitializeKeepsReplicasOfRemainingContainers}} and
> {{testReplicaUpdateDuringReinitializeIsNotLost}} in the existing
> {{TestContainerStateManager}}. Without the change both fail (empty replica
> sets). Copying the replicas without the stripe locks fails the second test,
> because a replica removed during the swap comes back. With the patch the
> container, report handler and replica suites in {{hadoop-hdds/server-scm}}
> (140 tests), the Recon container suites (22 tests) and
> {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} (5 tests)
> pass, and checkstyle is clean.
> Found by code review of the SCM Ratis snapshot and checkpoint install paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. HDDS-16896 (safe
> mode rules of a bootstrapped SCM) is related but a different mechanism. The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
