[
https://issues.apache.org/jira/browse/HDDS-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16891:
------------------------------
Attachment: TestBugMC8PendingAllocationFailover.java
> After an SCM restart or leader change, SCM has no pending container
> allocations and grants again the slots reserved for containers not yet written
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16891
> URL: https://issues.apache.org/jira/browse/HDDS-16891
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: TestBugMC8PendingAllocationFailover.java
>
>
> h3. Mechanism
> {{PendingContainerTracker}} keeps the reservations in memory, in the
> {{TwoWindowBucket}} of each {{DatanodeInfo}}. Only the leader fills it:
> {{ContainerManagerImpl.allocateContainer}} calls
> {{PipelineManagerImpl.checkSpaceAndRecordAllocation}} before the replicated
> {{containerStateManager.addContainer}}. Followers apply {{addContainer}}, so
> they know the container, but they do not record its slot. On restart,
> {{ContainerStateManagerImpl.initialize}} loads the containers from the SCM DB
> and nothing records their slots either. Nothing records the slots when an SCM
> becomes leader. Datanodes create a container on its first write, so until
> then no storage report counts it as committed. After a leader change or a
> restart, SCM therefore sees the slots of granted containers that are not
> written yet as free.
> Every SCM receives the replicated {{addContainer}} and the heartbeats and
> container reports that datanodes send to all SCMs (followers run the report
> handlers, which also remove pending entries). The gap is that only the leader
> records the reservation, before submitting.
> h3. Trigger
> # The leader allocates container C1 on a pipeline whose datanodes have one
> slot left. C1 is not written yet.
> # Leadership moves to another SCM (transfer, crash, long pause), or the SCM
> restarts. This includes a single SCM.
> # Once out of safe mode, the leader allocates C2 on the same datanodes.
> h3. Impact
> A leader change or restart drops every unconfirmed reservation on every
> datanode at once, as if all of them had aged out. Only reservations younger
> than two roll intervals ({{ozone.scm.pending.container.roll.interval}},
> default 5m) and not yet confirmed by a container report are affected, and
> only datanodes that are down to their last slots can be over granted. The
> expected effect is that the datanode refuses one of the containers for lack
> of space when it is created. On a Ratis pipeline the failed create marks the
> pipeline's {{ContainerStateMachine}} unhealthy, so the pipeline is closed and
> the in flight writes on it are retried elsewhere. This is argued from the
> code and was not run. Replica placement after the leader change reads the
> same empty buckets; that variant was not run either.
> h3. Reproduction
> [^TestBugMC8PendingAllocationFailover.java]
> ({{hadoop-ozone/integration-test}}) starts a mini cluster with 3 SCMs, 1 OM
> and 3 datanodes. Each datanode volume uses the existing
> {{MockSpaceUsageCheckFactory$HalfTera}} (512 GiB) and the container size is
> 300 GiB, so each datanode has one slot. Only the admin calls
> {{allocateContainer}} and {{transferLeadership}} are used. The leader grants
> C1 and holds one pending entry per datanode, and it refuses a second
> container. After the transfer, once the new leader is ready and out of safe
> mode, it has C1 OPEN with no replicas, committed space 0 and no pending entry
> on all three datanodes, and it grants C2 on the same three datanodes. A
> passing test means the defect is present (under a minute on an unmodified
> checkout). The restart case was not run.
> h3. Suggested fix
> No patch is attached yet, so that the design can be agreed first. The
> proposal is a rebuild of the buckets when an SCM becomes the serving leader:
> * Run once the SCM is leader ready and out of safe mode, the point where
> {{ReplicationManager}} starts. After a restart the datanodes are registered
> by then, and allocation is refused in safe mode anyway.
> * First drop all pending entries, then seed: for each OPEN container, record
> a slot on each datanode of its pipeline that has no replica of it.
> * Finish the rebuild before the first allocation is served.
> This covers a leader change, a restart of one SCM or of all of them, a
> follower that installed a Ratis snapshot, and a newly bootstrapped SCM. The
> drop also clears stale entries of a returning leader: replication
> reservations that are not removed when {{ReplicationManager}} clears its
> pending ops on leader change, and the case below. It keeps the apply path
> free of node manager calls. Its cost is that it also reserves for OPEN
> containers that the previous leader had already aged out (for example
> containers {{getMatchingContainer}} opened ahead of use), so for up to two
> roll intervals after the rebuild SCM grants fewer containers than before,
> never more. A datanode that registers after the rebuild starts with an empty
> bucket, as today.
> The alternative, recording the slot whenever an SCM applies the replicated
> {{addContainer}}, needs no scan, but it does not cover an SCM restart or a
> snapshot install, because containers loaded from the SCM DB are not applied
> again. It also puts a node manager call in the apply path (which Recon runs
> too), has to avoid counting the leader's own entry twice, and on a lagging
> follower the container report for a container can arrive before the follower
> applies its {{addContainer}}, leaving the entry to age out. Waiting for a
> container report interval after leader ready does not help either, because a
> container that is not written yet is in no report.
> A related case: when the replicated {{addContainer}} fails after the slot was
> reserved, the reservation is kept. That is deliberate.
> {{SCMRatisServerImpl.submitRequest}} gives up after
> {{ozone.scm.ha.ratis.request.timeout}} while the entry stays in the Raft log,
> and on step down Ratis (3.3.1) answers the requests already in the leader's
> log with {{NotLeaderException}} although the next leader may commit them. A
> timeout never says whether the entry will commit, and a step down reply has
> the same result code ({{SCM_NOT_LEADER}}) as a definite rejection, so
> releasing on failure could turn a bounded under grant into an over grant.
> Such an entry ages out within two roll intervals, or is dropped by the
> rebuild above when the SCM becomes leader again.
> Found by TLA+ model checking and code review of the SCM pending container
> allocation tracking under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
