[
https://issues.apache.org/jira/browse/HDDS-17559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17559:
------------------------------
Attachment: TestBugCR3DecommissionReconstructionTakesFreedSlot.java
> EC decommission reconstruction can take a global reconstruction slot while a
> container with a missing index waits for one
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17559
> URL: https://issues.apache.org/jira/browse/HDDS-17559
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-3-keep-freed-reconstruction-slot-for-deferred-missing-index.patch,
> TestBugCR3DecommissionReconstructionTakesFreedSlot.java
>
>
> h3. Mechanism
> With {{hdds.scm.replication.decommission.ec.reconstruction.enabled=true}} and
> {{hdds.scm.replication.reconstruction.global.limit}} above 0, both missing
> index recovery and decommission recovery of EC containers compete for the
> same global reconstruction slots, and a slot goes to whichever container asks
> first. When a container with a missing index is dequeued while the limit is
> reached, {{ReplicationManager.sendThrottledReconstructionCommand}} rejects it
> and {{UnhealthyReplicationProcessor.processAll}} requeues it after the pass
> over the queue. {{ECUnderReplicationHandler.processDecommissioningIndexes}}
> only checks {{isReconstructionLimitReached()}}, so a decommissioning
> container that asks while a slot is free takes it. That container still has
> every index readable and could have used a 1-1 copy; the container with the
> missing index has no alternative to reconstruction.
> There are two routes to a free slot reaching the decommissioning container
> first:
> * A slot is released later in the same pass, after the missing index
> container was turned away. {{ReplicationManager.opCompleted}} is called by
> the container report handlers and by the pending op expiry service,
> concurrently with the under replicated processor.
> * A slot is released between passes, but the missing index container no
> longer sorts first. {{UnderReplicatedHealthResult.getWeightedRedundancy}}
> adds the requeue count, so each deferral moves it back by one. On rs-3-2 with
> one index lost it starts at 2 and a decommissioning container at 6; after 4
> deferrals they tie and the tie break on requeue count dequeues a
> decommissioning container that earlier passes never reached first (for
> example because the in flight replication limit,
> {{hdds.scm.replication.inflight.limit.factor}} 0.75 by default, ended those
> passes early). The queue is only rebuilt every
> {{hdds.scm.replication.thread.interval}} (300s), so 4 passes of 30s fit in
> one queue lifetime.
> h3. Trigger
> # Enable decommission reconstruction and set a global reconstruction limit
> (for example 1).
> # All slots are taken by running reconstructions, a container loses an index,
> and datanodes being decommissioned are loaded enough for SCM to prefer
> reconstruction for their containers.
> # The missing index container is rejected at the limit, then a running
> reconstruction completes or expires either before the decommissioning
> containers of that pass are processed, or between passes once the missing
> index container has been deferred often enough to sort behind a
> decommissioning container.
> No fault is needed. Both settings are off by default. Introduced by
> HDDS-15072 on top of HDDS-15071; master only, not in any release.
> h3. Impact
> * The container with the missing index stays at reduced redundancy for at
> least one more reconstruction (up to {{hdds.scm.replication.event.timeout}},
> 12 minutes by default, if the slot is only released by expiry).
> * It can repeat: on the first route whenever a release lands after the
> missing index container was dequeued in a pass, on the second route on every
> release between passes for as long as decommissioning containers sort ahead
> of it. How often this happens on a real cluster depends on pass duration and
> the decommission backlog; I did not measure it.
> * No data is lost by this alone.
> h3. Reproduction
> Both tests PASS on the unmodified source, deterministic, about 1 second.
> [^TestBugCR3DecommissionReconstructionTakesFreedSlot.java]
> ({{hadoop-hdds/server-scm}}) runs the real {{ReplicationManager}},
> {{ECUnderReplicationHandler}}, {{UnderReplicatedProcessor}},
> {{ReplicationQueue}} and {{ContainerReplicaPendingOps}} with mocked node
> manager, container manager and placement policy, EC rs-3-2, limit 1, and
> decommissioning datanodes whose command queue is full. The only thing the
> tests control is when the reconstruction holding the slot completes.
> * First route: {{opCompleted}} is called when the first decommissioning
> container of a pass is dequeued. In three consecutive passes the freed slot
> goes to decommissioning containers 203, 201 and 202 and the missing index
> container gets no reconstruction command.
> * Second route: the missing index container is deferred in 4 passes, a
> decommissioning container with requeue count 1 is added, and the slot is
> freed between passes. The decommissioning container is dequeued first and
> takes it ({{reconstruct commands by container = [1, 201]}}).
> A passing test means the defect is present. Not run on a cluster.
> h3. Patch
> [^CR-3-keep-freed-reconstruction-slot-for-deferred-missing-index.patch],
> against efe0c7eba1527530d1001f6517b733576fdc6d61. It also applies to master
> at c7bd3772207.
> When a missing index reconstruction is rejected while the global limit is
> reached, {{ECUnderReplicationHandler}} records it on {{ReplicationManager}},
> and decommission recovery treats it like a reached limit and falls back to
> 1-1 replication. The record holds for the rest of that pass and the whole
> next pass ({{UnderReplicatedProcessor.processAll}} starts each pass), so a
> slot freed later in the pass, or before the next pass, is left to the missing
> index container whatever their order in the queue. If the missing index
> container is deferred again, the hold is extended; once it is no longer
> deferred, decommission reconstruction resumes one pass later. Nothing changes
> when decommission reconstruction is off or no global limit is set. The
> fallback is counted in the existing
> {{ecReconstructionGlobalLimitReachedTotal}}, which therefore also counts
> fallbacks while a missing index reconstruction is waiting for a slot, and the
> debug log says so. I did not reserve part of the limit for missing index
> work, because that would reduce decommission reconstruction even when no
> index is missing.
> Covered by two cases in the existing {{TestReplicationManager}}:
> {{testDecommissionReconstructionLeavesFreedSlotToDeferredMissingIndex}} (slot
> freed during the pass; a further pass checks that decommission reconstruction
> resumes once nothing is waiting) and
> {{testDecommissionReconstructionDoesNotTakeFreeSlotAheadOfDeferredMissingIndex}}
> (slot freed between passes after 4 deferrals). Without the change both fail
> because the decommissioning container gets the slot; with it the missing
> index container does. With the change the replication package tests pass (547
> tests) and checkstyle is clean. With the patch both reproduction tests fail
> at their defect assertion: the missing index container gets the slot in both
> routes.
> A remaining gap: the hold expires one pass after the last deferral, so if a
> pass ends early (in flight replication limit) before it reaches a missing
> index container that sorts behind decommissioning ones, the following pass
> can again give a free slot to decommission work. Argued from the code, not
> run.
> Found by code review of the SCM EC reconstruction global limit and
> decommission reconstruction paths, as part of the TLA+ verification effort
> under HDDS-15926, on commit efe0c7eba1527530d1001f6517b733576fdc6d61. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
