Siyao Meng created HDDS-17559:
---------------------------------
Summary: EC decommission reconstruction can take a global
reconstruction slot while a container with a missing index waits for one
Key: HDDS-17559
URL: https://issues.apache.org/jira/browse/HDDS-17559
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-3-keep-freed-reconstruction-slot-for-deferred-missing-index.patch,
TestBugCR3DecommissionReconstructionTakesFreedSlot.java
h3. Mechanism
With {{hdds.scm.replication.decommission.ec.reconstruction.enabled=true}} and
{{hdds.scm.replication.reconstruction.global.limit}} above 0, both missing
index recovery and decommission recovery of EC containers compete for the same
global reconstruction slots, and a slot goes to whichever container asks first.
When a container with a missing index is dequeued while the limit is reached,
{{ReplicationManager.sendThrottledReconstructionCommand}} rejects it and
{{UnhealthyReplicationProcessor.processAll}} requeues it after the pass over
the queue. {{ECUnderReplicationHandler.processDecommissioningIndexes}} only
checks {{isReconstructionLimitReached()}}, so a decommissioning container that
asks while a slot is free takes it. That container still has every index
readable and could have used a 1-1 copy; the container with the missing index
has no alternative to reconstruction.
There are two routes to a free slot reaching the decommissioning container
first:
* A slot is released later in the same pass, after the missing index container
was turned away. {{ReplicationManager.opCompleted}} is called by the container
report handlers and by the pending op expiry service, concurrently with the
under replicated processor.
* A slot is released between passes, but the missing index container no longer
sorts first. {{UnderReplicatedHealthResult.getWeightedRedundancy}} adds the
requeue count, so each deferral moves it back by one. On rs-3-2 with one index
lost it starts at 2 and a decommissioning container at 6; after 4 deferrals
they tie and the tie break on requeue count dequeues a decommissioning
container that earlier passes never reached first (for example because the in
flight replication limit, {{hdds.scm.replication.inflight.limit.factor}} 0.75
by default, ended those passes early). The queue is only rebuilt every
{{hdds.scm.replication.thread.interval}} (300s), so 4 passes of 30s fit in one
queue lifetime.
h3. Trigger
# Enable decommission reconstruction and set a global reconstruction limit (for
example 1).
# All slots are taken by running reconstructions, a container loses an index,
and datanodes being decommissioned are loaded enough for SCM to prefer
reconstruction for their containers.
# The missing index container is rejected at the limit, then a running
reconstruction completes or expires either before the decommissioning
containers of that pass are processed, or between passes once the missing index
container has been deferred often enough to sort behind a decommissioning
container.
No fault is needed. Both settings are off by default. Introduced by HDDS-15072
on top of HDDS-15071; master only, not in any release.
h3. Impact
* The container with the missing index stays at reduced redundancy for at least
one more reconstruction (up to {{hdds.scm.replication.event.timeout}}, 12
minutes by default, if the slot is only released by expiry).
* It can repeat: on the first route whenever a release lands after the missing
index container was dequeued in a pass, on the second route on every release
between passes for as long as decommissioning containers sort ahead of it. How
often this happens on a real cluster depends on pass duration and the
decommission backlog; I did not measure it.
* No data is lost by this alone.
h3. Reproduction
Both tests PASS on the unmodified source, deterministic, about 1 second.
[^TestBugCR3DecommissionReconstructionTakesFreedSlot.java]
({{hadoop-hdds/server-scm}}) runs the real {{ReplicationManager}},
{{ECUnderReplicationHandler}}, {{UnderReplicatedProcessor}},
{{ReplicationQueue}} and {{ContainerReplicaPendingOps}} with mocked node
manager, container manager and placement policy, EC rs-3-2, limit 1, and
decommissioning datanodes whose command queue is full. The only thing the tests
control is when the reconstruction holding the slot completes.
* First route: {{opCompleted}} is called when the first decommissioning
container of a pass is dequeued. In three consecutive passes the freed slot
goes to decommissioning containers 203, 201 and 202 and the missing index
container gets no reconstruction command.
* Second route: the missing index container is deferred in 4 passes, a
decommissioning container with requeue count 1 is added, and the slot is freed
between passes. The decommissioning container is dequeued first and takes it
({{reconstruct commands by container = [1, 201]}}).
A passing test means the defect is present. Not run on a cluster.
h3. Patch
[^CR-3-keep-freed-reconstruction-slot-for-deferred-missing-index.patch],
against efe0c7eba1527530d1001f6517b733576fdc6d61. It also applies to master at
c7bd3772207.
When a missing index reconstruction is rejected while the global limit is
reached, {{ECUnderReplicationHandler}} records it on {{ReplicationManager}},
and decommission recovery treats it like a reached limit and falls back to 1-1
replication. The record holds for the rest of that pass and the whole next pass
({{UnderReplicatedProcessor.processAll}} starts each pass), so a slot freed
later in the pass, or before the next pass, is left to the missing index
container whatever their order in the queue. If the missing index container is
deferred again, the hold is extended; once it is no longer deferred,
decommission reconstruction resumes one pass later. Nothing changes when
decommission reconstruction is off or no global limit is set. The fallback is
counted in the existing {{ecReconstructionGlobalLimitReachedTotal}}, which
therefore also counts fallbacks while a missing index reconstruction is waiting
for a slot, and the debug log says so. I did not reserve part of the limit for
missing index work, because that would reduce decommission reconstruction even
when no index is missing.
Covered by two cases in the existing {{TestReplicationManager}}:
{{testDecommissionReconstructionLeavesFreedSlotToDeferredMissingIndex}} (slot
freed during the pass; a further pass checks that decommission reconstruction
resumes once nothing is waiting) and
{{testDecommissionReconstructionDoesNotTakeFreeSlotAheadOfDeferredMissingIndex}}
(slot freed between passes after 4 deferrals). Without the change both fail
because the decommissioning container gets the slot; with it the missing index
container does. With the change the replication package tests pass (547 tests)
and checkstyle is clean. With the patch both reproduction tests fail at their
defect assertion: the missing index container gets the slot in both routes.
A remaining gap: the hold expires one pass after the last deferral, so if a
pass ends early (in flight replication limit) before it reaches a missing index
container that sorts behind decommissioning ones, the following pass can again
give a free slot to decommission work. Argued from the code, not run.
Found by code review of the SCM EC reconstruction global limit and decommission
reconstruction paths, as part of the TLA+ verification effort under HDDS-15926,
on commit efe0c7eba1527530d1001f6517b733576fdc6d61. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing. The attached patch
is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
