[
https://issues.apache.org/jira/browse/HDDS-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147112#comment-18147112
]
Gargi Jaiswal commented on HDDS-16457:
--------------------------------------
While resolving two EC copies with one CLOSED and other UNHEALTHY both the
copies are kept as of today.
Same explanation as this HDDS-16458 applies here as well for EC copies to not
have any rules to delete UNHEALTHY/RECOVERING ones.
- EC duplicates can't be resolved like Ratis duplicates. Ratis uses the BCSID,
but EC writes have BCSID 0, so there is no way to tell which copy is newer or
more complete. With different replica indexes, the copies hold different data,
so deleting either one loses part of the stripe.
cc: [~Sammi] I think this is the issue we have for all EC containers failures
mid move, but not decided to how delete one of them.
> DiskBalancer EC move plus crash leaves a durable CLOSED plus UNHEALTHY two
> copy inconsistency
> ---------------------------------------------------------------------------------------------
>
> Key: HDDS-16457
> URL: https://issues.apache.org/jira/browse/HDDS-16457
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Major
> Attachments: HDDS-16457.001.patch
>
>
> h3. Finding
> An EC replica move plus a mid-move crash and restart leaves a durable
> two-copy inconsistency (CLOSED source plus an UNHEALTHY destination) that no
> DN-local mechanism reclaims, and the UNHEALTHY copy can win the in-memory
> slot and be reported to SCM. Triggerable through normal DiskBalancer EC moves
> with a single crash in the AtomicMove-to-import window; harm is a permanent
> orphaned replica (disk leak plus volume-accounting drift) and a spurious
> UNHEALTHY report, so bounded-but-durable rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC replica (replica index 1), CLOSED, resides on volume A. DiskBalancer
> schedules a move to volume B.
> 2. The mover copies the replica to a tmp dir on B, stages the copied
> .container as RECOVERING, and AtomicMoves the directory to B's final current/
> location. It has not yet run importContainer to reset the state back to
> CLOSED, and has not yet marked the source for delete.
> 3. The datanode crashes in that window. Volume B now holds a RECOVERING EC
> replica at its final location; volume A still holds the CLOSED source.
> 4. On restart, ContainerReader scans the dest volume first: the RECOVERING EC
> replica is marked UNHEALTHY and added. Then it scans the source volume: the
> CLOSED replica hits a duplicate, and resolveDuplicate for EC unconditionally
> "leaves both containers on disk". The result is two on-disk authorities of
> one replica index (CLOSED plus UNHEALTHY), the UNHEALTHY copy winning the
> in-memory slot, with no datanode-local reconciliation.
> h3. Root cause
> The datanode move stages the destination replica at its final current/
> location before it commits, and restart reconciliation
> ({{ContainerReader.resolveDuplicate}}) has no rule for EC duplicates: it
> leaves both copies on disk because EC replicas carry no BCSID and two copies
> may legitimately have different replica indexes. It does not special-case the
> mid-move leftover, where two copies share the same replica index and exactly
> one is CLOSED.
> h3. Suggested fix
> Attached patch [^HDDS-16457.001.patch] extends {{resolveDuplicate}} so that,
> for two EC copies of the SAME replica index where exactly one is CLOSED and
> the other is not (RECOVERING/UNHEALTHY), the CLOSED copy wins and the
> non-CLOSED duplicate is removed, mirroring the mismatched-state handling
> already used for Ratis containers. Copies with different replica indexes, or
> any other state combination, keep the existing leave-both behavior. A unit
> test drives the crash-then-restart scan order through the real
> ContainerReader and asserts only the CLOSED replica survives on disk and in
> memory.
> Note: this fix reconciles the durable two-authority state at restart. Making
> the destination replica non-authoritative until the move commits (for example
> staging it where cleanupTmpDir reclaims it after a mid-move crash) would be a
> complementary, more invasive hardening in the mover itself.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]