[
https://issues.apache.org/jira/browse/HDDS-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-16458:
------------------------------------------
Assignee: Gargi Jaiswal
> DiskBalancer EC move crash before source delete mark leaves a permanent
> orphaned replica
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16458
> URL: https://issues.apache.org/jira/browse/HDDS-16458
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Major
> Attachments: HDDS-16458.001.patch
>
>
> h3. Finding
> After a DiskBalancer EC move, a crash (or swallowed mark failure) before the
> source is durably marked for delete drops the volatile pending-deletion
> queue, and restart reconciliation deliberately leaves both EC copies,
> producing a permanent orphaned replica untracked in memory. Triggerable via
> normal EC moves plus one crash; harm is a durable disk leak and under-counted
> volume accounting, bounded and not client-visible corruption.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. DiskBalancer selects a CLOSED EC replica for a move (there is no EC
> exclusion in the choosing policy).
> 2. The move imports the destination copy (CLOSED) and swaps the in-memory
> authority to it via updateContainer.
> 3. markContainerForDelete on the source fails and the exception is swallowed:
> the source's on-disk .container stays CLOSED, and the source is only enqueued
> in the volatile, non-persistent pendingDeletionContainers with a deferred
> deadline. The move is still reported successful.
> 4. The datanode crashes before the deadline, dropping the volatile queue with
> no durable record.
> 5. On restart, ContainerReader.resolveDuplicate for EC "leaves both
> containers on disk", so both CLOSED copies of the same replica index persist
> forever. Nothing reclaims the source. (Ratis self-heals via BCSID/state; EC
> does not.)
> h3. Root cause
> When markContainerForDelete fails, the source's DELETED intent is neither
> persisted (the mark, which writes it to the .container file, failed) nor
> durably recorded elsewhere: it lives only in the in-memory
> pendingDeletionContainers queue for the deferred-deletion window. A crash in
> that window loses it, and EC restart reconciliation does not remove a
> same-index CLOSED duplicate, so the source becomes a permanent orphan. The
> failure is swallowed with a comment that restart will handle it, which is not
> true for EC.
> h3. Suggested fix
> Attached patch [^HDDS-16458.001.patch] deletes the source replica
> synchronously when markContainerForDelete fails, instead of relying on the
> volatile queue plus a restart. At that point the imported destination is
> already the in-memory authority, so the source is non-authoritative and can
> be removed immediately via the existing deletion path; it is not also
> enqueued for deferred deletion. This removes the durable-duplicate window
> without depending on a persisted DELETED state or on restart reconciliation.
> A unit test drives a real DiskBalancer move with a failing source mark, then
> models a crash and restart, and asserts exactly one copy survives (and that
> the source is not left in the pending-deletion queue).
> Note: a crash landing between the in-memory authority swap and this
> synchronous cleanup is a narrower residual window; fully closing it would
> require persisting the source's delete intent durably at move-commit time (a
> larger change in the move protocol).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
