[
https://issues.apache.org/jira/browse/HDDS-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shuo Huang reassigned HDDS-16452:
---------------------------------
Assignee: Shuo Huang
> Inter SCM checkpoint served during an apply window ships data ahead of the
> durable index, diverging replicas
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16452
> URL: https://issues.apache.org/jira/browse/HDDS-16452
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Shuo Huang
> Priority: Critical
> Attachments: HDDS-16452.001.patch
>
>
> h3. Finding
> A checkpoint served by InterSCMGrpcService.download during an apply window
> ships data at index i under stale durable index i-1; a follower installs it
> and durably holds data ahead of its recorded index (CheckpointConsistent
> violated across replicas), then re-applies the entry on catch-up. Reproduced
> through the real concurrent apply/download APIs on a real store. For
> non-idempotent SCM transactions (e.g. deleted-block accounting, corroborated
> by a separate tracked issue) this diverges across SCM replicas with no
> automatic recovery; idempotent transactions self-heal via replay.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-ha-tx-buffer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/018-scm-ha-tx-buffer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-ha-tx-buffer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: two threads run concurrently on a leader SCM, the Ratis apply
> thread in SCMStateMachine.applyTransaction and the InterSCM gRPC download
> thread in InterSCMGrpcService.download.
> 1. The apply thread opens an apply window: beginApplyingTransaction, then
> process() stages transaction i's data via addToBuffer (the batch now holds
> data i), but has not yet reached updateLatestTrxInfo, so the durable
> transaction index is still i-1.
> 2. A follower requests a checkpoint. InterSCMGrpcService.download calls the
> unguarded SCMHADBTransactionBuffer.flush(), which takes the write lock and
> commits the current batch atomically: {data i, transactionInfo i-1}. The DB
> now holds data i under the stale index i-1.
> 3. download serves that checkpoint DB. The follower installs it
> (notifyInstallSnapshotFromLeader, reinitialize, transactionBuffer.init()) and
> reloads the persisted index i-1 while its DB already contains data i
> (CheckpointConsistent / DurableIndexMatchesData violated), then re-applies
> entry i on catch-up.
> For idempotent SCM transactions the replay self-heals, so the harm is masked.
> For a non-idempotent replicated transaction (e.g. deleted-block accounting)
> the re-apply diverges across SCM replicas with no automatic recovery.
> Reproduced with a focused test that drives the real download() against a real
> store and buffer with an apply in progress: on the unguarded flush() the
> store is left holding data i under index i-1; the assertion fails without the
> fix and passes with it.
> h3. Root cause
> hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcService.java:
> download() calls SCMHADBTransactionBuffer.flush(), which commits whatever is
> in the current batch under the current (not yet advanced) transaction index.
> flush() takes only the write lock; unlike flushIfNeeded it does not check
> applyingTransactions, so it is not deferred during an apply. The apply window
> is not atomic (addToBuffer releases the read lock on return,
> updateLatestTrxInfo holds no lock), so a flush can land between staging data
> i and advancing the index. HDDS-15065 added the applyingTransactions guard
> and moved the periodic/monitor flush onto it, but the download caller was
> left on the unguarded flush().
> h3. Suggested fix
> Route the download flush through the existing guarded flushIfNeeded(0)
> instead of flush(). While a transaction apply is in progress it defers
> (returns without committing), so the checkpoint is served from the last self
> consistent durable state and the follower catches the missing transactions up
> through the Ratis log. When no apply is in progress it flushes pending data
> as before. This reuses the same guard HDDS-15065 established for the periodic
> path. See [^HDDS-16452.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
