[
https://issues.apache.org/jira/browse/HDDS-17372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17372:
------------------------------
Attachment: CR-4-reopen-block-file-for-recreated-container-replica.patch
> Datanode keeps writing to the open block file of a deleted container replica
> after the replica is recreated at the same path
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17372
> URL: https://issues.apache.org/jira/browse/HDDS-17372
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2,
> 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-4-reopen-block-file-for-recreated-container-replica.patch,
> TestBugCR4StaleOpenBlockFileAfterContainerRecreate.java
>
>
> h3. Mechanism
> {{FilePerBlockStrategy.writeChunk}} gets its {{FileChannel}} from
> {{OpenFiles}}, a cache keyed by the block file path that keeps an entry for
> up to 10 minutes after its last access. Before it expires, an entry is closed
> only by PutBlock with end of block (also when sent inside a WriteChunk),
> PutSmallFile, FinalizeBlock and container reconciliation
> ({{finishWriteChunks}}, {{finalizeWriteChunk}}). Container delete
> ({{KeyValueHandler.deleteInternal}}), close, quasi close and import do not
> touch it. If a block received chunks but no final PutBlock and its replica is
> then deleted, the entry keeps the deleted file open. A new replica of the
> same container created on the same volume has the same block file paths, so
> its WriteChunks for that block hit the cached entry and go to the deleted
> file. The new replica's block file is never created.
> h3. Trigger
> * A replica of container C on datanode D receives chunks of block B without a
> final PutBlock, for example an EC reconstruction that fails midway through B.
> * The replica is deleted (the EC coordinator force deletes its RECOVERING
> targets after a failure) and a new replica of C is created on the same volume
> of D, for example the next reconstruction attempt choosing D again.
> * The new replica writes B within 10 minutes of the last write to B on the
> deleted replica. For EC reconstruction SCM sends the next attempt only after
> the pending operation expires ({{hdds.scm.replication.event.timeout}}, 12
> minutes by default), so the first attempt must have taken at least about 2
> minutes longer than the retry to reach B, counted from when SCM sent each
> command, for example because it waited in the datanode's queue (bounded by
> the datanode deadline, 6 minutes by default) or read from a slow source.
> Argued from the code, not run.
> * Container reconciliation of a replica that was deleted and imported back
> onto the same volume can also write through the stale entry. Argued, not run.
> h3. Impact
> The new replica's chunks for B are written to the deleted file. For EC
> reconstruction the final PutBlock for B then fails with UNABLE_TO_FIND_CHUNK
> ("Chunk file not found"), the datanode schedules a volume check
> ({{StorageVolumeUtil.onFailure}}), and the attempt fails and cleans up its
> targets. The failing PutBlock closes the stale entry, so the next attempt is
> not affected; the cost is one more failed reconstruction and a later restored
> redundancy. For reconciliation the block file already exists in the imported
> replica, so the PutBlock check passes while the repaired chunks are missing
> from the file; the scan that follows reconciliation would find the checksum
> mismatch and mark the replica unhealthy (argued, not run). In both cases the
> data is not acknowledged as stored on the new replica and the source replicas
> keep it; I found no sequence that loses committed data.
> The path keyed cache exists since HDDS-2717 (1.0.0), but I found no writer
> that reaches a recreated replica before EC reconstruction (1.3.0).
> h3. Reproduction
> [^TestBugCR4StaleOpenBlockFileAfterContainerRecreate.java]
> ({{hadoop-hdds/container-service}}) uses one volume and one
> {{KeyValueHandler}} and only the calls the datanode makes: CreateContainer in
> RECOVERING state, WriteChunk without PutBlock, force {{deleteContainer}},
> CreateContainer again (same path, asserted), WriteChunk, PutBlock. On
> unmodified source it fails with:
> {noformat}
> [PutBlock on the recreated replica: Chunk file not found:
> <tmp>/hdds/<clusterId>/current/containerDir0/1/chunks/1.block]
> expected: SUCCESS
> but was: UNABLE_TO_FIND_CHUNK
> {noformat}
> What was run is this datanode level sequence, on macOS only. Not exercised: a
> mini cluster with real EC reconstruction, SCM's retry timing, and
> reconciliation.
> h3. Suggested fix
> Remember which replica opened each cached file and reopen the path when a
> different replica writes to it. Closing the container's cached files on
> delete was considered and not chosen: it needs a new {{ChunkManager}} method
> and hooks in every place that removes a replica, and still misses a write in
> flight during the delete. Checking the file's identity ({{fileKey}}) on every
> write costs a {{stat}} per WriteChunk.
> h3. Patch
> [^CR-4-reopen-block-file-for-recreated-container-replica.patch], against
> 5fed99dab3c3204cd7bf1516074969a38d5eb00d. It also applies to master at
> 76b5c6947a2. It overlaps textually with two open pull requests that change
> the same file: apache/ozone#11668 changes the line after the changed
> {{getChannel}} call in {{writeChunk}}, and apache/ozone#11678 adds imports to
> {{TestFilePerBlockStrategy}} in the same place. Whichever lands later needs a
> trivial rebase.
> Each cached {{OpenFile}} now keeps the {{KeyValueContainerData}} of the
> replica that opened it. {{OpenFiles.getChannel}} receives the writing
> replica's data; if the cached entry belongs to another replica object, it
> removes that entry (the removal listener closes the old file) and opens the
> path again. A recreated replica always has a new {{KeyValueContainerData}},
> while all writes of a live replica pass the same object, so an unchanged
> replica keeps reusing its channel. A write from a replica already in DELETED
> state (one that was in flight when its replica was deleted) is refused in
> that case instead of taking over the new replica's file, with the error a
> write to a DELETED container gets from {{KeyValueHandler}}. The cost is one
> reference comparison per WriteChunk.
> Covered by
> {{TestFilePerBlockStrategy#testWriteChunkAfterContainerRecreatedOnSameVolume}}:
> write a chunk to a replica without PutBlock, delete the replica, create it
> again at the same path, write two chunks, finish the block, and expect the
> file to hold both chunks; then, with a new chunk open on the new replica,
> expect a write from the deleted replica to be refused and the file to be
> unchanged. Without the change it fails with "Chunk file not found". With the
> change {{TestFilePerBlockStrategy}}, {{TestFilePerChunkStrategy}},
> {{TestChunkManagerDummyImpl}}, {{TestKeyValueHandler}},
> {{TestKeyValueStreamDataChannel}}, {{TestBlockManagerImpl}},
> {{TestKeyValueContainer}} and
> {{TestContainerReconciliationWithMockDatanodes}} pass (411 tests, 26 skipped
> by the suites' own parameters) and checkstyle is clean. The attached
> reproduction passes with the change.
> Found by code review of the datanode memory mapped chunk read path, as part
> of the TLA+ verification effort under HDDS-15926, on commit
> 5fed99dab3c3204cd7bf1516074969a38d5eb00d. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
