Siyao Meng created HDDS-17372:
---------------------------------
Summary: Datanode keeps writing to the open block file of a
deleted container replica after the replica is recreated at the same path
Key: HDDS-17372
URL: https://issues.apache.org/jira/browse/HDDS-17372
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-reopen-block-file-for-recreated-container-replica.patch,
TestBugCR4StaleOpenBlockFileAfterContainerRecreate.java
h3. Mechanism
{{FilePerBlockStrategy.writeChunk}} gets its {{FileChannel}} from
{{OpenFiles}}, a cache keyed by the block file path that keeps an entry for up
to 10 minutes after its last access. Before it expires, an entry is closed only
by PutBlock with end of block (also when sent inside a WriteChunk),
PutSmallFile, FinalizeBlock and container reconciliation
({{finishWriteChunks}}, {{finalizeWriteChunk}}). Container delete
({{KeyValueHandler.deleteInternal}}), close, quasi close and import do not
touch it. If a block received chunks but no final PutBlock and its replica is
then deleted, the entry keeps the deleted file open. A new replica of the same
container created on the same volume has the same block file paths, so its
WriteChunks for that block hit the cached entry and go to the deleted file. The
new replica's block file is never created.
h3. Trigger
* A replica of container C on datanode D receives chunks of block B without a
final PutBlock, for example an EC reconstruction that fails midway through B.
* The replica is deleted (the EC coordinator force deletes its RECOVERING
targets after a failure) and a new replica of C is created on the same volume
of D, for example the next reconstruction attempt choosing D again.
* The new replica writes B within 10 minutes of the last write to B on the
deleted replica. For EC reconstruction SCM sends the next attempt only after
the pending operation expires ({{hdds.scm.replication.event.timeout}}, 12
minutes by default), so the first attempt must have taken at least about 2
minutes longer than the retry to reach B, counted from when SCM sent each
command, for example because it waited in the datanode's queue (bounded by the
datanode deadline, 6 minutes by default) or read from a slow source. Argued
from the code, not run.
* Container reconciliation of a replica that was deleted and imported back onto
the same volume can also write through the stale entry. Argued, not run.
h3. Impact
The new replica's chunks for B are written to the deleted file. For EC
reconstruction the final PutBlock for B then fails with UNABLE_TO_FIND_CHUNK
("Chunk file not found"), the datanode schedules a volume check
({{StorageVolumeUtil.onFailure}}), and the attempt fails and cleans up its
targets. The failing PutBlock closes the stale entry, so the next attempt is
not affected; the cost is one more failed reconstruction and a later restored
redundancy. For reconciliation the block file already exists in the imported
replica, so the PutBlock check passes while the repaired chunks are missing
from the file; the scan that follows reconciliation would find the checksum
mismatch and mark the replica unhealthy (argued, not run). In both cases the
data is not acknowledged as stored on the new replica and the source replicas
keep it; I found no sequence that loses committed data.
The path keyed cache exists since HDDS-2717 (1.0.0), but I found no writer that
reaches a recreated replica before EC reconstruction (1.3.0).
h3. Reproduction
[^TestBugCR4StaleOpenBlockFileAfterContainerRecreate.java]
({{hadoop-hdds/container-service}}) uses one volume and one {{KeyValueHandler}}
and only the calls the datanode makes: CreateContainer in RECOVERING state,
WriteChunk without PutBlock, force {{deleteContainer}}, CreateContainer again
(same path, asserted), WriteChunk, PutBlock. On unmodified source it fails with:
{noformat}
[PutBlock on the recreated replica: Chunk file not found:
<tmp>/hdds/<clusterId>/current/containerDir0/1/chunks/1.block]
expected: SUCCESS
but was: UNABLE_TO_FIND_CHUNK
{noformat}
What was run is this datanode level sequence, on macOS only. Not exercised: a
mini cluster with real EC reconstruction, SCM's retry timing, and
reconciliation.
h3. Suggested fix
Remember which replica opened each cached file and reopen the path when a
different replica writes to it. Closing the container's cached files on delete
was considered and not chosen: it needs a new {{ChunkManager}} method and hooks
in every place that removes a replica, and still misses a write in flight
during the delete. Checking the file's identity ({{fileKey}}) on every write
costs a {{stat}} per WriteChunk.
h3. Patch
[^CR-4-reopen-block-file-for-recreated-container-replica.patch], against
5fed99dab3c3204cd7bf1516074969a38d5eb00d. It also applies to master at
76b5c6947a2. It overlaps textually with two open pull requests that change the
same file: apache/ozone#11668 changes the line after the changed {{getChannel}}
call in {{writeChunk}}, and apache/ozone#11678 adds imports to
{{TestFilePerBlockStrategy}} in the same place. Whichever lands later needs a
trivial rebase.
Each cached {{OpenFile}} now keeps the {{KeyValueContainerData}} of the replica
that opened it. {{OpenFiles.getChannel}} receives the writing replica's data;
if the cached entry belongs to another replica object, it removes that entry
(the removal listener closes the old file) and opens the path again. A
recreated replica always has a new {{KeyValueContainerData}}, while all writes
of a live replica pass the same object, so an unchanged replica keeps reusing
its channel. A write from a replica already in DELETED state (one that was in
flight when its replica was deleted) is refused in that case instead of taking
over the new replica's file, with the error a write to a DELETED container gets
from {{KeyValueHandler}}. The cost is one reference comparison per WriteChunk.
Covered by
{{TestFilePerBlockStrategy#testWriteChunkAfterContainerRecreatedOnSameVolume}}:
write a chunk to a replica without PutBlock, delete the replica, create it
again at the same path, write two chunks, finish the block, and expect the file
to hold both chunks; then, with a new chunk open on the new replica, expect a
write from the deleted replica to be refused and the file to be unchanged.
Without the change it fails with "Chunk file not found". With the change
{{TestFilePerBlockStrategy}}, {{TestFilePerChunkStrategy}},
{{TestChunkManagerDummyImpl}}, {{TestKeyValueHandler}},
{{TestKeyValueStreamDataChannel}}, {{TestBlockManagerImpl}},
{{TestKeyValueContainer}} and {{TestContainerReconciliationWithMockDatanodes}}
pass (411 tests, 26 skipped by the suites' own parameters) and checkstyle is
clean. The attached reproduction passes with the change.
Found by code review of the datanode memory mapped chunk read path, as part of
the TLA+ verification effort under HDDS-15926, on commit
5fed99dab3c3204cd7bf1516074969a38d5eb00d. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
