Chu Cheng Li created HDDS-16731:
-----------------------------------
Summary: [hsync] Container scanner and reconciliation ignore
last_chunk_info rows
Key: HDDS-16731
URL: https://issues.apache.org/jira/browse/HDDS-16731
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Chu Cheng Li
h3. Background
A client with {{ozone.client.incremental.chunk.list=true}} sends an incremental
chunk list in PutBlock. The datanode keeps the last partial chunk of the block
in the {{last_chunk_info}} table. The other chunks are in the {{block_data}}
table. {{getBlock}} reads both tables and merges the chunk lists.
The datanode moves the last chunk to {{block_data}} when the chunk is full, at
the end of the block, or when FinalizeBlock merges it. Thus, a closed container
can keep {{last_chunk_info}} rows:
* The client does not end the block, for example because the client fails. Then
the container closes before lease recovery. FinalizeBlock fails on a closed
container. Thus, no merge occurs.
* A datanode with release 2.0.0 to 2.2.1 finalized the block. These releases do
not persist the merge (HDDS-16730).
h3. Problem
Some code for closed containers reads only the {{block_data}} table:
* The container scanner does not verify the checksum of the last chunk.
* The container merkle tree does not include the last chunk. Thus,
reconciliation cannot find or repair a bad last chunk.
* When reconciliation repairs a chunk of the block, it writes the full chunk
list to {{block_data}}. It does not delete the {{last_chunk_info}} row. Thus,
the next {{getBlock}} adds the last chunk a second time and fails with
{{IllegalStateException: chunk offset does not match}}.
h3. Effect
* The datanode does not find corruption in the last chunk of these blocks.
* After reconciliation repairs such a block, the block is unreadable on that
datanode. Clients must read from other datanodes. Other datanodes also cannot
reconcile the block from this datanode.
h3. Possible fix
* Merge the {{last_chunk_info}} rows into {{block_data}} when the datanode
closes or quasi-closes a container.
* For containers that are already closed, make the scanner and the merkle tree
use the merged block data. Make {{putBlockForClosedContainer}} delete the
{{last_chunk_info}} row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
