peterxcli opened a new pull request, #11416: URL: https://github.com/apache/ozone/pull/11416
## What changes were proposed in this pull request? A block written with an incremental chunk list keeps its last partial chunk in the `last_chunk_info` table. `getBlock` merges that row into the chunk list from `block_data` when it reads the block. A closed container can still have such rows, for example when the writer stopped before the end of the block and the container closed before lease recovery, or when a datanode on 2.0.0 to 2.2.1 finalized the block ([HDDS-16730](https://issues.apache.org/jira/browse/HDDS-16730), #11414). Some datanode code for closed containers reads only `block_data`: - The container scanner never verifies the last chunk, and the container merkle tree (built by the scanner, or from metadata when the container closes) leaves it out. So reconciliation can neither find nor repair a bad last chunk. - When reconciliation repairs a chunk of such a block, it writes the full chunk list to `block_data` but keeps the `last_chunk_info` row. The next `getBlock` then adds the last chunk a second time and fails with `IllegalStateException: chunk offset does not match`. This PR: 1. Makes `KeyValueContainerCheck` and `KeyValueHandler#updateAndGetContainerChecksumFromMetadata` use `getCompleteBlockData`, so the scanner and the merkle tree cover the same chunks that a read returns. 2. Makes `BlockManagerImpl#putBlockForClosedContainer` delete the `last_chunk_info` row in the same batch, because the block data it writes already has the full chunk list. 3. When the merge on read finds that `block_data` already has the last chunk, it now skips the stale `last_chunk_info` row instead of failing. This makes blocks that older versions already broke this way readable again. It also keeps the scanner, which now does this merge, from hitting the exception: an unchecked exception stops the background scanner thread. The Jira also suggests merging the rows into `block_data` when a container closes. That is not needed once the readers merge, and it would not help containers that are already closed, so this PR leaves it out. After the upgrade, the next scan of an affected container includes the last chunks, so its data checksum changes. Replicas on different versions can show different data checksums until all of them are upgraded and rescanned. Reconciliation only starts from the admin command, so this does not start repairs by itself. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-16731 ## How was this patch tested? - New `TestContainerReconciliationWithMockDatanodes#testContainerReconciliationWithLastChunkInfo`. Three replicas have blocks that keep the last chunk in `last_chunk_info`. The test checks that the checksum built when the container closes matches the one from a scan. It corrupts the last chunk on one replica and the first chunk on another, then checks that the scan finds both, that reconciliation repairs them, that the repaired replicas no longer have the `last_chunk_info` row, and that every replica reads the full block. Without this change, the scan misses the corrupted last chunk. - New `TestBlockManagerImpl#testPutBlockForClosedContainerWithLastChunkInfo`. Reading a block that older reconciliation left with a stale `last_chunk_info` row works, and `putBlockForClosedContainer` deletes the row. Without this change, the read throws `IllegalStateException: chunk offset does not match`. - Ran `TestBlockManagerImpl`, `TestContainerReconciliationWithMockDatanodes`, `TestKeyValueContainerCheck`, `TestKeyValueHandler`, `TestFilePerBlockStrategy`, `TestReconcileChunksPerBlockHoleBcsId`, `TestKeyValueHandlerWithUnhealthyContainer`, `TestKeyValueBlockIterator`, `TestBackgroundContainerDataScanner` and `TestOnDemandContainerScanner` locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
