chungen0126 opened a new pull request, #10764:
URL: https://github.com/apache/ozone/pull/10764
## What changes were proposed in this pull request?
### Problem
Currently, the `getChecksums` method in `KeyValueHandler.java` fails or
returns incorrect checksums when chunks have variable sizes or are smaller than
bytesPerChunk.
The original code assumes all chunks (except the last one) have a fixed size
equal to `bytesPerChunk`. This assumption is incorrect when chunks have
different lengths, leading to wrong index calculations.
Crucially, when checksum verification is enabled, the DataNode must return
the actual ChunkInfo (which contains chunk lengths) to the client. Without this
metadata, the client cannot perceive the boundaries of each varying chunk,
making it impossible to correctly map the offsets and compute/verify the
checksums.
### Solution
This PR resolves the issue by tracking exact chunk boundaries on both the
server and client sides:
1. Server-Side (KeyValueHandler & Protocol Updates)
- Correct Indexing in getChecksums: Rewrote the checksum slice calculation
in KeyValueHandler#getChecksums. Instead of calculating chunk indexes using a
fixed bytesPerChunk division, we now iterate over chunks, keep track of the
cumulative offset (currentChunkOffset), and determine overlap intervals between
the read range and each chunk. This ensures we correctly map and collect the
target checksums.
- Send Chunk Info: Modified
ContainerCommandResponseBuilders#getReadBlockResponse and
KeyValueHandler#readBlockImpl to populate and return the chunkInfoList to the
client when performing a block read.
2. Client-Side (StreamBlockInputStream Checksum Verification)
Boundary-Aware Verification: In StreamBlockInputStream#onNext, if the
response contains chunkInfoList and checksum verification is enabled:
- Iterate through each chunk to identify the exact overlapping region
between the read block and the chunk boundaries.
- Calculate correct start and end checksum indexes using bytesPerChecksum
relative to that specific chunk's start.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15857
## How was this patch tested?
Added `testGetChecksumsWithVaryingChunkSizes` and
`testGetChecksumsWithSmallChunks` in `TestKeyValueHandler.java`.
Added `testSmallChunksWithLargeChecksum` in `TestStreamRead.java`.
CI: https://github.com/chungen0126/ozone/actions/runs/29343261204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
