[
https://issues.apache.org/jira/browse/HDDS-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chung-En Lee reassigned HDDS-16258:
-----------------------------------
Assignee: Chung-En Lee
> Support boundary-aware checksum verification for variable-sized chunks in
> StreamBlockInputStream
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16258
> URL: https://issues.apache.org/jira/browse/HDDS-16258
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chung-En Lee
> Assignee: Chung-En Lee
> Priority: Major
>
> *Description:*
> *Background* As part of HDDS-15857, we fixed the server-side checksum
> calculation in {{KeyValueHandler}} for variable-sized chunks and updated the
> protocol to optionally include {{chunkInfoList}} in the
> {{{}ReadBlockResponseProto{}}}. To keep the original patch focused and
> manageable, the client-side verification logic has been split into this
> ticket.
> *Problem* Currently, when a client actively triggers a flush or hsync, the
> resulting chunk size may not perfectly align with {{{}bytesPerChunk{}}}. The
> client-side {{StreamBlockInputStream}} incorrectly assumes all chunks (except
> the last one) have a fixed size when verifying checksums.
> Because of this fixed-size assumption, the client cannot correctly map the
> offsets for variable-sized chunks. This leads to misaligned checksum
> verification, causing false {{{}OzoneChecksumException{}}}s when processing
> chunks that are variable-sized or smaller than {{{}bytesPerChunk{}}}.
> *Proposed Solution* Update the client-side streaming read architecture to be
> aware of exact chunk boundaries:
> # *Boundary-Aware Verification:* In {{{}StreamBlockInputStream#onNext{}}},
> if the server response contains {{chunkInfoList}} and checksum verification
> is enabled:
> ** Iterate through each chunk to identify the exact overlapping region
> between the read block and the chunk boundaries.
> ** Calculate the correct start and end checksum indices using
> {{bytesPerChecksum}} relative to that specific chunk's starting offset.
> # *Backward Compatibility:* If {{chunkInfoList}} is absent from the response
> (e.g., when reading from an older DataNode version), the client must
> gracefully fall back to the legacy verification path
> ({{{}Checksum.verifyChecksum(data, checksumData, 0){}}}) to avoid breaking
> upgrades.
> *Testing*
> * Update {{TestStreamBlockInputStream.java}} to validate boundary-aware
> verification using {{{}chunkInfoList{}}}.
> * Add integration tests in {{TestStreamRead.java}} (e.g.,
> {{testSmallChunksWithLargeChecksum}} and
> {{{}testShiftedChecksumBoundaryVerification{}}}) to cover varying/small chunk
> sizes with larger {{{}bytesPerChecksum{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
