Chung-En Lee created HDDS-16258:
-----------------------------------
Summary: Support boundary-aware checksum verification for
variable-sized chunks in StreamBlockInputStream
Key: HDDS-16258
URL: https://issues.apache.org/jira/browse/HDDS-16258
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Chung-En Lee
*Description:*
*Background* As part of HDDS-15857, we fixed the server-side checksum
calculation in {{KeyValueHandler}} for variable-sized chunks and updated the
protocol to optionally include {{chunkInfoList}} in the
{{{}ReadBlockResponseProto{}}}. To keep the original patch focused and
manageable, the client-side verification logic has been split into this ticket.
*Problem* Currently, when a client actively triggers a flush or hsync, the
resulting chunk size may not perfectly align with {{{}bytesPerChunk{}}}. The
client-side {{StreamBlockInputStream}} incorrectly assumes all chunks (except
the last one) have a fixed size when verifying checksums.
Because of this fixed-size assumption, the client cannot correctly map the
offsets for variable-sized chunks. This leads to misaligned checksum
verification, causing false {{{}OzoneChecksumException{}}}s when processing
chunks that are variable-sized or smaller than {{{}bytesPerChunk{}}}.
*Proposed Solution* Update the client-side streaming read architecture to be
aware of exact chunk boundaries:
# *Boundary-Aware Verification:* In {{{}StreamBlockInputStream#onNext{}}}, if
the server response contains {{chunkInfoList}} and checksum verification is
enabled:
** Iterate through each chunk to identify the exact overlapping region between
the read block and the chunk boundaries.
** Calculate the correct start and end checksum indices using
{{bytesPerChecksum}} relative to that specific chunk's starting offset.
# *Backward Compatibility:* If {{chunkInfoList}} is absent from the response
(e.g., when reading from an older DataNode version), the client must gracefully
fall back to the legacy verification path ({{{}Checksum.verifyChecksum(data,
checksumData, 0){}}}) to avoid breaking upgrades.
*Testing*
* Update {{TestStreamBlockInputStream.java}} to validate boundary-aware
verification using {{{}chunkInfoList{}}}.
* Add integration tests in {{TestStreamRead.java}} (e.g.,
{{testSmallChunksWithLargeChecksum}} and
{{{}testShiftedChecksumBoundaryVerification{}}}) to cover varying/small chunk
sizes with larger {{{}bytesPerChecksum{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
