peterxcli opened a new pull request, #11302:
URL: https://github.com/apache/ozone/pull/11302
## What changes were proposed in this pull request?
Streaming ReadBlock collects checksums assuming uniform chunks, but flushes
can produce short chunks whose checksum intervals restart at different block
offsets. This replaces the split range bookkeeping with `BlockReadCursor` and
verifies each response against its overlapping chunks' stored checksums.
```text
Before After
readBlockImpl + ReadBlockComputation readBlockImpl + BlockReadCursor
| |
getChecksums(): assumes uniform chunks Read at chunk-relative boundaries
(can fail before sending a response) |
| Bytes + overlapping chunkInfoList
Bytes + flattened checksumData (original offsets and checksums)
| |
Client verifies one flat buffer Client verifies each chunk
|
Enqueue, then trim the seek prefix
```
With 4 bytes per checksum:
```text
Stored chunks: [ABC][DEFGHIJKL]
Stored checksum groups: [ABC][DEFG][HIJK][L]
Old flat client grouping*: [ABCD][EFGH][IJKL]
New client grouping: [ABC][DEFG][HIJK][L]
Read from offset 4 (E): receive from offset 3 (D), verify, then trim D.
* If a full response reaches the client.
```
The production path now handles variable chunks without `testVariableChunks`
or the legacy flattened checksum collector. Shared verification rejects missing
coverage and invalid boundaries, handles `NONE`, and preserves buffer
positions. Unexpected EOF fails before sending partial checksum data;
OUT_OF_RANGE retains stream error delivery and file cleanup.
**Protocol compatibility:** `ReadBlockResponseProto.checksumData` is
replaced with `chunkInfoList`; field 1 and its old name are reserved. This
intentionally requires matching clients and datanodes for streaming reads,
which are disabled by default. There is no legacy fallback. Ordinary chunk
reads and the stored format are unchanged.
## What is the link to the Apache JIRA
[HDDS-16258](https://issues.apache.org/jira/browse/HDDS-16258)
## How was this patch tested?
JDK 21: rebuilt affected modules and dependencies, including regenerated
protobuf sources. All 70 focused tests passed with no skips (`TestChecksum`,
`TestStreamBlockInputStream`, `TestKeyValueHandler`, `TestBlockReadCursor`, and
`TestStreamRead`); checkstyle and RAT passed. The integration test inspects
persisted chunks to confirm an uneven layout, then checks full and seek/range
reads through `StreamBlockInputStream` with verification enabled.
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
