yihua opened a new pull request, #20321: URL: https://github.com/apache/hudi/pull/20321
### Describe the issue this Pull Request addresses closes #20320 The HFile format stores block sizes and key/value lengths as 4-byte integers, so a single block is bounded to less than 2 GiB even though the file, addressed by 8-byte offsets, is not. The native reader and writer never state that bound: a corrupt block header surfaces as `NegativeArraySizeException`, `ArithmeticException` or `ArrayIndexOutOfBoundsException` from inside the block code, or is accepted at parse time and fails later, and the writer's size accounting would overflow on a record or same-key run approaching 2 GiB instead of failing clearly. This also picks up the two review comments left on #20309 after it merged. ### Summary and Changelog - `HFileBlock`: validate the header on read and throw `HoodieIOException` naming the block type, buffer offset and sizes when a size is negative, `bytesPerChecksum` is zero, the on-disk size runs past the buffer, the uncompressed size exceeds the on-disk size of an uncompressed block, or the unpacked block would not fit one byte array. On write, `serialize` checks the compressed size plus checksum bytes against the same array bound in `long` math before writing the header. - `HFileWriterImpl`: compute the per-record size as a `long` and reject an append that would push a block's uncompressed data past `MAX_UNCOMPRESSED_DATA_BLOCK_BYTES` (2 MiB of headroom for the header, checksum bytes and deflate growth of incompressible data), before the writer's state is touched, with the message naming the two ways it can happen: one oversized record, or records sharing a key, which stay in one block. The flush threshold is capped at that limit. A package-private constructor lets tests lower the limit. - `HFileCursor` and `HFileDataBlock`: comments from the #20309 review, stating that the cursor offset is block start plus an offset within the uncompressed block, and why the `(int)` narrowing of the relative offset is safe. - Tests: `TestHfileBlock` parses a serialized data block, then corrupts each header field in turn and asserts the new error (each case fails without the guard: three were previously accepted at parse time, one threw `ArithmeticException`). `TestHFileWriter` lowers the block limit to 100 bytes, asserts that a same-key record and an oversized record are both rejected through `append`, and reads the closed file back to confirm the rejected records left no trace in the entry count, last key or average lengths. The `serialize` bound is not unit-tested since reaching it needs a 2 GiB block. ### Impact No behavior change for valid files; all 157 `hudi-io` tests and the HFile tests in `hudi-common` and `hudi-hadoop-common` pass, including the HBase-written fixtures. Corrupt block headers now fail with a descriptive error instead of an unrelated runtime exception or a deferred failure; a write that the format cannot represent fails at `append` or `serialize` with a clear message instead of producing an undecodable block. ### Risk Level low ### Documentation Update none; the format bounds are documented in #20319 ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
