yihua opened a new issue, #20320: URL: https://github.com/apache/hudi/issues/20320
The HFile format stores block sizes and key/value lengths as 4-byte integers, so a single block is bounded to less than 2 GiB even though the file itself, addressed by 8-byte offsets, is not (see #20308 for the file-size side). The native reader and writer in `hudi-io` do not state that bound anywhere: - On read, `HFileBlock` takes the header sizes at face value. A corrupt or truncated header with a negative size, a zero `bytesPerChecksum`, or an on-disk size running past the buffer surfaces as `NegativeArraySizeException`, `ArithmeticException` or `ArrayIndexOutOfBoundsException` from deep inside the block code, with no indication that the block header is the problem. - On write, `HFileWriterImpl.append` accumulates the record size in an `int` and only flushes when a block passes the configured block size. A single record of about 2 GiB, or a run of records sharing one key (which the writer keeps in a single block), would overflow that accounting and produce a block the header cannot describe, instead of failing with a clear error. Neither case occurs with metadata table payloads today, so this is hardening rather than a live bug: reject these inputs up front with an error that names the block and the sizes involved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
