iemejia opened a new pull request, #3919:
URL: https://github.com/apache/avro/pull/3919
## What changes were proposed in this pull request?
When reading an Avro data (container) file, `DataFileStream` reads each
block's declared size as a `long` and validated it only against the `Integer`
range before allocating the block `byte[]` buffer (in
`DataFileStream.DataBlock`). For a malformed, corrupted, or truncated file, the
declared block size can be much larger than the number of bytes actually
present, so the reader eagerly allocated a very large buffer on the first
`hasNext()`/`next()` call before any block byte had been read.
This adds a check: when the number of bytes remaining in the input is known
(byte-array- or known-length-stream-backed decoders), a declared block size
larger than the bytes remaining is rejected with a clear `IOException` before
allocating. The check is skipped when the remaining count is unknown (`-1`), so
non-seekable streams are unaffected.
## How was this patch tested?
- New tests in `TestDataFileReader`:
- `oversizedBlockSizeIsRejectedBeforeAllocation` — a crafted file whose
block header declares a size near `Integer.MAX_VALUE` with no block bytes now
fails fast instead of attempting a large allocation.
- `validFileWithSingleRecordStillReads` — negative control confirming a
valid file still reads.
- Full `avro` module test suite passes.
### JIRA
- [AVRO-4323](https://issues.apache.org/jira/browse/AVRO-4323)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
