Ismaël Mejía created AVRO-4323:
----------------------------------
Summary: [Java] Bound DataFileStream block size against available
input before allocating the block buffer
Key: AVRO-4323
URL: https://issues.apache.org/jira/browse/AVRO-4323
Project: Apache Avro
Issue Type: Improvement
Components: java
Reporter: Ismaël Mejía
When reading an Avro data (container) file, {{DataFileStream}} reads each
block's declared size as a long and currently validates it only against the
{{Integer}} range before allocating the block {{byte[]}} buffer (in
{{DataFileStream.DataBlock}}). For a malformed, corrupted, or truncated file,
the declared block size can be much larger than the number of bytes actually
present, so the reader eagerly allocates a very large buffer on the first
{{hasNext()}}/{{next()}} call before any block byte has been read.
We should make block reading more robust for malformed input:
* validate the declared block size against the number of bytes actually
remaining in the input (when that is known, e.g. a seekable/known-length
source), and/or
* route the block-size allocation through the existing {{SystemLimits}}
configuration already used for bytes/collection sizing elsewhere in the reader,
so that reading a malformed file fails fast with a clear {{IOException}}
instead of attempting an oversized allocation.
This can be reproduced with a small hand-crafted file whose block header
declares a size near {{Integer.MAX_VALUE}} while carrying no block bytes. A
valid file continues to read normally.
Relevant code:
* {{lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java}} —
{{hasNextBlock()}} (block-size read/validation) and {{DataBlock}} constructor
({{new byte[blockSize]}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
