[
https://issues.apache.org/jira/browse/AVRO-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102491#comment-18102491
]
ASF subversion and git services commented on AVRO-4323:
-------------------------------------------------------
Commit 23aad5216f874d0b7f3f2f43826cfb336747a970 in avro's branch
refs/heads/main from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=23aad5216f ]
AVRO-4323: [Java] Bound DataFileStream block size against available input
before allocating the block buffer (#3919)
* AVRO-4323: [Java] Bound DataFileStream block size against available input
When reading a data (container) file, DataFileStream validated the declared
block size only against the Integer range before allocating the block buffer.
For a malformed, corrupted, or truncated file the declared size can greatly
exceed the bytes actually present, so the reader eagerly allocated a large
buffer before reading any block byte.
Reject a declared block size that exceeds the number of bytes remaining in the
input when that count is known (byte-array- or known-length-stream-backed
decoders), so reading a malformed file fails fast with a clear IOException.
The check is skipped when the remaining count is unknown (-1).
* AVRO-4323: Address review: account for sync marker in block-size guard; close
reader in test
Include the trailing sync-marker length (DataFileConstants.SYNC_SIZE) in the
remaining-bytes check so a file truncated right before the sync marker is
rejected before allocating the block buffer, and use try-with-resources in the
regression test.
> [Java] Bound DataFileStream block size against available input before
> allocating the block buffer
> -------------------------------------------------------------------------------------------------
>
> Key: AVRO-4323
> URL: https://issues.apache.org/jira/browse/AVRO-4323
> Project: Apache Avro
> Issue Type: Improvement
> Components: java
> Reporter: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When reading an Avro data (container) file, {{DataFileStream}} reads each
> block's declared size as a long and currently validates it only against the
> {{Integer}} range before allocating the block {{byte[]}} buffer (in
> {{DataFileStream.DataBlock}}). For a malformed, corrupted, or truncated file,
> the declared block size can be much larger than the number of bytes actually
> present, so the reader eagerly allocates a very large buffer on the first
> {{hasNext()}}/{{next()}} call before any block byte has been read.
> We should make block reading more robust for malformed input:
> * validate the declared block size against the number of bytes actually
> remaining in the input (when that is known, e.g. a seekable/known-length
> source), and/or
> * route the block-size allocation through the existing {{SystemLimits}}
> configuration already used for bytes/collection sizing elsewhere in the
> reader,
> so that reading a malformed file fails fast with a clear {{IOException}}
> instead of attempting an oversized allocation.
> This can be reproduced with a small hand-crafted file whose block header
> declares a size near {{Integer.MAX_VALUE}} while carrying no block bytes. A
> valid file continues to read normally.
> Relevant code:
> * {{lang/java/avro/src/main/java/org/apache/avro/file/DataFileStream.java}} —
> {{hasNextBlock()}} (block-size read/validation) and {{DataBlock}} constructor
> ({{new byte[blockSize]}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
