[
https://issues.apache.org/jira/browse/AVRO-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4300:
-------------------------------
Parent Issue: AVRO-4292 (was: AVRO-4277)
> [java] Bound array allocation when decoding on both the classic and fast
> readers (zero-byte elements + missing fast-reader available-bytes check)
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AVRO-4300
> URL: https://issues.apache.org/jira/browse/AVRO-4300
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.2
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When GenericDatumReader decodes an array it reads the block item count from
> the stream and pre-allocates the backing store (newArray -> new
> Object[count]) before decoding any element. Two independent gaps let a tiny
> payload drive an unbounded allocation and exhaust the heap:
> 1. Zero-byte elements (classic and fast reader).
> Elements whose schema encodes to zero bytes (null, a zero-length fixed, or a
> record with only zero-byte fields) consume no input, so
> ensureAvailableCollectionBytes (AVRO-4241) skips the check for them
> (minBytesPerElement == 0). The collection-length cap is Integer.MAX_VALUE-8,
> a JVM array-size ceiling rather than a memory budget. So an array such as
> {"type":"array","items":"null"} declaring a block count of 200,000,000 is a
> ~6 byte payload that allocates a 200M-slot array (~1.6 GB).
> 2. The fast reader never had the AVRO-4241 available-bytes guard at all.
> AVRO-4241 only modified the classic GenericDatumReader (and BinaryDecoder /
> Decoder / ValidatingDecoder); it never touched FastReaderBuilder, which is
> the default decode path (avro.io.fastread defaults to true). As a result, on
> the default reader even a non-zero-byte array such as array<long> or
> array<int> with a huge block count and no data pre-allocated new
> GenericData.Array<>((int) count) and exhausted the heap. Only the classic
> reader was protected.
> Fix (applied identically on both the classic and fast reader paths):
> - Add SystemLimitException.checkMaxCollectionAllocation, a heap-aware
> cumulative cap for zero-byte elements (default maxMemory()/4/8 elements,
> overridable via the org.apache.avro.limits.collectionItems.maxAllocation
> system property, mirroring the existing decompression limit).
> - Expose GenericDatumReader.ensureAvailableCollectionBytes and apply it,
> together with the zero-byte cap, before allocating each array block in
> FastReaderBuilder, so the fast and classic readers enforce identical guards.
> - Maps were already safe on both paths because each entry carries a string
> key of at least one byte (ensureAvailableMapBytes on the classic path; key
> reads consume bytes on the fast path).
> Verification: a full matrix of array<null|long|int> and map<null|long> at
> count 200,000,000 under -Xmx256m is rejected on both the fast and classic
> readers (SystemLimitException for zero-byte elements, EOFException otherwise)
> instead of OutOfMemoryError; legitimate collections within the limit still
> decode. Unit tests cover all ten collection/reader combinations plus
> cumulative multi-block and positive-decode cases. This completes AVRO-4277 by
> covering the Java SDK's residual gaps.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
