[
https://issues.apache.org/jira/browse/AVRO-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4300:
-------------------------------
While implementing and verifying the fix it emerged that the default decode
path, the fast reader (FastReaderBuilder), never received the AVRO-4241
bytes-remaining guard - that change only touched the classic
GenericDatumReader. So on the default path an array of *non-zero-byte* elements
with a huge block count and no data (e.g. array<long> / array<int> with count
200,000,000) also pre-allocated the full backing array and exhausted the heap.
The fix therefore does two things on both reader paths (classic and fast):
1. Heap-aware cap for zero-byte elements (null, all-null record) via
SystemLimitException.checkMaxCollectionAllocation.
2. The AVRO-4241 available-bytes check (ensureAvailableCollectionBytes) for all
other element types.
Maps were already safe on both paths because each entry carries a string key of
at least one byte.
Verified against the PoC and a full matrix (array<null|long|int>,
map<null|long>) at count 200,000,000 under -Xmx256m: every combination is
rejected (SystemLimitException for zero-byte elements, EOFException otherwise)
on both the fast and classic readers, instead of OutOfMemoryError.
> [java] Bound collection allocation for zero-byte elements when decoding
> arrays and maps
> ---------------------------------------------------------------------------------------
>
> Key: AVRO-4300
> URL: https://issues.apache.org/jira/browse/AVRO-4300
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.12.2
>
>
> When GenericDatumReader decodes an array or map it reads the block item count
> from the stream and pre-allocates the full backing store up front
> (newArray/newMap -> new Object[count]) before decoding any element.
> The two existing guards do not bound a collection whose elements encode to
> zero bytes:
> - ensureAvailableCollectionBytes (AVRO-4241) only enforces count *
> minBytesPerElement <= remainingBytes when minBytesPerElement > 0.
> minBytesPerElement returns 0 for null and for self-referencing records, so
> the check is skipped for those element types.
> - SystemLimitException.checkMaxCollectionLength caps the count at
> Integer.MAX_VALUE - 8, which is a JVM array-size ceiling rather than a memory
> budget.
> As a result, a small input declaring an array/map of many zero-byte elements
> (e.g. {"type":"array","items":"null"} with a block count of 200,000,000)
> drives a single large up-front allocation (new Object[200000000], ~1.6 GB)
> and exhausts the heap. Confirmed against 1.11.5, 1.12.1 and current main.
> Proposed fix: do not pre-size the backing collection to the untrusted block
> count; clamp the initial capacity and let the collection grow as elements are
> actually decoded, keeping the cumulative checkMaxCollectionLength cap as the
> ultimate bound. Add tests and verify the behaviour with a zero-byte-element
> reproduction. This completes AVRO-4277 by covering the Java SDK's residual
> gap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
