Ismaël Mejía created AVRO-4300:
----------------------------------
Summary: [java] Bound collection allocation for zero-byte elements
when decoding arrays and maps
Key: AVRO-4300
URL: https://issues.apache.org/jira/browse/AVRO-4300
Project: Apache Avro
Issue Type: Sub-task
Components: java
Affects Versions: 1.12.1, 1.11.5
Reporter: Ismaël Mejía
Fix For: 1.12.2
When GenericDatumReader decodes an array or map it reads the block item count
from the stream and pre-allocates the full backing store up front
(newArray/newMap -> new Object[count]) before decoding any element.
The two existing guards do not bound a collection whose elements encode to zero
bytes:
- ensureAvailableCollectionBytes (AVRO-4241) only enforces count *
minBytesPerElement <= remainingBytes when minBytesPerElement > 0.
minBytesPerElement returns 0 for null and for self-referencing records, so the
check is skipped for those element types.
- SystemLimitException.checkMaxCollectionLength caps the count at
Integer.MAX_VALUE - 8, which is a JVM array-size ceiling rather than a memory
budget.
As a result, a small input declaring an array/map of many zero-byte elements
(e.g. {"type":"array","items":"null"} with a block count of 200,000,000) drives
a single large up-front allocation (new Object[200000000], ~1.6 GB) and
exhausts the heap. Confirmed against 1.11.5, 1.12.1 and current main.
Proposed fix: do not pre-size the backing collection to the untrusted block
count; clamp the initial capacity and let the collection grow as elements are
actually decoded, keeping the cumulative checkMaxCollectionLength cap as the
ultimate bound. Add tests and verify the behaviour with a zero-byte-element
reproduction. This completes AVRO-4277 by covering the Java SDK's residual gap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
