[
https://issues.apache.org/jira/browse/AVRO-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía reassigned AVRO-4300:
----------------------------------
Assignee: Ismaël Mejía
> [java] Bound collection allocation for zero-byte elements when decoding
> arrays and maps
> ---------------------------------------------------------------------------------------
>
> Key: AVRO-4300
> URL: https://issues.apache.org/jira/browse/AVRO-4300
> Project: Apache Avro
> Issue Type: Sub-task
> Components: java
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.12.2
>
>
> When GenericDatumReader decodes an array or map it reads the block item count
> from the stream and pre-allocates the full backing store up front
> (newArray/newMap -> new Object[count]) before decoding any element.
> The two existing guards do not bound a collection whose elements encode to
> zero bytes:
> - ensureAvailableCollectionBytes (AVRO-4241) only enforces count *
> minBytesPerElement <= remainingBytes when minBytesPerElement > 0.
> minBytesPerElement returns 0 for null and for self-referencing records, so
> the check is skipped for those element types.
> - SystemLimitException.checkMaxCollectionLength caps the count at
> Integer.MAX_VALUE - 8, which is a JVM array-size ceiling rather than a memory
> budget.
> As a result, a small input declaring an array/map of many zero-byte elements
> (e.g. {"type":"array","items":"null"} with a block count of 200,000,000)
> drives a single large up-front allocation (new Object[200000000], ~1.6 GB)
> and exhausts the heap. Confirmed against 1.11.5, 1.12.1 and current main.
> Proposed fix: do not pre-size the backing collection to the untrusted block
> count; clamp the initial capacity and let the collection grow as elements are
> actually decoded, keeping the cumulative checkMaxCollectionLength cap as the
> ultimate bound. Add tests and verify the behaviour with a zero-byte-element
> reproduction. This completes AVRO-4277 by covering the Java SDK's residual
> gap.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
