Ismaël Mejía created AVRO-4277:
----------------------------------
Summary: Apply collection allocation limits when decoding arrays
and maps across language SDKs
Key: AVRO-4277
URL: https://issues.apache.org/jira/browse/AVRO-4277
Project: Apache Avro
Issue Type: Improvement
Components: perl, c, php, c++, python
Affects Versions: 1.13.0
Reporter: Ismaël Mejía
When decoding an array or map, the binary decoder reads a block count from the
input and uses it to drive the loop that builds the resulting collection.
Several SDKs use this count directly to size or grow the collection without
validating it against a configurable upper bound or against the amount of data
actually available in the stream. For item types that encode to very few bytes
(for example {{null}} or {{boolean}}), a small input can request a very large
collection, leading to excessive memory allocation and poor failure behavior on
malformed or truncated input.
The Java SDK already centralizes these bounds via {{SystemLimitException}} and
{{org.apache.avro.limits.collectionItems.maxLength}} (AVRO-3819). This umbrella
tracks bringing the C, C++, Perl, PHP, and Python SDKs in line with that
behavior so all SDKs handle unexpected block counts consistently and
predictably.
h3. Proposed approach
* Validate each block count, and the running collection total across blocks,
against a configurable maximum before allocating or growing the collection.
* Reject negative or overflowing counts with a clear decode error instead of
allocating.
* Default the limit to a runtime-safe value; allow it to be tuned per SDK
(mirroring the Java system property).
* Add regression tests: a small input declaring a very large block count must
fail with a bounded, well-defined error rather than exhausting memory.
Subtasks: one per SDK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
