[
https://issues.apache.org/jira/browse/AVRO-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4277:
-------------------------------
Affects Version/s: 1.12.1
1.11.5
(was: 1.13.0)
> Apply collection allocation limits when decoding arrays and maps across
> language SDKs
> -------------------------------------------------------------------------------------
>
> Key: AVRO-4277
> URL: https://issues.apache.org/jira/browse/AVRO-4277
> Project: Apache Avro
> Issue Type: Improvement
> Components: c, c++, perl, php, python
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Priority: Major
>
> When decoding an array or map, the binary decoder reads a block count from
> the input and uses it to drive the loop that builds the resulting collection.
> Several SDKs use this count directly to size or grow the collection without
> validating it against a configurable upper bound or against the amount of
> data actually available in the stream. For item types that encode to very few
> bytes (for example {{null}} or {{boolean}}), a small input can request a very
> large collection, leading to excessive memory allocation and poor failure
> behavior on malformed or truncated input.
> The Java SDK already centralizes these bounds via {{SystemLimitException}}
> and {{org.apache.avro.limits.collectionItems.maxLength}} (AVRO-3819). This
> umbrella tracks bringing the C, C++, Perl, PHP, and Python SDKs in line with
> that behavior so all SDKs handle unexpected block counts consistently and
> predictably.
> h3. Proposed approach
> * Validate each block count, and the running collection total across blocks,
> against a configurable maximum before allocating or growing the collection.
> * Reject negative or overflowing counts with a clear decode error instead of
> allocating.
> * Default the limit to a runtime-safe value; allow it to be tuned per SDK
> (mirroring the Java system property).
> * Add regression tests: a small input declaring a very large block count must
> fail with a bounded, well-defined error rather than exhausting memory.
> Subtasks: one per SDK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
