[
https://issues.apache.org/jira/browse/AVRO-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía resolved AVRO-4277.
--------------------------------
Resolution: Duplicate
Superseded by AVRO-4292, which validates available bytes before allocating for
length-prefixed values and now also incorporates the collection block-count cap
that this umbrella tracked. Each language SDK fix has been folded into the
corresponding AVRO-4292 sub-task (AVRO-4293 c, AVRO-4294 c++, AVRO-4296 python,
AVRO-4298 php, AVRO-4299 perl), and the Java work (AVRO-4300) has been moved
under AVRO-4292 as well. The five sub-tasks here (AVRO-4278..AVRO-4282) were
already resolved as superseded, and their PRs were closed in favour of the
consolidated AVRO-4292 PRs. Closing this umbrella as superseded by AVRO-4292.
> Apply collection allocation limits when decoding arrays and maps across
> language SDKs
> -------------------------------------------------------------------------------------
>
> Key: AVRO-4277
> URL: https://issues.apache.org/jira/browse/AVRO-4277
> Project: Apache Avro
> Issue Type: Improvement
> Components: c, c++, java, perl, php, python
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.12.2
>
>
> When decoding an array or map, the binary decoder reads a block count from
> the input and uses it to drive the loop that builds the resulting collection.
> Several SDKs use this count directly to size or grow the collection without
> validating it against a configurable upper bound or against the amount of
> data actually available in the stream. For item types that encode to very few
> bytes (for example {{null}} or {{boolean}}), a small input can request a very
> large collection, leading to excessive memory allocation and poor failure
> behavior on malformed or truncated input.
> The Java SDK already centralizes these bounds via {{SystemLimitException}}
> and {{org.apache.avro.limits.collectionItems.maxLength}} (AVRO-3819). This
> umbrella tracks bringing the C, C++, Perl, PHP, and Python SDKs in line with
> that behavior so all SDKs handle unexpected block counts consistently and
> predictably.
> h3. Proposed approach
> * Validate each block count, and the running collection total across blocks,
> against a configurable maximum before allocating or growing the collection.
> * Reject negative or overflowing counts with a clear decode error instead of
> allocating.
> * Default the limit to a runtime-safe value; allow it to be tuned per SDK
> (mirroring the Java system property).
> * Add regression tests: a small input declaring a very large block count must
> fail with a bounded, well-defined error rather than exhausting memory.
> Subtasks: one per SDK.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
