[
https://issues.apache.org/jira/browse/AVRO-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4292:
-------------------------------
Description:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a
correspondingly large allocation before the shortfall is noticed.
AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
the source can report how many bytes remain, that a declared length does not
exceed the bytes actually available before allocating for it. This umbrella
applies the equivalent guard to the other language SDKs and extends it to
collections:
- Length-prefixed bytes/string: reject a declared length that exceeds the bytes
actually available (the JavaScript SDK already bounds these against its
in-memory buffer).
- Collections: reject a block whose element count could not be backed by the
bytes remaining, using the minimum on-wire size of the element schema so
zero-byte element types (e.g. null) are not falsely rejected.
- Zero-byte element collections: because zero-byte elements consume no input
they bypass the available-bytes check, so additionally cap the cumulative item
count of such blocks (e.g. array<null> with a huge block count), apply a
structural cap to all collections, and bound the array/map skip paths so
skipping a huge block cannot loop unboundedly. This subsumes the separate
collection-limit work previously tracked under AVRO-4277, which is now
superseded by this issue.
Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python (AVRO-4296),
Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), Java (AVRO-4300), and
JavaScript (AVRO-4301). All decoding SDKs now include the available-bytes
check, the zero-byte-element cap, the structural cap and the skip bounding;
both limits can be overridden (to a single value) via the
AVRO_MAX_COLLECTION_ITEMS environment variable (except the Java SDK, AVRO-4300,
which is configured through the system properties
org.apache.avro.limits.collectionItems.maxAllocation and
org.apache.avro.limits.collectionItems.maxLength rather than an environment
variable).
All SDKs additionally reject a block count whose absolute value cannot be
safely negated (INT64_MIN / the platform minimum), folding in AVRO-4275 (C) and
AVRO-4276 (C++).
was:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a
correspondingly large allocation before the shortfall is noticed.
AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
the source can report how many bytes remain, that a declared length does not
exceed the bytes actually available before allocating for it. This umbrella
applies the equivalent guard to the other language SDKs and extends it to
collections:
- Length-prefixed bytes/string: reject a declared length that exceeds the bytes
actually available (the JavaScript SDK already bounds these against its
in-memory buffer).
- Collections: reject a block whose element count could not be backed by the
bytes remaining, using the minimum on-wire size of the element schema so
zero-byte element types (e.g. null) are not falsely rejected.
- Zero-byte element collections: because zero-byte elements consume no input
they bypass the available-bytes check, so additionally cap the cumulative item
count of such blocks (e.g. array<null> with a huge block count), apply a
structural cap to all collections, and bound the array/map skip paths so
skipping a huge block cannot loop unboundedly. This subsumes the separate
collection-limit work previously tracked under AVRO-4277, which is now
superseded by this issue.
Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python (AVRO-4296),
Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), Java (AVRO-4300), and
JavaScript (AVRO-4301). All decoding SDKs now include the available-bytes
check, the zero-byte-element cap, the structural cap and the skip bounding;
both limits can be overridden (to a single value) via the
AVRO_MAX_COLLECTION_ITEMS environment variable.
All SDKs additionally reject a block count whose absolute value cannot be
safely negated (INT64_MIN / the platform minimum), folding in AVRO-4275 (C) and
AVRO-4276 (C++).
> Bound allocation when decoding length-prefixed values and collections
> ---------------------------------------------------------------------
>
> Key: AVRO-4292
> URL: https://issues.apache.org/jira/browse/AVRO-4292
> Project: Apache Avro
> Issue Type: Bug
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> correspondingly large allocation before the shortfall is noticed.
> AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
> the source can report how many bytes remain, that a declared length does not
> exceed the bytes actually available before allocating for it. This umbrella
> applies the equivalent guard to the other language SDKs and extends it to
> collections:
> - Length-prefixed bytes/string: reject a declared length that exceeds the
> bytes actually available (the JavaScript SDK already bounds these against its
> in-memory buffer).
> - Collections: reject a block whose element count could not be backed by the
> bytes remaining, using the minimum on-wire size of the element schema so
> zero-byte element types (e.g. null) are not falsely rejected.
> - Zero-byte element collections: because zero-byte elements consume no input
> they bypass the available-bytes check, so additionally cap the cumulative
> item count of such blocks (e.g. array<null> with a huge block count), apply a
> structural cap to all collections, and bound the array/map skip paths so
> skipping a huge block cannot loop unboundedly. This subsumes the separate
> collection-limit work previously tracked under AVRO-4277, which is now
> superseded by this issue.
> Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python
> (AVRO-4296), Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), Java
> (AVRO-4300), and JavaScript (AVRO-4301). All decoding SDKs now include the
> available-bytes check, the zero-byte-element cap, the structural cap and the
> skip bounding; both limits can be overridden (to a single value) via the
> AVRO_MAX_COLLECTION_ITEMS environment variable (except the Java SDK,
> AVRO-4300, which is configured through the system properties
> org.apache.avro.limits.collectionItems.maxAllocation and
> org.apache.avro.limits.collectionItems.maxLength rather than an environment
> variable).
> All SDKs additionally reject a block count whose absolute value cannot be
> safely negated (INT64_MIN / the platform minimum), folding in AVRO-4275 (C)
> and AVRO-4276 (C++).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
