[
https://issues.apache.org/jira/browse/AVRO-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4292:
-------------------------------
Description:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a
correspondingly large allocation before the shortfall is noticed.
AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
the source can report how many bytes remain, that a declared length does not
exceed the bytes actually available before allocating for it. This umbrella
applies the equivalent guard to the other language SDKs and extends it to
collections:
- Length-prefixed bytes/string: reject a declared length that exceeds the bytes
actually available (the JavaScript SDK already bounds these against its
in-memory buffer).
- Collections: reject a block whose element count could not be backed by the
bytes remaining, using the minimum on-wire size of the element schema so
zero-byte element types (e.g. null) are not falsely rejected.
- Zero-byte element collections: because zero-byte elements consume no input
they bypass the available-bytes check, so additionally cap the cumulative item
count of such blocks (e.g. array<null> with a huge block count), apply a
structural cap to all collections, and bound the array/map skip paths so
skipping a huge block cannot loop unboundedly. This subsumes the separate
collection-limit work previously tracked under AVRO-4277, which is now
superseded by this issue.
Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python (AVRO-4296),
Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), Java (AVRO-4300), and
JavaScript (AVRO-4301). The zero-byte-element cap, structural cap and skip
bounding are included for C, C++, Python, PHP, Perl, Java and JavaScript; C#
and Ruby currently carry the available-bytes checks plus negative-length and
overflow hardening.
was:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a
correspondingly large allocation before the shortfall is noticed.
AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
the source can report how many bytes remain, that a declared length does not
exceed the bytes actually available before allocating for it. This umbrella
applies the equivalent guard to the other language SDKs and extends it to
collections:
- Length-prefixed bytes/string: reject a declared length that exceeds the bytes
actually available (the JavaScript SDK already bounds against its in-memory
buffer).
- Collections: reject a block whose element count could not be backed by the
bytes remaining, using the minimum on-wire size of the element schema so
zero-byte element types (e.g. null) are not falsely rejected.
- Zero-byte element collections: because zero-byte elements consume no input
they bypass the available-bytes check, so additionally cap the cumulative item
count of such blocks (e.g. array<null> with a huge block count), apply a
structural cap to all collections, and bound the array/map skip paths so
skipping a huge block cannot loop unboundedly. This subsumes the separate
collection-limit work previously tracked under AVRO-4277, which is now
superseded by this issue.
Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python (AVRO-4296),
Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), and Java (AVRO-4300). The
zero-byte-element cap, structural cap and skip bounding are included for C,
C++, Python, PHP, Perl and Java; C# and Ruby currently carry the
available-bytes checks plus negative-length and overflow hardening.
> Bound allocation when decoding length-prefixed values and collections
> ---------------------------------------------------------------------
>
> Key: AVRO-4292
> URL: https://issues.apache.org/jira/browse/AVRO-4292
> Project: Apache Avro
> Issue Type: Bug
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> correspondingly large allocation before the shortfall is noticed.
> AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
> the source can report how many bytes remain, that a declared length does not
> exceed the bytes actually available before allocating for it. This umbrella
> applies the equivalent guard to the other language SDKs and extends it to
> collections:
> - Length-prefixed bytes/string: reject a declared length that exceeds the
> bytes actually available (the JavaScript SDK already bounds these against its
> in-memory buffer).
> - Collections: reject a block whose element count could not be backed by the
> bytes remaining, using the minimum on-wire size of the element schema so
> zero-byte element types (e.g. null) are not falsely rejected.
> - Zero-byte element collections: because zero-byte elements consume no input
> they bypass the available-bytes check, so additionally cap the cumulative
> item count of such blocks (e.g. array<null> with a huge block count), apply a
> structural cap to all collections, and bound the array/map skip paths so
> skipping a huge block cannot loop unboundedly. This subsumes the separate
> collection-limit work previously tracked under AVRO-4277, which is now
> superseded by this issue.
> Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python
> (AVRO-4296), Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), Java
> (AVRO-4300), and JavaScript (AVRO-4301). The zero-byte-element cap,
> structural cap and skip bounding are included for C, C++, Python, PHP, Perl,
> Java and JavaScript; C# and Ruby currently carry the available-bytes checks
> plus negative-length and overflow hardening.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
