[
https://issues.apache.org/jira/browse/AVRO-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4292:
-------------------------------
Description:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a
correspondingly large allocation before the shortfall is noticed.
AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
the source can report how many bytes remain, that a declared length does not
exceed the bytes actually available before allocating for it. This umbrella
applies the equivalent guard to the other language SDKs and extends it to
collections:
- Length-prefixed bytes/string: reject a declared length that exceeds the bytes
actually available (the JavaScript SDK already bounds against its in-memory
buffer).
- Collections: reject a block whose element count could not be backed by the
bytes remaining, using the minimum on-wire size of the element schema so
zero-byte element types (e.g. null) are not falsely rejected.
- Zero-byte element collections: because zero-byte elements consume no input
they bypass the available-bytes check, so additionally cap the cumulative item
count of such blocks (e.g. array<null> with a huge block count), apply a
structural cap to all collections, and bound the array/map skip paths so
skipping a huge block cannot loop unboundedly. This subsumes the separate
collection-limit work previously tracked under AVRO-4277, which is now
superseded by this issue.
Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python (AVRO-4296),
Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), and Java (AVRO-4300). The
zero-byte-element cap, structural cap and skip bounding are included for C,
C++, Python, PHP, Perl and Java; C# and Ruby currently carry the
available-bytes checks plus negative-length and overflow hardening.
was:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data. A malicious or truncated input can declare a very large length
while carrying little or no actual data, which causes a correspondingly large
buffer to be allocated before the shortfall is noticed.
AVRO-4241 addressed this in the Java SDK by verifying, when the source can
report how many bytes remain, that the declared length does not exceed the
bytes actually available before allocating for it. This is an umbrella issue to
apply the equivalent check to the other language SDKs.
The JavaScript SDK already performs an equivalent bounds check against its
in-memory buffer. Sub-tasks cover C, C++, C#, Python, Ruby, PHP, and Perl.
Summary: Bound allocation when decoding length-prefixed values and
collections (was: Validate available bytes before allocating for
length-prefixed values)
> Bound allocation when decoding length-prefixed values and collections
> ---------------------------------------------------------------------
>
> Key: AVRO-4292
> URL: https://issues.apache.org/jira/browse/AVRO-4292
> Project: Apache Avro
> Issue Type: Bug
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> correspondingly large allocation before the shortfall is noticed.
> AVRO-4241 addressed the length-prefix case in the Java SDK by verifying, when
> the source can report how many bytes remain, that a declared length does not
> exceed the bytes actually available before allocating for it. This umbrella
> applies the equivalent guard to the other language SDKs and extends it to
> collections:
> - Length-prefixed bytes/string: reject a declared length that exceeds the
> bytes actually available (the JavaScript SDK already bounds against its
> in-memory buffer).
> - Collections: reject a block whose element count could not be backed by the
> bytes remaining, using the minimum on-wire size of the element schema so
> zero-byte element types (e.g. null) are not falsely rejected.
> - Zero-byte element collections: because zero-byte elements consume no input
> they bypass the available-bytes check, so additionally cap the cumulative
> item count of such blocks (e.g. array<null> with a huge block count), apply a
> structural cap to all collections, and bound the array/map skip paths so
> skipping a huge block cannot loop unboundedly. This subsumes the separate
> collection-limit work previously tracked under AVRO-4277, which is now
> superseded by this issue.
> Sub-tasks: C (AVRO-4293), C++ (AVRO-4294), C# (AVRO-4295), Python
> (AVRO-4296), Ruby (AVRO-4297), PHP (AVRO-4298), Perl (AVRO-4299), and Java
> (AVRO-4300). The zero-byte-element cap, structural cap and skip bounding are
> included for C, C++, Python, PHP, Perl and Java; C# and Ruby currently carry
> the available-bytes checks plus negative-length and overflow hardening.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
