[
https://issues.apache.org/jira/browse/AVRO-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4297:
-------------------------------
Description:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a large
allocation before the shortfall is noticed. When the source can report how many
bytes remain, reject a declared length (or a collection block count) that
exceeds the bytes actually available before allocating for it. Companion to
AVRO-4241 (Java).
BinaryDecoder#bytes_remaining reports the bytes still readable; #read consults
it directly for a declared length above a threshold, while
DatumReader#read_array/#read_map consult it via min_bytes_per_element.
ensure_collection_available tracks the cumulative count and enforces the
limits, and the skip path (skip_blocks, used by skip_array/skip_map) is bounded
the same way. Rejections raise the dedicated Avro::IO::CollectionSizeError (a
subclass of AvroError).
Zero-byte elements (null, or a record with only zero-byte fields) consume no
input, so the available-bytes check cannot bound their count: a tiny payload
such as {"type":"array","items":"null"} declaring a block count of 200,000,000
would otherwise drive an unbounded allocation. In addition to the
available-bytes check this therefore caps the cumulative count of zero-byte
elements (default 10,000,000), applies a structural cap (Integer.MAX_VALUE - 8)
to every non-zero-byte-element collection (which also covers collections read
from a source that cannot report the bytes remaining), and bounds the array/map
skip paths. When set, the AVRO_MAX_COLLECTION_ITEMS environment variable caps
both limits.
was:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a large
allocation before the shortfall is noticed. When the source can report how many
bytes remain, reject a declared length (or a collection block count) that
exceeds the bytes actually available before allocating for it. Companion to
AVRO-4241 (Java).
BinaryDecoder#bytes_remaining reports the bytes still readable; #read and
DatumReader#read_array/#read_map consult it via min_bytes_per_element.
ensure_collection_available tracks the cumulative count and enforces the
limits, and the skip path (skip_blocks, used by skip_array/skip_map) is bounded
the same way. Rejections raise the dedicated Avro::IO::CollectionSizeError (a
subclass of AvroError).
Zero-byte elements (null, or a record with only zero-byte fields) consume no
input, so the available-bytes check cannot bound their count: a tiny payload
such as {"type":"array","items":"null"} declaring a block count of 200,000,000
would otherwise drive an unbounded allocation. In addition to the
available-bytes check this therefore caps the cumulative count of zero-byte
elements (default 10,000,000), applies a structural cap to every collection
(Integer.MAX_VALUE - 8) for readers that cannot report bytes remaining, and
bounds the array/map skip paths. When set, the AVRO_MAX_COLLECTION_ITEMS
environment variable caps both limits.
> [ruby] Bound allocation when decoding length-prefixed values and collections
> ----------------------------------------------------------------------------
>
> Key: AVRO-4297
> URL: https://issues.apache.org/jira/browse/AVRO-4297
> Project: Apache Avro
> Issue Type: Sub-task
> Components: ruby
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> large allocation before the shortfall is noticed. When the source can report
> how many bytes remain, reject a declared length (or a collection block count)
> that exceeds the bytes actually available before allocating for it. Companion
> to AVRO-4241 (Java).
> BinaryDecoder#bytes_remaining reports the bytes still readable; #read
> consults it directly for a declared length above a threshold, while
> DatumReader#read_array/#read_map consult it via min_bytes_per_element.
> ensure_collection_available tracks the cumulative count and enforces the
> limits, and the skip path (skip_blocks, used by skip_array/skip_map) is
> bounded the same way. Rejections raise the dedicated
> Avro::IO::CollectionSizeError (a subclass of AvroError).
> Zero-byte elements (null, or a record with only zero-byte fields) consume no
> input, so the available-bytes check cannot bound their count: a tiny payload
> such as {"type":"array","items":"null"} declaring a block count of
> 200,000,000 would otherwise drive an unbounded allocation. In addition to the
> available-bytes check this therefore caps the cumulative count of zero-byte
> elements (default 10,000,000), applies a structural cap (Integer.MAX_VALUE -
> 8) to every non-zero-byte-element collection (which also covers collections
> read from a source that cannot report the bytes remaining), and bounds the
> array/map skip paths. When set, the AVRO_MAX_COLLECTION_ITEMS environment
> variable caps both limits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
