[
https://issues.apache.org/jira/browse/AVRO-4293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4293:
-------------------------------
Description:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a large
allocation before the shortfall is noticed. When the source can report how many
bytes remain, reject a declared length (or a collection block count) that
exceeds the bytes actually available before allocating for it. Companion to
AVRO-4241 (Java).
avro_reader_bytes_available() reports the bytes still readable from a
memory-backed reader; read_bytes/read_string and
read_array_value/read_map_value consult it, using min_bytes_per_element() from
the element schema. The collection limits and the min-bytes helper are exposed
via avro_private.h so the datum skip path (datum_skip.c skip_array/skip_map) is
bounded the same way. Also fixes a latent NULL-dereference in
avro_raw_map_get_or_create where the result of avro_raw_array_append was
dereferenced before the NULL check.
Zero-byte elements (null, a zero-length fixed, or a record with only zero-byte
fields) consume no input, so the available-bytes check cannot bound their
count: a tiny payload such as {"type":"array","items":"null"} declaring a block
count of 200,000,000 would otherwise drive an unbounded allocation. In addition
to the available-bytes check this therefore caps the cumulative count of
zero-byte elements (default 10,000,000), applies a structural cap to every
collection (Integer.MAX_VALUE - 8) for readers that cannot report bytes
remaining, and bounds the array/map skip paths. When set, the
AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This
supersedes the separate collection-limit sub-task.
Also folds in AVRO-4275: a negative block count whose absolute value is
INT64_MIN cannot be negated in int64_t (block_count * -1 is signed-overflow
undefined behavior, CWE-190), which drove an unbounded loop after the cast to
size_t. It is now rejected across all three C decoder paths (value-read.c,
consume-binary.c, datum_skip.c). Supersedes AVRO-4275.
was:
A bytes or string value is encoded as a length prefix followed by that many
bytes of data, and an array or map block is encoded as an element count
followed by that many items. A malicious or truncated input can declare a very
large length or count while carrying little or no actual data, causing a large
allocation before the shortfall is noticed. When the source can report how many
bytes remain, reject a declared length (or a collection block count) that
exceeds the bytes actually available before allocating for it. Companion to
AVRO-4241 (Java).
avro_reader_bytes_available() reports the bytes still readable from a
memory-backed reader; read_bytes/read_string and
read_array_value/read_map_value consult it, using min_bytes_per_element() from
the element schema. The collection limits and the min-bytes helper are exposed
via avro_private.h so the datum skip path (datum_skip.c skip_array/skip_map) is
bounded the same way. Also fixes a latent NULL-dereference in
avro_raw_map_get_or_create where the result of avro_raw_array_append was
dereferenced before the NULL check.
Zero-byte elements (null, a zero-length fixed, or a record with only zero-byte
fields) consume no input, so the available-bytes check cannot bound their
count: a tiny payload such as {"type":"array","items":"null"} declaring a block
count of 200,000,000 would otherwise drive an unbounded allocation. In addition
to the available-bytes check this therefore caps the cumulative count of
zero-byte elements (default 10,000,000), applies a structural cap to every
collection (Integer.MAX_VALUE - 8) for readers that cannot report bytes
remaining, and bounds the array/map skip paths. When set, the
AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This
supersedes the separate collection-limit sub-task.
> [c] Bound allocation when decoding length-prefixed values and collections
> -------------------------------------------------------------------------
>
> Key: AVRO-4293
> URL: https://issues.apache.org/jira/browse/AVRO-4293
> Project: Apache Avro
> Issue Type: Sub-task
> Components: c
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.11.6, 1.12.2
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> large allocation before the shortfall is noticed. When the source can report
> how many bytes remain, reject a declared length (or a collection block count)
> that exceeds the bytes actually available before allocating for it. Companion
> to AVRO-4241 (Java).
> avro_reader_bytes_available() reports the bytes still readable from a
> memory-backed reader; read_bytes/read_string and
> read_array_value/read_map_value consult it, using min_bytes_per_element()
> from the element schema. The collection limits and the min-bytes helper are
> exposed via avro_private.h so the datum skip path (datum_skip.c
> skip_array/skip_map) is bounded the same way. Also fixes a latent
> NULL-dereference in avro_raw_map_get_or_create where the result of
> avro_raw_array_append was dereferenced before the NULL check.
> Zero-byte elements (null, a zero-length fixed, or a record with only
> zero-byte fields) consume no input, so the available-bytes check cannot bound
> their count: a tiny payload such as {"type":"array","items":"null"} declaring
> a block count of 200,000,000 would otherwise drive an unbounded allocation.
> In addition to the available-bytes check this therefore caps the cumulative
> count of zero-byte elements (default 10,000,000), applies a structural cap to
> every collection (Integer.MAX_VALUE - 8) for readers that cannot report bytes
> remaining, and bounds the array/map skip paths. When set, the
> AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This
> supersedes the separate collection-limit sub-task.
> Also folds in AVRO-4275: a negative block count whose absolute value is
> INT64_MIN cannot be negated in int64_t (block_count * -1 is signed-overflow
> undefined behavior, CWE-190), which drove an unbounded loop after the cast to
> size_t. It is now rejected across all three C decoder paths (value-read.c,
> consume-binary.c, datum_skip.c). Supersedes AVRO-4275.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
