[
https://issues.apache.org/jira/browse/AVRO-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía updated AVRO-4294:
-------------------------------
Fix Version/s: (was: 1.13.0)
(was: 1.11.6)
(was: 1.12.2)
> [c++] Bound allocation when decoding length-prefixed values and collections
> ---------------------------------------------------------------------------
>
> Key: AVRO-4294
> URL: https://issues.apache.org/jira/browse/AVRO-4294
> Project: Apache Avro
> Issue Type: Sub-task
> Components: c++
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> large allocation before the shortfall is noticed. When the source can report
> how many bytes remain, reject a declared length (or a collection block count)
> that exceeds the bytes actually available before allocating for it. Companion
> to AVRO-4241 (Java).
> remainingBytes() on InputStream (exposed via StreamReader) and
> bytesRemaining() on the Decoder interface back the check;
> decodeString/decodeBytes and GenericReader's array/map decoding consult it.
> minBytesPerElement() computes the per-element minimum consulted by
> ensureCollectionAvailable; under schema resolution the array bytes-remaining
> check is effectively disabled (minimum 0) while maps keep a >=1-byte key, and
> BinaryDecoder::skipArray is bounded by the structural cap.
> Zero-byte elements (null, a zero-length fixed, or a record with only
> zero-byte fields) consume no input, so the available-bytes check cannot bound
> their count: a tiny payload such as {"type":"array","items":"null"} declaring
> a block count of 200,000,000 would otherwise drive an unbounded allocation.
> In addition to the available-bytes check this therefore caps the cumulative
> count of zero-byte elements (default 10,000,000), applies a structural cap
> (Integer.MAX_VALUE - 8) to every non-zero-byte-element collection (which also
> covers collections read from a source that cannot report the bytes
> remaining), and bounds the array/map skip paths. When set, the
> AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. This
> supersedes the separate collection-limit sub-task.
> Also folds in AVRO-4276: doDecodeItemCount() turned an INT64_MIN block count
> into 2^63 (an impossible allocation downstream); it now rejects INT64_MIN
> early and uses the simpler -result negation. Supersedes AVRO-4276.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
