[
https://issues.apache.org/jira/browse/AVRO-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102858#comment-18102858
]
ASF subversion and git services commented on AVRO-4295:
-------------------------------------------------------
Commit 1ed2417604a362e7102f68d43f69247d97fe4191 in avro's branch
refs/heads/AVRO-4295-csharp-available-bytes from Ismaël Mejía
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=1ed2417604 ]
AVRO-4295: [csharp] Apply collection allocation caps to all readers
Commit ed310ad6b hardened only DefaultReader (used by GenericReader<T>).
The other public readers -- GenericDatumReader<T>/SpecificDatumReader<T>
(via PreresolvingDatumReader<T>), SpecificDefaultReader, and
ReflectDefaultReader -- read array/map blocks without any bound, so a
tiny payload could still decode a huge collection (e.g. 100,000,000
zero-byte elements from a few bytes) or preallocate an oversized array.
Extract the shared guards (min-bytes-per-element, structural/item caps,
and EnsureCollectionAvailable) into an internal CollectionBounds helper so
the caps cannot drift between the reader implementations, and wire every
reader's ReadArray/ReadMap/skip paths through it. The block count is read
as a long and validated before the int cast, and PreresolvingDatumReader
grows its backing array in bounded, geometric chunks so a huge count on a
non-seekable stream cannot preallocate the whole block up front.
The per-datum zero-byte-element budget is tracked in a thread-static
nesting scope (mirroring the Java fix) rather than on the reader instance,
preserving PreresolvingDatumReader's documented thread-sharing contract;
nested scopes accumulate into the enclosing datum and only the outermost
resets. Adds regression tests for the Generic, Specific, and Reflect
reader paths, including a concurrency test for a shared reader.
> [csharp] Bound allocation when decoding length-prefixed values and collections
> ------------------------------------------------------------------------------
>
> Key: AVRO-4295
> URL: https://issues.apache.org/jira/browse/AVRO-4295
> Project: Apache Avro
> Issue Type: Sub-task
> Components: csharp
> Affects Versions: 1.11.5, 1.12.1
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> A bytes or string value is encoded as a length prefix followed by that many
> bytes of data, and an array or map block is encoded as an element count
> followed by that many items. A malicious or truncated input can declare a
> very large length or count while carrying little or no actual data, causing a
> large allocation before the shortfall is noticed. When the source can report
> how many bytes remain, reject a declared length (or a collection block count)
> that exceeds the bytes actually available before allocating for it. Companion
> to AVRO-4241 (Java).
> BinaryDecoder.RemainingBytes() reports the bytes still readable for a
> seekable stream; ReadBytes/ReadString consult it directly
> (EnsureAvailableBytes), while DefaultReader.ReadArray/ReadMap consult it via
> MinBytesPerElement(). EnsureCollectionAvailable tracks the cumulative count
> and enforces the limits, and the schema-resolution Skip path for arrays and
> maps is bounded the same way. Negative and out-of-range counts are rejected
> before the int cast.
> Zero-byte elements (null, or a record with only zero-byte fields) consume no
> input, so the available-bytes check cannot bound their count: a tiny payload
> such as {"type":"array","items":"null"} declaring a block count of
> 200,000,000 would otherwise drive an unbounded allocation. In addition to the
> available-bytes check this therefore caps the cumulative count of zero-byte
> elements (default 10,000,000), applies a structural cap (int.MaxValue - 8,
> further clamped to the runtime's maximum array length) to every
> non-zero-byte-element collection (which also covers collections read from a
> source that cannot report the bytes remaining), and bounds the array/map skip
> paths. When set, the AVRO_MAX_COLLECTION_ITEMS environment variable caps both
> limits.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
