iemejia opened a new pull request, #581: URL: https://github.com/apache/avro-rs/pull/581
## What An `array` or `map` block is encoded as an element count followed by that many items. When decoding, `items.reserve(len)` allocates `len * size_of::<Value>()` bytes (`Value` is 56 bytes), but the count was only validated with `safe_len`, which bounds it as if it were a *byte* count. As a result a tiny payload declaring a huge block count could drive a multi-gigabyte allocation far exceeding the configured `max_allocation_bytes` budget. At the `safe_len` ceiling this reaches ~28 GiB. Reproduced with a **5-byte** payload (`array<null>`, block count `10_000_000`) decoding into a 10M-element array (534 MiB) — *exceeding* the default 512 MiB budget while being accepted. This affects: - both **arrays and maps**; - both **zero-byte** on-wire elements (`null`) and **non-zero-byte** elements — the `reserve` happens before any element is read; - and the zero-byte "flood" where the decode loop grows the collection unboundedly (zero-byte elements consume no input, so nothing else bounds the count). ## How - Add `util::safe_collection_len(total_items, item_size)`, which bounds the **cumulative** element count against `max_allocation_bytes`, accounting for the in-memory size of each decoded element (mirroring `safe_len`, which only covers byte-length allocations). - Apply it in the `Schema::Array` and `Schema::Map` decode loops before `reserve`, using `size_of::<Value>()` and `size_of::<(String, Value)>()` respectively, with a `checked_add` guard for the running total. This caps both the up-front reservation and the total number of decoded elements. The limit stays configurable through `max_allocation_bytes` (the default 512 MiB budget yields ~9.6M `Value`s, close to the item caps used by the other Avro SDKs). `bytes`/`string` length-prefixed values are already bounded by `safe_len`; since `decode` operates on a bare `Read` with no known remaining size, that budget cap is the appropriate guard (equivalent to the non-seekable path in the other SDKs). ## Tests - New: huge `array<null>`, `array<long>`, and `map` block counts are rejected instead of allocating; a small `array<null>` still decodes. - `cargo test -p apache-avro` (564 lib tests) passes; `cargo fmt --check` and `cargo clippy` are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
