Ismaël Mejía created AVRO-4301:
----------------------------------
Summary: [js] Bound collection allocation when decoding arrays and
maps
Key: AVRO-4301
URL: https://issues.apache.org/jira/browse/AVRO-4301
Project: Apache Avro
Issue Type: Sub-task
Components: javascript
Reporter: Ismaël Mejía
The JavaScript SDK already bounds length-prefixed bytes/string/fixed values
against its in-memory buffer (Tap.readFixed/readString check the position
against the buffer length before allocating), which is why it had no
available-bytes sub-task. Collections, however, are not bounded:
ArrayType._read and MapType._read read the block item count and push elements
incrementally into a plain array/object with no cap and no bytes-remaining
check. Because zero-byte elements (e.g. null) consume no input, a ~6 byte
payload such as {"type":"array","items":"null"} declaring a block count of
200,000,000 builds a 200M-entry array and exhausts the heap (reproduced: FATAL
ERROR: JavaScript heap out of memory under --max-old-space-size=256). Even
truncated non-zero-byte collections are affected, because an out-of-range
Buffer read returns undefined rather than throwing, so the loop runs to the
full declared count; the tap validity check only happens after the read returns.
Bound the array/map read and skip paths, consistent with the other SDKs: reject
a block whose element count could not be backed by the bytes remaining (using
the minimum on-wire size of the element schema), cap the cumulative count of
zero-byte elements (default 10,000,000), and apply a structural cap to every
collection (Integer.MAX_VALUE - 8). When set, the AVRO_MAX_COLLECTION_ITEMS
environment variable caps both limits. The limit is computed from the writer
element type under schema resolution so legitimate collections are not falsely
rejected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
