[
https://issues.apache.org/jira/browse/AVRO-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismaël Mejía reassigned AVRO-4301:
----------------------------------
Assignee: Ismaël Mejía
> [js] Bound collection allocation when decoding arrays and maps
> --------------------------------------------------------------
>
> Key: AVRO-4301
> URL: https://issues.apache.org/jira/browse/AVRO-4301
> Project: Apache Avro
> Issue Type: Sub-task
> Components: javascript
> Reporter: Ismaël Mejía
> Assignee: Ismaël Mejía
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The JavaScript SDK already bounds length-prefixed bytes/string/fixed values
> against its in-memory buffer (Tap.readFixed/readString check the position
> against the buffer length before allocating), which is why it had no
> available-bytes sub-task. Collections, however, are not bounded:
> ArrayType._read and MapType._read read the block item count and push elements
> incrementally into a plain array/object with no cap and no bytes-remaining
> check. Because zero-byte elements (e.g. null) consume no input, a ~6 byte
> payload such as {"type":"array","items":"null"} declaring a block count of
> 200,000,000 builds a 200M-entry array and exhausts the heap (reproduced:
> FATAL ERROR: JavaScript heap out of memory under --max-old-space-size=256).
> Even truncated non-zero-byte collections are affected, because an
> out-of-range Buffer read returns undefined rather than throwing, so the loop
> runs to the full declared count; the tap validity check only happens after
> the read returns.
> Bound the array/map read and skip paths, consistent with the other SDKs:
> reject a block whose element count could not be backed by the bytes remaining
> (using the minimum on-wire size of the element schema), cap the cumulative
> count of zero-byte elements (default 10,000,000), and apply a structural cap
> to every collection (Integer.MAX_VALUE - 8). When set, the
> AVRO_MAX_COLLECTION_ITEMS environment variable caps both limits. The limit is
> computed from the writer element type under schema resolution so legitimate
> collections are not falsely rejected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
