[
https://issues.apache.org/jira/browse/FLINK-41361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib reassigned FLINK-41361:
------------------------------------
Assignee: Ramin Gharib
> Bound VARIANT object and array sizes by the value length before allocating
> --------------------------------------------------------------------------
>
> Key: FLINK-41361
> URL: https://issues.apache.org/jira/browse/FLINK-41361
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Minor
>
> A malformed VARIANT value whose object or array header claims more elements
> than its bytes hold makes readers allocate arrays of that size.
> Example: metadata {{01 00 00}} and value {{42 ff ff ff 7f}}. The value is 5
> bytes, but its header declares an object with 2,147,483,647 fields.
> {{toJson()}}, {{toString()}} and {{JSON_STRING}} allocate {{new
> int[2147483647]}} and throw an OutOfMemoryError. A VariantTypeException is
> the expected failure.
> A value can reach these readers without being decoded first through formats
> that store the binary encoding, such as {{'raw.variant-encoding' = 'binary'}}.
> *Where*
> - {{BinaryVariantUtil.handleObject}} (L790) and {{handleArray}} (L843) read
> the element count from the header and pass it to the handler as is.
> - Allocations sized by that count: {{JsonVariantFormatter}} L126, L127, L146;
> {{BinaryVariant.getFieldNames}} L313; {{BinaryVariantInternalBuilder}}
> L734-735, when it appends an existing variant.
> *Proposal*
> In {{handleObject}} and {{handleArray}}, check that the id and offset tables
> fit in the value before calling the handler, computed in long arithmetic, and
> throw a VariantTypeException such as "The VARIANT object header declares more
> fields than the value holds." Every element takes at least one id byte and
> one offset byte, so this bounds every allocation by the value length. Values
> are still read as is, and the check runs only when a reader walks the
> container.
> Spark has the same {{handleObject}}. Its {{toJson}} streams fields with
> bounds-checked reads and fails with MALFORMED_VARIANT. Its
> {{VariantBuilder.appendVariant}} has the same {{new ArrayList<>(size)}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
