Ramin Gharib created FLINK-40936:
------------------------------------
Summary: Built-in functions compare VARIANT array elements and map
keys by their binary encoding
Key: FLINK-40936
URL: https://issues.apache.org/jira/browse/FLINK-40936
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner, Table SQL / Runtime
Reporter: Ramin Gharib
Assignee: Ramin Gharib
Functions that compare array elements or map keys compare VARIANT values by
their binary encoding. One VARIANT value has many valid encodings. Object keys
can be stored in any order, a field read from an object keeps the key
dictionary of its parent, and a number keeps its storage width. So these
functions treat equal values as different and return wrong results, without an
error.
{code:sql}
-- t(s STRING) has three rows: \{"a":1,"b":2}, \{"b":2,"a":1}, \{"a":1,"c":3}
SELECT ARRAY_CONTAINS(ARRAY[PARSE_JSON(s)['a']], PARSE_JSON('1')) FROM t;
-- returns false
{code}
Affected functions: ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION,
ARRAY_REMOVE, ARRAY_UNION, ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY and
MAP_UNION. GREATEST, LEAST and MAP_FROM_ENTRIES also accept VARIANT during type
inference.
FLINK-40910 rejects VARIANT in GROUP BY, DISTINCT, joins, set operations and
comparison operators. This issue rejects VARIANT, also when nested in a ROW or
structured type, as the element or key that these functions compare.
LogicalTypeChecks.areComparable rejects types that are not comparable key
types, and the functions above check element or key equality in their input
type strategy. Users cast the value to a concrete type first.
Compatibility: these functions build their equality check again during code
generation. A job restored from a compiled plan that calls one of them on
VARIANT elements or keys fails when it is executed, before the job is submitted.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
