The GitHub Actions job "Docs 404 check (beta)" on 
flink.git/FLINK-40936-variant-function-equality has succeeded.
Run started by GitHub user raminqaf (triggered by raminqaf).

Head commit for run:
2491ce91f48743a51d33c8816d0f46cba57d5ce2 / Ramin Gharib <[email protected]>
[FLINK-40936][table] Reject VARIANT elements and keys in functions and 
operators that compare them

Flink compares VARIANT values by their binary encoding, but one value has many 
encodings. So functions and operators that compare array elements, map keys or 
collected values treated equal VARIANT values as different and silently 
returned wrong results.

ComparableTypeValidator now checks the array elements and map keys of 
ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION, ARRAY_REMOVE, ARRAY_UNION, 
ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY and MAP_UNION, the map key of 
an item access m[k], and the argument of COLLECT, also in an OVER window. An 
index into an array and a field access on a VARIANT compare nothing and stay 
allowed. The validator names what it checks in sets: the functions, the 
operators that compare elements, and the comparison operators that need 
equality or order. The comparison operator sets hold the same kinds that the 
validator matched before.

The check lives in the validator and not in the input type strategy of each 
function. That would need a new argument strategy and a flag in three 
collection strategies, for a check that a future VARIANT equality may remove.

LogicalTypeChecks.areComparable rejects every type that is not comparable, see 
LogicalTypeChecks.isComparableType. This also covers a structured type with a 
VARIANT attribute. GREATEST, LEAST, MAP_FROM_ENTRIES and Table API comparisons 
reject VARIANT through it during type inference.

Ten functions build their equality check again during code generation: 
ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION, ARRAY_REMOVE, ARRAY_UNION, 
ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY, MAP_UNION and 
MAP_FROM_ENTRIES. That check goes through LogicalTypeChecks.areComparable, not 
through the validator, which only runs when a query is planned. So a job 
restored from a compiled plan that calls one of them on VARIANT elements or 
keys now fails when it is executed, before the job is submitted. Such a plan 
returned wrong results before. A user SpecializedFunction that builds an = 
evaluator on VARIANT values now fails too, in new and in restored plans.

The VARIANT section of the data types page lists these functions and operators.

Report URL: https://github.com/apache/flink/actions/runs/40227581907

With regards,
GitHub Actions via GitBox

Reply via email to