The GitHub Actions job "Docs 404 check (beta)" on 
flink.git/FLINK-40936-variant-function-equality has succeeded.
Run started by GitHub user raminqaf (triggered by raminqaf).

Head commit for run:
7b3c4f1d0625c90b0212ce69b338846f8ff1cbc4 / Ramin Gharib <[email protected]>
[FLINK-40936][table] Reject VARIANT elements and keys in functions that compare 
them

Flink compares VARIANT values by their binary encoding, but one value has many 
encodings. So functions that compare array elements or map keys treated equal 
VARIANT values as different and silently returned wrong results.

LogicalTypeChecks.areComparable rejects every type that is not a comparable key 
type, see LogicalTypeChecks.isComparableKeyType. This also covers a structured 
type with a VARIANT attribute. ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION, 
ARRAY_REMOVE, ARRAY_UNION, ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY and 
MAP_UNION declare in their input type strategy that they compare elements or 
keys, so they reject VARIANT elements and keys during type inference. Their 
runtime already compares through isEqual evaluators, so for other types the 
error only moves from code generation to validation. GREATEST, LEAST, 
MAP_FROM_ENTRIES and Table API comparisons reject VARIANT the same way.

Ten functions build their equality check again during code generation: 
ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION, ARRAY_REMOVE, ARRAY_UNION, 
ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY, MAP_UNION and 
MAP_FROM_ENTRIES. That check goes through LogicalTypeChecks.areComparable, not 
through the new input type strategies. So a job restored from a compiled plan 
that calls one of them on VARIANT elements or keys now fails when it is 
executed, before the job is submitted. Such a plan returned wrong results 
before. A user SpecializedFunction that builds an = evaluator on VARIANT values 
now fails too, in new and in restored plans.

The VARIANT section of the data types page lists these functions.

Report URL: https://github.com/apache/flink/actions/runs/39755080693

With regards,
GitHub Actions via GitBox

Reply via email to