The GitHub Actions job "Docs 404 check (beta)" on 
flink.git/FLINK-40910-reject-equality has failed.
Run started by GitHub user raminqaf (triggered by raminqaf).

Head commit for run:
41abe43f178734a142bda085be80baf3ffe03679 / Ramin Gharib <[email protected]>
[FLINK-40910][table] Reject VARIANT where a query needs equality or ordering

Flink compares VARIANT values by their binary encoding, but one value has many 
encodings. Object keys can be stored in any order, a field read from an object 
keeps the key dictionary of its parent, and a number keeps its storage width. 
So GROUP BY, DISTINCT, joins, set operations, PARTITION BY and = treated equal 
values as different and silently returned wrong results in streaming. Batch 
rejected most of these late, during code generation.

LogicalTypeChecks.isComparableKeyType defines which types the planner can 
group, join, partition, sort and compare. VARIANT, also nested in ROW, ARRAY or 
MAP, is no comparable key type. A new KeyTypeValidator checks the logical plan 
before optimization, so it covers SQL and the Table API. It rejects such types 
in grouping keys, DISTINCT aggregates, UNION, INTERSECT and EXCEPT except UNION 
ALL, ORDER BY, the PARTITION BY and ORDER BY of OVER windows, MATCH_RECOGNIZE 
and table arguments, comparison operators and IN subqueries. Joins are covered 
by the comparisons in their conditions.

LogicalTypeChecks.areComparable now treats VARIANT as not comparable. The 
functions that compare array elements or map keys, such as ARRAY_CONTAINS, 
ARRAY_DISTINCT and MAP_CONTAINS_KEY, declare this in their input type strategy, 
so they reject VARIANT elements during type inference. Table API comparisons 
and functions such as GREATEST reject VARIANT the same way.

A job restored from a compiled plan skips KeyTypeValidator and keeps running. A 
plan that calls a function on VARIANT elements or keys, such as ARRAY_CONTAINS, 
now fails when the job starts, because these functions build their equality 
check again during code generation.

The VARIANT section of the data types page lists where a query cannot compare 
VARIANT values and shows the cast to use instead.

Report URL: https://github.com/apache/flink/actions/runs/37486047633

With regards,
GitHub Actions via GitBox

Reply via email to