raminqaf opened a new pull request, #29416:
URL: https://github.com/apache/flink/pull/29416
Based on #29411. Only the last commit belongs to this PR.
## What is the purpose of the change
Flink compares VARIANT values by their binary encoding, but one value has
many encodings. So functions that compare array elements or map keys treat
equal VARIANT values as different and return wrong results, without an error.
#29411 rejects VARIANT in GROUP BY, joins and comparisons. This PR does the
same for these functions.
```sql
-- t(s STRING) has three rows: {"a":1,"b":2}, {"b":2,"a":1}, {"a":1,"c":3}
SELECT ARRAY_CONTAINS(ARRAY[PARSE_JSON(s)['a']], PARSE_JSON('1')) FROM t;
```
| Query | Before | After
|
|----------------------------|---------|------------------------------------------------------------------|
| the `ARRAY_CONTAINS` above | `false` | `Type 'VARIANT' should support
'EQUALS' comparison with itself.` |
Users cast the value to a concrete type first.
## Brief change log
- `LogicalTypeChecks#areComparable` rejects every type that is not a
comparable key type, see `LogicalTypeChecks#isComparableKeyType` from #29411.
This also covers a structured type with a VARIANT attribute.
- ARRAY_CONTAINS, ARRAY_DISTINCT, ARRAY_POSITION, ARRAY_REMOVE, ARRAY_UNION,
ARRAY_EXCEPT, ARRAY_INTERSECT, MAP_CONTAINS_KEY and MAP_UNION declare in their
input type strategy that they compare elements or keys. Their runtime already
compares through `isEqual` evaluators, so for other types the error only moves
from code generation to validation.
- GREATEST, LEAST, MAP_FROM_ENTRIES and Table API comparisons reject VARIANT
the same way, through `areComparable`.
- The VARIANT section of the data types page lists these functions.
These functions build their equality check again during code generation. So
a job restored from a compiled plan that calls one of them on VARIANT elements
or keys now fails when it is executed, before the job is submitted. Other
restored plans are not affected.
## Verifying this change
This change added tests and can be verified as follows:
- `CollectionFunctionsITCase` and `MapFunctionITCase` cover VARIANT elements
and keys for each affected function in SQL and the Table API, including a
structured type with a VARIANT attribute.
- `ComparableInputTypeStrategyTest`,
`EqualsComparableElementArgumentTypeStrategyTest` and
`CommonCollectionInputTypeStrategyTest` cover the type checks.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes. `BuiltInFunctionDefinitions` changes the input type
strategy of the nine functions above. No signature changes. Calls with VARIANT
elements or keys now fail, which needs a release note.
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes. A restored
compiled plan that calls one of these functions on VARIANT elements or keys
fails when it is executed.
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.292 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
