raminqaf opened a new pull request, #29801:
URL: https://github.com/apache/flink/pull/29801
## What is the purpose of the change
Flink rejects `VARIANT` where a query needs equality, such as `GROUP BY`,
joins and `=` (FLINK-40910). A primary key needs equality too, but the schema
resolver accepted a `VARIANT` key column. Changelog normalization and upsert
sinks key their state by the primary key and match keys by their binary
encoding, and one `VARIANT` value has many encodings.
```sql
CREATE TABLE src (k VARIANT NOT NULL, x INT, PRIMARY KEY (k) NOT ENFORCED)
WITH ('connector' = 'values', 'changelog-mode' = 'I,UA,D', ...);
SELECT JSON_STRING(k), x FROM src;
```
`k_ab` is `PARSE_JSON('{"a":1,"b":2}')` and `k_ba` is
`PARSE_JSON('{"b":2,"a":1}')`.
| Source changelog | Before
| After |
|------------------------------|----------------------------------------------|---------------------------|
| `+I[k_ab, 1]`, `+U[k_ba, 2]` | `+I[.., 1]`, `+I[.., 2]`: two rows, one key
| the `CREATE TABLE` fails |
| `+I[k_ab, 1]`, `-D[k_ba, 1]` | `+I[.., 1]`: the delete is dropped
| the `CREATE TABLE` fails |
## Brief change log
- `DefaultSchemaResolver` rejects a primary key column whose type is not
comparable (`LogicalTypeChecks#isComparableType`), the rule that `GROUP BY` and
joins use. This covers `VARIANT` and every type that contains one, such as
`ROW<v VARIANT>`. The error reads `Invalid primary key 'PK_v'. Column 'v' of
type VARIANT has no equality.`
- Docs: the `VARIANT` section of the data types page and the `PRIMARY KEY`
section of `CREATE TABLE`.
Compatibility: like the existing NOT NULL check, the rule also runs when a
stored catalog table is read. A stored table with a `VARIANT` primary key no
longer resolves.
Not in this PR: `MAP` and `MULTISET` keys have a similar problem, because
their binary encoding follows the entry order. That predates `VARIANT` and also
affects streaming `GROUP BY` on a `MAP`, so it needs its own ticket.
## Verifying this change
This change added tests and can be verified as follows:
- `SchemaResolutionTest#testSchemaResolutionErrors` checks the new error.
It fails without the fix.
- The before/after table above comes from a throwaway probe against the
`values` connector, not part of this PR, since after the fix the `CREATE TABLE`
fails.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code 2.1.318 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
