[
https://issues.apache.org/jira/browse/FLINK-41362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib closed FLINK-41362.
--------------------------------
Fix Version/s: 2.5.0
Release Note: A PRIMARY KEY column can no longer be a VARIANT or a type
that contains one, such as ROW<v VARIANT>. Flink matches primary keys by their
binary encoding, and one VARIANT value has many encodings, so such a key could
keep two rows for one key and drop deletes. CREATE TABLE now fails with
"Invalid primary key '<name>'. Column '<column>' of type VARIANT has no
equality." A table stored with such a key by an earlier version fails to
resolve, so it cannot be queried, altered or dropped through SQL. Drop it with
Catalog#dropTable through TableEnvironment#getCatalog, then recreate it with a
key column of a concrete type.
Resolution: Fixed
> Reject VARIANT in primary key columns
> -------------------------------------
>
> Key: FLINK-41362
> URL: https://issues.apache.org/jira/browse/FLINK-41362
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
> Reporter: Ramin Gharib
> Assignee: Ramin Gharib
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.5.0
>
>
> Flink rejects {{VARIANT}} where a query needs equality, such as {{GROUP BY}},
> {{DISTINCT}}, join keys and {{=}} (FLINK-40910). A {{PRIMARY KEY}} needs
> equality too, but {{DefaultSchemaResolver}} accepts a {{VARIANT}} key column.
> Changelog normalization and upsert sinks key their state by the primary key
> and match keys by their binary encoding. One {{VARIANT}} value has many
> encodings. For example, {{PARSE_JSON('\{"a":1,"b":2\}')}} and
> {{PARSE_JSON('\{"b":2,"a":1\}')}} print the same JSON, but their key
> dictionaries are stored in a different order. So the primary key does not
> identify a row.
> h3. Reproducer
> {code:sql}
> CREATE TABLE src (
> k VARIANT NOT NULL,
> x INT,
> PRIMARY KEY (k) NOT ENFORCED
> ) WITH (
> 'connector' = 'values',
> 'changelog-mode' = 'I,UA,D',
> 'data-id' = '...'
> );
> SELECT JSON_STRING(k), x FROM src;
> {code}
> {{k_ab}} is {{PARSE_JSON('\{"a":1,"b":2\}')}} and {{k_ba}} is
> {{PARSE_JSON('\{"b":2,"a":1\}')}}. Streaming mode, parallelism 1.
> || Source changelog || Expected || Actual ||
> | {{+I\[k_ab, 1\]}}, {{+U\[k_ab, 2\]}} | one row, x = 2 | {{+I\[.., 1\]}},
> {{-U\[.., 1\]}}, {{+U\[.., 2\]}}: one row, x = 2 |
> | {{+I\[k_ab, 1\]}}, {{+U\[k_ba, 2\]}} | the {{CREATE TABLE}} fails |
> {{+I\[.., 1\]}}, {{+I\[.., 2\]}}: two rows for one key |
> | {{+I\[k_ab, 1\]}}, {{-D\[k_ba, 1\]}} | the {{CREATE TABLE}} fails |
> {{+I\[.., 1\]}}: the delete is dropped |
> The first row is the control with identical bytes.
> h3. Proposal
> Reject a primary key column whose type is not comparable, using
> {{LogicalTypeChecks#isComparableType}}, the rule that {{GROUP BY}} and joins
> already use. This covers {{VARIANT}} and every type that contains one, such
> as {{ARRAY<VARIANT>}} or {{ROW<v VARIANT>}}. The check goes into
> {{DefaultSchemaResolver#validatePrimaryKey}}, next to the NOT NULL check.
> Like that check, it also runs when a stored catalog table is read, so a
> stored table with such a key no longer resolves.
> Out of scope: {{MAP}} and {{MULTISET}} keys have a similar problem, because
> their binary encoding follows the entry order. That is older than {{VARIANT}}
> and also affects streaming {{GROUP BY}} on a {{MAP}}, so it needs its own
> ticket.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
