Ramin Gharib created FLINK-41362:
------------------------------------
Summary: Reject VARIANT in primary key columns
Key: FLINK-41362
URL: https://issues.apache.org/jira/browse/FLINK-41362
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Reporter: Ramin Gharib
Assignee: Ramin Gharib
Flink rejects {{VARIANT}} where a query needs equality, such as {{GROUP BY}},
{{DISTINCT}}, join keys and {{=}} (FLINK-40910). A {{PRIMARY KEY}} needs
equality too, but {{DefaultSchemaResolver}} accepts a {{VARIANT}} key column.
Changelog normalization and upsert sinks key their state by the primary key and
match keys by their binary encoding. One {{VARIANT}} value has many encodings.
For example, {{PARSE_JSON('\{"a":1,"b":2\}')}} and
{{PARSE_JSON('\{"b":2,"a":1\}')}} print the same JSON, but their key
dictionaries are stored in a different order. So the primary key does not
identify a row.
h3. Reproducer
{code:sql}
CREATE TABLE src (
k VARIANT NOT NULL,
x INT,
PRIMARY KEY (k) NOT ENFORCED
) WITH (
'connector' = 'values',
'changelog-mode' = 'I,UA,D',
'data-id' = '...'
);
SELECT JSON_STRING(k), x FROM src;
{code}
{{k_ab}} is {{PARSE_JSON('\{"a":1,"b":2\}')}} and {{k_ba}} is
{{PARSE_JSON('\{"b":2,"a":1\}')}}. Streaming mode, parallelism 1.
|| Source changelog || Expected || Actual ||
| {{+I\[k_ab, 1\]}}, {{+U\[k_ab, 2\]}} | one row, x = 2 | {{+I\[.., 1\]}},
{{-U\[.., 1\]}}, {{+U\[.., 2\]}}: one row, x = 2 |
| {{+I\[k_ab, 1\]}}, {{+U\[k_ba, 2\]}} | the {{CREATE TABLE}} fails | {{+I\[..,
1\]}}, {{+I\[.., 2\]}}: two rows for one key |
| {{+I\[k_ab, 1\]}}, {{-D\[k_ba, 1\]}} | the {{CREATE TABLE}} fails | {{+I\[..,
1\]}}: the delete is dropped |
The first row is the control with identical bytes.
h3. Proposal
Reject a primary key column whose type is not comparable, using
{{LogicalTypeChecks#isComparableType}}, the rule that {{GROUP BY}} and joins
already use. This covers {{VARIANT}} and every type that contains one, such as
{{ARRAY<VARIANT>}} or {{ROW<v VARIANT>}}. The check goes into
{{DefaultSchemaResolver#validatePrimaryKey}}, next to the NOT NULL check. Like
that check, it also runs when a stored catalog table is read, so a stored table
with such a key no longer resolves.
Out of scope: {{MAP}} and {{MULTISET}} keys have a similar problem, because
their binary encoding follows the entry order. That is older than {{VARIANT}}
and also affects streaming {{GROUP BY}} on a {{MAP}}, so it needs its own
ticket.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
