[
https://issues.apache.org/jira/browse/FLINK-41468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramin Gharib closed FLINK-41468.
--------------------------------
Fix Version/s: 2.5.0
(was: 2.4.0)
Resolution: Fixed
Fixed in master: 13cc6d08d96fb7f016ceea460f49604068c8f816
> Converting table to materialized table with different partition keys makes
> existing data unreadable
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-41468
> URL: https://issues.apache.org/jira/browse/FLINK-41468
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
> Affects Versions: 2.4.0
> Reporter: Mika Naylor
> Assignee: Timo Theusner
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.5.0
>
>
> CREATE OR ALTER MATERIALIZED TABLE can turn a regular table into an MT, which
> keeps the old table's data, but its partition keys come only from the
> statement and nothing ensures that they match the old table's partition keys.
> If theyre different, the existing data no longer match the MT's partitioning.
> The conversion succeeds without a warning, but afterwards:
> * If the partition key changes, batch reads of the MT fail.
> * If an unpartitioned table is converted with PARTITION BY, batch reads fail.
> * If a partitioned table is converted without PARTITIONED BY, reads work,
> but rows written before the conversion lose their parition column value (it
> becomes NULL).
> FLIP-578 says that partition keys are inherited from the existing table,
> unless the statement overrides them, but the current impl does not inherit
> them.
> h4. Reproduction Steps
> On 2.4 with table.materialized-table.conversion-from-table.enabled: true:
>
> {code:java}
> CREATE CATALOG mt_cat WITH ('type' = 'test-filesystem', 'path' =
> '/tmp/flip578/catalog', 'default-database' = 'mydb');
> USE CATALOG mt_cat;
> CREATE TABLE orders (user_id BIGINT NOT NULL, amount BIGINT, ds STRING) WITH
> ('format' = 'json');
> INSERT INTO orders VALUES (1, 30, '2026-11-02');
> -- A regular table partitioned by ds, with one row
> CREATE TABLE p (ds STRING, user_id BIGINT NOT NULL, amount BIGINT)
> PARTITIONED BY (ds) WITH ('format' = 'json');
> INSERT INTO p VALUES ('2026-10-31', 100, 500);
> -- Convert it, partitioned by a different key
> CREATE OR ALTER MATERIALIZED TABLE p
> PARTITIONED BY (user_id)
> WITH ('format' = 'json')
> FRESHNESS = INTERVAL '30' SECOND
> AS SELECT ds, user_id, amount FROM orders;SET 'execution.runtime-mode' =
> 'batch';
> SELECT * FROM p;
> -- TableException: Partition keys are: [user_id], incomplete partition spec:
> {ds=2026-10-31}
> {code}
> h4. Expected Behaviour
> After a successful conversion, I would expect data the table held before
> conversion could still be read, and if the conversion can't guarantee that,
> it should be rejected with a clear error.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)