[ 
https://issues.apache.org/jira/browse/FLINK-41468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ramin Gharib closed FLINK-41468.
--------------------------------
    Fix Version/s: 2.5.0
                       (was: 2.4.0)
       Resolution: Fixed

Fixed in master: 13cc6d08d96fb7f016ceea460f49604068c8f816

> Converting table to materialized table with different partition keys makes 
> existing data unreadable
> ---------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-41468
>                 URL: https://issues.apache.org/jira/browse/FLINK-41468
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Gateway
>    Affects Versions: 2.4.0
>            Reporter: Mika Naylor
>            Assignee: Timo Theusner
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.5.0
>
>
> CREATE OR ALTER MATERIALIZED TABLE can turn a regular table into an MT, which 
> keeps the old table's data, but its partition keys come only from the 
> statement and nothing ensures that they match the old table's partition keys.
> If theyre different, the existing data no longer match the MT's partitioning. 
> The conversion succeeds without a warning, but afterwards:
>  * If the partition key changes, batch reads of the MT fail.
>  * If an unpartitioned table is converted with PARTITION BY, batch reads fail.
>  * If a partitioned table is converted without PARTITIONED BY, reads work, 
> but rows written before the conversion lose their parition column value (it 
> becomes NULL).
> FLIP-578 says that partition keys are inherited from the existing table, 
> unless the statement overrides them, but the current impl does not inherit 
> them.
> h4. Reproduction Steps
> On 2.4 with table.materialized-table.conversion-from-table.enabled: true:
>  
> {code:java}
> CREATE CATALOG mt_cat WITH ('type' = 'test-filesystem', 'path' = 
> '/tmp/flip578/catalog', 'default-database' = 'mydb');
> USE CATALOG mt_cat;
> CREATE TABLE orders (user_id BIGINT NOT NULL, amount BIGINT, ds STRING) WITH 
> ('format' = 'json');
> INSERT INTO orders VALUES (1, 30, '2026-11-02');
> -- A regular table partitioned by ds, with one row
> CREATE TABLE p (ds STRING, user_id BIGINT NOT NULL, amount BIGINT)
>   PARTITIONED BY (ds) WITH ('format' = 'json');
> INSERT INTO p VALUES ('2026-10-31', 100, 500);
> -- Convert it, partitioned by a different key
> CREATE OR ALTER MATERIALIZED TABLE p
>   PARTITIONED BY (user_id)
>   WITH ('format' = 'json')
>   FRESHNESS = INTERVAL '30' SECOND
>   AS SELECT ds, user_id, amount FROM orders;SET 'execution.runtime-mode' = 
> 'batch';
> SELECT * FROM p;
> -- TableException: Partition keys are: [user_id], incomplete partition spec: 
> {ds=2026-10-31}
> {code}
> h4. Expected Behaviour
> After a successful conversion, I would expect data the table held before 
> conversion could still be read, and if the conversion can't guarantee that, 
> it should be rejected with a clear error.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to