Yibo Dong created FLINK-41552:
---------------------------------

             Summary: Filesystem INSERT OVERWRITE leaves stale rows when the 
replacement partition is empty
                 Key: FLINK-41552
                 URL: https://issues.apache.org/jira/browse/FLINK-41552
             Project: Flink
          Issue Type: Bug
         Environment: Apache Flink 2.3.0 (stock), batch Table API, local 
filesystem connector with CSV format. No optimizer rules disabled.
            Reporter: Yibo Dong


h2. Description

INSERT OVERWRITE on a statically specified partition completes successfully but 
leaves its existing rows unchanged when the replacement query produces no rows.

The same final data written to a fresh table confirms the discrepancy.
h2. How to Reproduce

Run the following SQL in a fresh session with initially nonexistent filesystem 
paths:
{code:sql}
SET 'execution.runtime-mode' = 'BATCH';

CREATE TABLE t_fresh (id INT, k STRING, v INT)
PARTITIONED BY (k)
WITH (
'connector' = 'filesystem',
'format' = 'csv',
'path' = 'file:///tmp/flink-overwrite-repro/fresh'
);

CREATE TABLE t_overwrite (id INT, k STRING, v INT)
PARTITIONED BY (k)
WITH (
'connector' = 'filesystem',
'format' = 'csv',
'path' = 'file:///tmp/flink-overwrite-repro/overwrite'
);

INSERT INTO t_fresh VALUES (2, 'b', 12);

INSERT INTO t_overwrite VALUES
(1, 'a', 11),
(2, 'b', 12);

INSERT OVERWRITE t_overwrite PARTITION (k='a')
SELECT id, v
FROM (VALUES (3, 13)) AS t(id, v)
WHERE id = 0;

SELECT id, k, v FROM t_fresh ORDER BY id;
SELECT id, k, v FROM t_overwrite ORDER BY id;
{code}
The overwrite query produces zero rows, while the static PARTITION clause 
explicitly targets k='a'.
h2. Expected Result

Both tables should contain only:
{code}
Unable to find source-code formatter for language: text. Available languages 
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, 
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, 
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml(2, 
b, 12)
{code}
Partition k='a' should contain no rows, while partition k='b' should remain 
unchanged.
h2. Actual Result

The INSERT OVERWRITE statement completes without an error.

The fresh table returns:
{code}
(2, b, 12)
{code}
However, the overwritten table still returns:
{code}
(1, a, 11)
(2, b, 12)
{code}
The old data file under partition k='a' remains after the overwrite.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to