Yibo Dong created FLINK-41552:
---------------------------------
Summary: Filesystem INSERT OVERWRITE leaves stale rows when the
replacement partition is empty
Key: FLINK-41552
URL: https://issues.apache.org/jira/browse/FLINK-41552
Project: Flink
Issue Type: Bug
Environment: Apache Flink 2.3.0 (stock), batch Table API, local
filesystem connector with CSV format. No optimizer rules disabled.
Reporter: Yibo Dong
h2. Description
INSERT OVERWRITE on a statically specified partition completes successfully but
leaves its existing rows unchanged when the replacement query produces no rows.
The same final data written to a fresh table confirms the discrepancy.
h2. How to Reproduce
Run the following SQL in a fresh session with initially nonexistent filesystem
paths:
{code:sql}
SET 'execution.runtime-mode' = 'BATCH';
CREATE TABLE t_fresh (id INT, k STRING, v INT)
PARTITIONED BY (k)
WITH (
'connector' = 'filesystem',
'format' = 'csv',
'path' = 'file:///tmp/flink-overwrite-repro/fresh'
);
CREATE TABLE t_overwrite (id INT, k STRING, v INT)
PARTITIONED BY (k)
WITH (
'connector' = 'filesystem',
'format' = 'csv',
'path' = 'file:///tmp/flink-overwrite-repro/overwrite'
);
INSERT INTO t_fresh VALUES (2, 'b', 12);
INSERT INTO t_overwrite VALUES
(1, 'a', 11),
(2, 'b', 12);
INSERT OVERWRITE t_overwrite PARTITION (k='a')
SELECT id, v
FROM (VALUES (3, 13)) AS t(id, v)
WHERE id = 0;
SELECT id, k, v FROM t_fresh ORDER BY id;
SELECT id, k, v FROM t_overwrite ORDER BY id;
{code}
The overwrite query produces zero rows, while the static PARTITION clause
explicitly targets k='a'.
h2. Expected Result
Both tables should contain only:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml(2,
b, 12)
{code}
Partition k='a' should contain no rows, while partition k='b' should remain
unchanged.
h2. Actual Result
The INSERT OVERWRITE statement completes without an error.
The fresh table returns:
{code}
(2, b, 12)
{code}
However, the overwritten table still returns:
{code}
(1, a, 11)
(2, b, 12)
{code}
The old data file under partition k='a' remains after the overwrite.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)