Yaoxuan Wu created FLINK-41181:
----------------------------------
Summary: Filesystem sink silently drops all rows when the last
partition column name starts with '_' or '.'
Key: FLINK-41181
URL: https://issues.apache.org/jira/browse/FLINK-41181
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 2.2.1, 2.3.0, 2.0.1
Reporter: Yaoxuan Wu
{code:sql}
CREATE TABLE m (i INT, `_p` STRING) PARTITIONED BY (`_p`)
WITH ('connector' = 'filesystem', 'path' = 'file:///tmp/m', 'format' =
'json');
INSERT INTO m VALUES (1, 'a'), (2, 'b'); -- finishes without an error
SELECT * FROM m;
-- expected: (1, a), (2, b)
-- actual: no rows; /tmp/m is empty
{code}
Batch mode, any format. The same for a column named {{.p}}, and for
{{PARTITIONED BY (p, _q)}}; {{PARTITIONED BY (_p, q)}}, {{p}} and {{a_p}} work.
Cause: {{FileSystemCommitter.commitPartitions}} finds the written partitions in
the staging directory with {{PartitionPathUtils.searchPartSpecAndPaths}}, which
skips every leaf directory whose name starts with {{_}} or {{.}}
({{isHiddenFile}}). A partition directory is named {{<column>=<value>}}, so
with such a last partition column every partition is skipped, nothing is moved
to the table path, and the staging directory is deleted in the {{finally}}
block. The job reports success. The DDL should reject such partition column
names, or the hidden-file check should not apply to {{<partition column>=}}
directories.
Reproduces on 2.0.1, 2.2.1, 2.3.0 and master.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
