[
https://issues.apache.org/jira/browse/FLINK-41181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132600#comment-18132600
]
Shawn commented on FLINK-41181:
-------------------------------
[~gaborgsomogyi] I'd like to fix this so `<column>=value` partition directories
aren't skipped as hidden files. Could you assign it to me?
> Filesystem sink silently drops all rows when the last partition column name
> starts with '_' or '.'
> --------------------------------------------------------------------------------------------------
>
> Key: FLINK-41181
> URL: https://issues.apache.org/jira/browse/FLINK-41181
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 2.0.1, 2.3.0, 2.2.1
> Reporter: Yaoxuan Wu
> Priority: Major
>
> {code:sql}
> CREATE TABLE m (i INT, `_p` STRING) PARTITIONED BY (`_p`)
> WITH ('connector' = 'filesystem', 'path' = 'file:///tmp/m', 'format' =
> 'json');
> INSERT INTO m VALUES (1, 'a'), (2, 'b'); -- finishes without an error
> SELECT * FROM m;
> -- expected: (1, a), (2, b)
> -- actual: no rows; /tmp/m is empty
> {code}
> Batch mode, any format. The same for a column named {{.p}}, and for
> {{PARTITIONED BY (p, _q)}}; {{PARTITIONED BY (_p, q)}}, {{p}} and {{a_p}}
> work.
> Cause: {{FileSystemCommitter.commitPartitions}} finds the written partitions
> in the staging directory with {{PartitionPathUtils.searchPartSpecAndPaths}},
> which skips every leaf directory whose name starts with {{_}} or {{.}}
> ({{isHiddenFile}}). A partition directory is named {{<column>=<value>}}, so
> with such a last partition column every partition is skipped, nothing is
> moved to the table path, and the staging directory is deleted in the
> {{finally}} block. The job reports success. The DDL should reject such
> partition column names, or the hidden-file check should not apply to
> {{<partition column>=}} directories.
> Reproduces on 2.0.1, 2.2.1, 2.3.0 and master.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
