Yaroslav Tkachenko created FLINK-30873:
------------------------------------------
Summary: FileSystem Table API connector doesn't discover new files
when partitions are specified
Key: FLINK-30873
URL: https://issues.apache.org/jira/browse/FLINK-30873
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.16.1
Reporter: Yaroslav Tkachenko
When *source.monitor-interval* is configured I expect the source connector to
discover new files. However, if the source table was created with partitions
(using PARTITIONED BY), it only discovers new files in the partition folders
that existed during the startup; it doesn't discover new partitions.
I believe the problem is
[here|https://github.com/apache/flink/blob/5f2d088a2713ced5c6ce072db92f4378f73bc739/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSource.java#L276-L286],
only these file paths are monitored internally, no partition discovery is
currently implemented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
