[
https://issues.apache.org/jira/browse/FLINK-30873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683320#comment-17683320
]
Mate Czagany commented on FLINK-30873:
--------------------------------------
If you're not working on it yet and a committer could assign this to me, I'd
gladly pick it up. I looked at it briefly and as you said currently
FileSystemTableSource initially finds all partition directories in the root
path and uses those to initialize FileSource.
By only passing the root path to FileSource it seems to solve this shortcoming,
but it definitely needs extensive testing to ensure it doesn't break current
functionality.
> FileSystem Table API connector doesn't discover new files when partitions are
> specified
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-30873
> URL: https://issues.apache.org/jira/browse/FLINK-30873
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.16.1
> Reporter: Yaroslav Tkachenko
> Priority: Major
>
> When *source.monitor-interval* is configured I expect the source connector to
> discover new files. However, if the source table was created with partitions
> (using PARTITIONED BY), it only discovers new files in the partition folders
> that existed during the startup; it doesn't discover new partitions.
> I believe the problem is
> [here|https://github.com/apache/flink/blob/5f2d088a2713ced5c6ce072db92f4378f73bc739/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSource.java#L276-L286],
> only these file paths are monitored internally, no partition discovery is
> currently implemented.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
