Francesco Guardiani created FLINK-25918: -------------------------------------------
Summary: Use FileEnumerator to implement filter pushdown of filepath metadata Key: FLINK-25918 URL: https://issues.apache.org/jira/browse/FLINK-25918 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Reporter: Francesco Guardiani Right now, unless you configure partition keys, the table file source will ingest all the files in the provided {{path}}. Which means that a query like: {code:sql} SELECT * FROM MyFileTable WHERE filepath LIKE "%.csv" {code} Will ingest all the files and then, after the records are loaded in flink, the filtering happens and discards all the records not coming from a file with pattern "%.csv". Using the filter push down feature provided by the DynamicTableSource stack, we could instead provide the {{FileSourceBuilder}} directly a {{FileEnumerator}} that does the filtering of input files, so we can effectively skip reading them. -- This message was sent by Atlassian Jira (v8.20.1#820001)