Aitozi opened a new issue, #2127: URL: https://github.com/apache/incubator-paimon/issues/2127
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Motivation Currently, Paimon Flink source do not implement `SupportsPartitionPushdown`, and rely on filter push down to prune the unrelated partition. However, this brings two optimization bad case: - Currently, the paimon's `SupportsFilterPushDown` do not consume the filter expression after applyFilters. ```java @Override public Result applyFilters(List<ResolvedExpression> filters) { source.pushFilters(filters); return Result.of(filters, filters); } ``` So in the case below, the limit can not be pushed into scan. ```sql SELECT * FROM T where a = 1 limit 3 ``` <img width="693" alt="image" src="https://github.com/apache/incubator-paimon/assets/9486140/1bf95bda-8c82-45e3-8c7a-a40ec2bc0c87";> - Using filter push down to prune partition is not as effective as partition push down. For example, the filter condition '%pattern%' can not generate a valid partition predicate. And, in partition push down, some complex condition on partition field even function call condition is workable. So, I proposal to support partition push down for paimon flink source, for more effective partition pruning. ### Solution _No response_ ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
