beyond1920 commented on PR #8102: URL: https://github.com/apache/hudi/pull/8102#issuecomment-1483842164
@danny0405 Thanks for your review. I want to explain why I introduced a new `HoodieContinuousPartitionTableSource` class and a configuration to determine whether this improvement is enabled. Currently, Flink's partition pruning mechanism has two limitations: 1. The partition pruning is happened in optimization phase. The runtime will not perform a subsequent partition pruning. 2. Once partition pruning is done, Flink optimizer removes the partition-related filter conditions from the filter conditions, so it is no longer possible to access the partition-related filter conditions in the later filter pruning phase. So The new `TableSource` could not be a `SupportsPartitionPushDown` tableSource to ensure it could fetch all the filter conditions of the SQL query. But currently it could not support complex filter conditions like Flink native partition pruning optimization, such as the filter conditions contains UDF and so on. It might cause some complex query statements to execute more slowly. That's the reason I want to introduce a new class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
