peter-toth commented on issue #23802: [SPARK-26893][SQL] Allow pushdown of partition pruning subquery filters to file source URL: https://github.com/apache/spark/pull/23802#issuecomment-464350647 > @peter-toth Thanks for replying the comments. > > I understood more about what this fix goes to do. #22518 removed subquery filters in `FileSourceStrategy` to avoid re-execution of the subquery when pushing down it to data source. But it also removes it from partition filters. And this PR proposes to add subquery back to partition filters. > > Can you update the PR description? I think the current description is unclear. Based on https://github.com/apache/spark/pull/22518/files#diff-7393042ad384133558061bee01daa3f7L166 subquery filters were never included in partition filtering. Sorry for the description, I will make it more detailed. But it seems @mgaido91 is right and somehow the subquery is executed 2 times though it is once in the plan. I want to understand that first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
