mskapilks commented on PR #40266: URL: https://github.com/apache/spark/pull/40266#issuecomment-1469424781
@wangyum @peter-toth Thanks for pointing on previous attempts. It does seem moving `RewritePredicateSubquery` rule is right way so that in future we don't add anymore rule to that batch (`RewriteSubquery`). In this pr https://github.com/apache/spark/pull/17520, they tried to put RewritePredicateSubquery right after `Subquery` batch (of `OptimizeSubqueries`). `operatorOptimizationBatch` will run after this. They also added one rule to push LeftSemi/LeftAnti through join, but that has been added in 3.0 by [SPARK-19712](https://issues.apache.org/jira/browse/SPARK-19712). So now we only need to change rule position. If this seems right to you guys, I can update this PR to move `RewritePredicateSubquery` after `Subqury` batch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
