agubichev commented on code in PR #45125:
URL: https://github.com/apache/spark/pull/45125#discussion_r1514771822
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:
##########
@@ -34,7 +34,7 @@ import
org.apache.spark.sql.catalyst.trees.TreePattern.{COMMON_EXPR_REF, WITH_EX
*/
object RewriteWithExpression extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan = {
- plan.transformWithPruning(_.containsPattern(WITH_EXPRESSION)) {
+
plan.transformDownWithSubqueriesAndPruning(_.containsPattern(WITH_EXPRESSION)) {
Review Comment:
To be more precise, this PR deals with the following subqueries:
S:
Aggregate (mayhavecountbug = true)
|
<subquery body>
Before this PR, OptimizeSubqueries applied on S, with this PR
OptimizeSubqueries applies on <subquery body>.
After we unnest the subquery, the rules apply on the entire query (including
the aggregate), so we don't miss any optimization opportunities by excluding
Aggregate in OptimizeSubqueries.
However, a rule RewriteWithExpressions is done before OptimizeSubqueries, so
we need to make sure subqueries are handled there (in case Aggregate node has a
WITH expresion in it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
