tanelk commented on a change in pull request #31740:
URL: https://github.com/apache/spark/pull/31740#discussion_r588885759
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -946,6 +947,45 @@ object TransposeWindow extends Rule[LogicalPlan] {
}
}
+/**
+ * Replaces duplicate window expressions with an alias in a Project above the
Window node.
Review comment:
I see, thanks for the reference.
Indeed, the common subexpression elimination seems to eliminate these
duplicates (if I did follow the code correctly).
I would argue, that there still is benefit in removing these duplicates in
the optimizer. If nothing else, then it at least would clean up the plan.
Also the subexpression cache has a limited size (although configurable) and
this would reduce the runtime overhead of cache lookups.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
