karuppayya commented on a change in pull request #28424:
URL: https://github.com/apache/spark/pull/28424#discussion_r421686884
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1553,13 +1553,84 @@ object ReplaceDeduplicateWithAggregate extends
Rule[LogicalPlan] {
* 1. This rule is only applicable to INTERSECT DISTINCT. Do not use it for
INTERSECT ALL.
* 2. This rule has to be done after de-duplicating the attributes; otherwise,
the generated
* join conditions will be incorrect.
+ *
+ * This rule also pushed down the Distinct to left/right side based on the
config
Review comment:
1. Can we mention that CBO needs to be enabled?
2. Also column stats is required, is this right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
