[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r643628995
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
##
@@ -165,6 +170,23 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with
PredicateHelper {
case f @ Filter(condition, j @ Join(_, _, RightOuter | LeftOuter |
FullOuter, _, _)) =>
val newJoinType = buildNewJoinType(f, j)
if (j.joinType == newJoinType) f else Filter(condition, j.copy(joinType
= newJoinType))
+
+case a @ Aggregate(_, _, join @ Join(left, _, LeftOuter, _, _))
+if a.isDistinct && a.references.subsetOf(AttributeSet(left.output)) &&
+ !canPlanAsBroadcastHashJoin(join, conf) =>
Review comment:
The aggregate should still be there. I mean we can remove this
`canPlanAsBroadcastHashJoin` check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r643622282
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
##
@@ -165,6 +170,23 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with
PredicateHelper {
case f @ Filter(condition, j @ Join(_, _, RightOuter | LeftOuter |
FullOuter, _, _)) =>
val newJoinType = buildNewJoinType(f, j)
if (j.joinType == newJoinType) f else Filter(condition, j.copy(joinType
= newJoinType))
+
+case a @ Aggregate(_, _, join @ Join(left, _, LeftOuter, _, _))
+if a.isDistinct && a.references.subsetOf(AttributeSet(left.output)) &&
+ !canPlanAsBroadcastHashJoin(join, conf) =>
Review comment:
Sorry I missed this. It's outer join and it will never reduce data
volume of the left side. So we can always remove the join, no matter it's
broadcast or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r640567454
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/AggregateOptimizeSuite.scala
##
@@ -71,4 +74,65 @@ class AggregateOptimizeSuite extends AnalysisTest {
comparePlans(optimized, correctAnswer)
}
+
+ test("SPARK-34808: Remove left join if it only has distinct on left side") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, LeftOuter, Some("x.a".attr ===
"y.a".attr)).select("x.b".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = if (autoBroadcastJoinThreshold < 0) {
+ x.select("x.b".attr).groupBy("x.b".attr)("x.b".attr)
+} else {
+ Aggregate(query.child.output, query.child.output, query.child)
+}
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: Remove right join if it only has distinct on right side")
{
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, RightOuter, Some("x.a".attr ===
"y.a".attr)).select("y.b".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = if (autoBroadcastJoinThreshold < 0) {
+ y.select("y.b".attr).groupBy("y.b".attr)("y.b".attr)
+} else {
+ Aggregate(query.child.output, query.child.output, query.child)
+}
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: Should not remove left join if select 2 join sides") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, RightOuter, Some("x.a".attr === "y.a".attr))
+ .select("x.b".attr, "y.c".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = Aggregate(query.child.output, query.child.output,
query.child)
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: EliminateOuterJoin must before
RemoveRepetitionFromGroupExpressions") {
Review comment:
let's address
https://github.com/apache/spark/pull/31908#discussion_r640359116 then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r640359116
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -659,6 +659,12 @@ case class Aggregate(
val nonAgg =
aggregateExpressions.filter(_.find(_.isInstanceOf[AggregateExpression]).isEmpty)
getAllValidConstraints(nonAgg)
}
+
+ // Whether this Aggregate operator is equally the Distinct operator.
+ private[sql] def isEquallyDistinct: Boolean = {
+groupingExpressions.size == aggregateExpressions.size &&
Review comment:
I think we only need `aggregateExpressions` to only contains grouping
columns. @wangyum can you refine it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r640358612
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/AggregateOptimizeSuite.scala
##
@@ -71,4 +74,65 @@ class AggregateOptimizeSuite extends AnalysisTest {
comparePlans(optimized, correctAnswer)
}
+
+ test("SPARK-34808: Remove left join if it only has distinct on left side") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, LeftOuter, Some("x.a".attr ===
"y.a".attr)).select("x.b".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = if (autoBroadcastJoinThreshold < 0) {
+ x.select("x.b".attr).groupBy("x.b".attr)("x.b".attr)
+} else {
+ Aggregate(query.child.output, query.child.output, query.child)
+}
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: Remove right join if it only has distinct on right side")
{
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, RightOuter, Some("x.a".attr ===
"y.a".attr)).select("y.b".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = if (autoBroadcastJoinThreshold < 0) {
+ y.select("y.b".attr).groupBy("y.b".attr)("y.b".attr)
+} else {
+ Aggregate(query.child.output, query.child.output, query.child)
+}
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: Should not remove left join if select 2 join sides") {
+val x = testRelation.subquery('x)
+val y = testRelation.subquery('y)
+val query = Distinct(x.join(y, RightOuter, Some("x.a".attr === "y.a".attr))
+ .select("x.b".attr, "y.c".attr))
+
+Seq(-1, 1).foreach { autoBroadcastJoinThreshold =>
+ withSQLConf(AUTO_BROADCASTJOIN_THRESHOLD.key ->
s"$autoBroadcastJoinThreshold") {
+val correctAnswer = Aggregate(query.child.output, query.child.output,
query.child)
+comparePlans(Optimize.execute(query.analyze), correctAnswer.analyze)
+ }
+}
+ }
+
+ test("SPARK-34808: EliminateOuterJoin must before
RemoveRepetitionFromGroupExpressions") {
Review comment:
why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r638585140
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
##
@@ -165,6 +170,19 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with
PredicateHelper {
case f @ Filter(condition, j @ Join(_, _, RightOuter | LeftOuter |
FullOuter, _, _)) =>
val newJoinType = buildNewJoinType(f, j)
if (j.joinType == newJoinType) f else Filter(condition, j.copy(joinType
= newJoinType))
+
+case a @ Aggregate(_, _, Join(left, _, LeftOuter, _, _))
Review comment:
If we treat left join as a filter, then removing the filter may lose the
chance to reduce data volume and cause perf regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r638114867
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##
@@ -844,6 +844,12 @@ case class Aggregate(
override protected def withNewChildInternal(newChild: LogicalPlan):
Aggregate =
copy(child = newChild)
+
+ // Whether this Aggregate operator is equally the Distinct operator.
+ private[sql] def isEquallyDistinct: Boolean = {
+groupingExpressions.size == aggregateExpressions.size &&
+ groupingExpressions.zip(aggregateExpressions).forall(e =>
e._1.fastEquals(e._2))
Review comment:
I'd use `semanticEqual` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r638114676
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
##
@@ -165,6 +170,19 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with
PredicateHelper {
case f @ Filter(condition, j @ Join(_, _, RightOuter | LeftOuter |
FullOuter, _, _)) =>
val newJoinType = buildNewJoinType(f, j)
if (j.joinType == newJoinType) f else Filter(condition, j.copy(joinType
= newJoinType))
+
+case a @ Aggregate(_, _, Join(left, _, LeftOuter, _, _))
+if a.isEquallyDistinct &&
a.references.subsetOf(AttributeSet(left.output)) =>
Review comment:
`isEquallyDistinct` looks weird, how about `isDistinct`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r603145189
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateUnnecessaryOuterJoin.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.plans.{LeftOuter, RightOuter}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Removes outer join if it only has distinct on streamed side.
+ */
+object EliminateUnnecessaryOuterJoin extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case a @ Aggregate(_, _, p @ Project(_, Join(left, _, LeftOuter, _, _)))
Review comment:
ping @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r598809303
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateUnnecessaryOuterJoin.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.plans.{LeftOuter, RightOuter}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Removes outer join if it only has distinct on streamed side.
+ */
+object EliminateUnnecessaryOuterJoin extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case a @ Aggregate(_, _, p @ Project(_, Join(left, _, LeftOuter, _, _)))
Review comment:
When this happens, column pruning will insert a Project between them. So
I agree that the code here is correct in reality. But the code here assumes
that column pruning will kick in first, and it's not necessary to have this
assumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r598714293
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateUnnecessaryOuterJoin.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.plans.{LeftOuter, RightOuter}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Removes outer join if it only has distinct on streamed side.
+ */
+object EliminateUnnecessaryOuterJoin extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case a @ Aggregate(_, _, p @ Project(_, Join(left, _, LeftOuter, _, _)))
Review comment:
why not? `Aggregate` defines its output and it won't change with the
child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r598566142
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateUnnecessaryOuterJoin.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.plans.{LeftOuter, RightOuter}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Removes outer join if it only has distinct on streamed side.
+ */
+object EliminateUnnecessaryOuterJoin extends Rule[LogicalPlan] {
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+case a @ Aggregate(_, _, p @ Project(_, Join(left, _, LeftOuter, _, _)))
Review comment:
what if there is no Project between Aggregate and Join?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31908: [SPARK-34808][SQL] Removes outer join if it only has DISTINCT on streamed side
cloud-fan commented on a change in pull request #31908:
URL: https://github.com/apache/spark/pull/31908#discussion_r598565769
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/EliminateUnnecessaryOuterJoin.scala
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.plans.{LeftOuter, RightOuter}
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Join,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * Removes outer join if it only has distinct on streamed side.
Review comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
