wangyum commented on a change in pull request #35214:
URL: https://github.com/apache/spark/pull/35214#discussion_r787589160
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1322,6 +1322,12 @@ object CombineUnions extends Rule[LogicalPlan] {
case Union(children, byName, allowMissingCol)
if byName == topByName && allowMissingCol == topAllowMissingCol =>
stack.pushAll(children.reverse)
+ case Project(projectList, Distinct(u @ Union(children, byName,
allowMissingCol)))
+ if projectList.forall(_.deterministic) && children.nonEmpty &&
+ flattenDistinct && byName == topByName && allowMissingCol ==
topAllowMissingCol =>
+ val newChildren =
PushProjectionThroughUnion.pushProjectionThroughUnion(projectList, u)
+ .map(CollapseProject(_))
Review comment:
We will put all children to stack and use it later. For example, if we
do not collapse projects. we can not [extract union‘s
children](https://github.com/apache/spark/blob/ab18cc8a7d3df476f045f84bae4ff121eacc91b9/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1326-L1343):
```
Project [cast(id#34 as decimal(22,5)) AS id#36]
+- Project [cast(id#32 as decimal(21,4)) AS id#34]
+- Distinct
+- Union false, false
:- Project [cast(id#30 as decimal(20,3)) AS id#32]
: +- Distinct
: +- Union false, false
: :- Project [cast(id#25 as decimal(19,2)) AS id#30]
: : +- Relation default.t1[id#25] parquet
: +- Project [cast(id#26 as decimal(19,2)) AS id#31]
: +- Relation default.t2[id#26] parquet
+- Project [cast(id#27 as decimal(20,3)) AS id#33]
+- Relation default.t3[id#27] parquet
```
And later we will do `ReplaceDistinctWithAggregate` and then `CombineUnions`
can not combine any unions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
