Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20541#discussion_r166870474
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -400,13 +400,24 @@ object PushProjectionThroughUnion extends
Rule[LogicalPlan] with PredicateHelper
// Push down deterministic projection through UNION ALL
case p @ Project(projectList, Union(children)) =>
assert(children.nonEmpty)
- if (projectList.forall(_.deterministic)) {
- val newFirstChild = Project(projectList, children.head)
+ val (deterministicList, nonDeterministic) =
projectList.partition(_.deterministic)
+
+ if (deterministicList.nonEmpty) {
+ val newFirstChild = Project(deterministicList, children.head)
val newOtherChildren = children.tail.map { child =>
val rewrites = buildRewrites(children.head, child)
- Project(projectList.map(pushToRight(_, rewrites)), child)
+ Project(deterministicList.map(pushToRight(_, rewrites)), child)
--- End diff --
do we push `a + 1` to union children? or just `a`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
