This is an automated email from the ASF dual-hosted git repository.
yangzy pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new c85db55d5 [GLUTEN-4999] Fix ColumnarUnionExec to get
PartitionerAwareUnionRDD used if child RDDs share same partitioner (#5021)
c85db55d5 is described below
commit c85db55d535ea953f046511d3782334f712f2517
Author: guixiaowen <[email protected]>
AuthorDate: Wed Mar 20 12:38:32 2024 +0800
[GLUTEN-4999] Fix ColumnarUnionExec to get PartitionerAwareUnionRDD used if
child RDDs share same partitioner (#5021)
For example:
select * from test a
union all
select * from test b
union all
select * from test c
They have the same partitioner.
In ColumnarUnionExec, they will be transformed to
rdd1.union(rdd2).union(rdd3).
After this pr,
In ColumnarUnionExec, they will be transformed to
PartitionerAwareUnionRDD(sc, Seq(rdd1, rdd2, rdd3)) if they have the same
partitioner.
---
.../glutenproject/execution/BasicPhysicalOperatorTransformer.scala | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git
a/gluten-core/src/main/scala/io/glutenproject/execution/BasicPhysicalOperatorTransformer.scala
b/gluten-core/src/main/scala/io/glutenproject/execution/BasicPhysicalOperatorTransformer.scala
index 75da80cb5..9fa252016 100644
---
a/gluten-core/src/main/scala/io/glutenproject/execution/BasicPhysicalOperatorTransformer.scala
+++
b/gluten-core/src/main/scala/io/glutenproject/execution/BasicPhysicalOperatorTransformer.scala
@@ -327,10 +327,7 @@ case class ColumnarUnionExec(children: Seq[SparkPlan])
extends SparkPlan with Gl
if (children.isEmpty) {
throw new IllegalArgumentException(s"Empty children")
}
- children
- .map(c => Seq(c.executeColumnar()))
- .reduce((a, b) => a ++ b)
- .reduce((a, b) => a.union(b))
+ sparkContext.union(children.map(c => c.executeColumnar()))
}
override protected def doExecute()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
