zhztheplayer commented on code in PR #10838:
URL:
https://github.com/apache/incubator-gluten/pull/10838#discussion_r2432205362
##########
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##########
@@ -298,27 +298,13 @@ case class WholeStageTransformer(child: SparkPlan,
materializeInput: Boolean = f
wsCtx: WholeStageTransformContext,
inputRDDs: ColumnarInputRDDsWrapper,
pipelineTime: SQLMetric): RDD[ColumnarBatch] = {
- val isKeyGroupPartition = leafTransformers.exists {
- // TODO: May can apply to BatchScanExecTransformer without key group
partitioning
- case b: BatchScanExecTransformerBase if
b.keyGroupedPartitioning.isDefined => true
- case _ => false
- }
/**
* If containing leaf exec transformer this "whole stage" generates a RDD
which itself takes
* care of [[LeafTransformSupport]] there won't be any other RDD for leaf
operator. As a result,
* genFirstStageIterator rather than genFinalStageIterator will be invoked
*/
- val allInputPartitions = leafTransformers.map(
- leafTransformer => {
- if (isKeyGroupPartition) {
-
leafTransformer.asInstanceOf[BatchScanExecTransformerBase].getPartitionsWithIndex
- } else {
- Seq(leafTransformer.getPartitions)
- }
- })
-
- val allSplitInfos = getSplitInfosFromPartitions(isKeyGroupPartition,
leafTransformers)
+ val allSplitInfos = leafTransformers.map(_.getSplitInfos).transpose
Review Comment:
Do we need to preserve a similar comment like
[this](https://github.com/apache/incubator-gluten/blob/84f040f5fd57e3940bfba17713dd55ba47287224/gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala#L373-L408)?
Which will help user understand the `.transpose` usage here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
