maryannxue commented on a change in pull request #33541:
URL: https://github.com/apache/spark/pull/33541#discussion_r678571949
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEShuffleReadExec.scala
##########
@@ -69,6 +70,21 @@ case class AQEShuffleReadExec private(
case _ =>
throw new IllegalStateException("operating on canonicalization plan")
}
+ } else if (isCoalescedRead) {
+ // For coalesced shuffle read, the data distribution is not changed,
only the number of
+ // partitions is changed.
+ child.outputPartitioning match {
+ case h: HashPartitioning =>
+ CurrentOrigin.withOrigin(h.origin)(h.copy(numPartitions =
partitionSpecs.length))
+ case r: RangePartitioning =>
+ CurrentOrigin.withOrigin(r.origin)(r.copy(numPartitions =
partitionSpecs.length))
+ // This can only happen for `REBALANCE_PARTITIONS_BY_NONE`, which uses
+ // `RoundRobinPartitioning` but we don't need to retain the number of
partitions.
+ case r: RoundRobinPartitioning =>
+ r.copy(numPartitions = partitionSpecs.length)
+ case other => throw new IllegalStateException(
Review comment:
too strict? can user-specified partitioning appear here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
