HyukjinKwon commented on a change in pull request #32932:
URL: https://github.com/apache/spark/pull/32932#discussion_r656793880
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
##########
@@ -1351,6 +1351,28 @@ object RepartitionByExpression {
}
}
+/**
+ * This operator does not guarantee the output partitioning, because the
partition number will be
+ * optimized by AQE.
+ */
+case class AdaptiveRepartition(
+ partitionExpressions: Seq[Expression],
+ child: LogicalPlan) extends UnaryNode {
+ override def maxRows: Option[Long] = child.maxRows
+ override def output: Seq[Attribute] = child.output
+
+ lazy val numPartitions: Int = conf.numShufflePartitions
+
+ def partitioning: Partitioning = if (partitionExpressions.nonEmpty) {
+ HashPartitioning(partitionExpressions, numPartitions)
Review comment:
Does it 1. expand to that static number first and 2. attempt to coalesce
with 3. falling back to the local shuffle readers?
Or does AQE has a logic to automatically figure out the number fo expand?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
