manuzhang commented on a change in pull request #29797:
URL: https://github.com/apache/spark/pull/29797#discussion_r491281977
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
##########
@@ -705,7 +705,8 @@ abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
exchange.ShuffleExchangeExec(
r.partitioning,
planLater(r.child),
- noUserSpecifiedNumPartition = r.optNumPartitions.isEmpty) :: Nil
+ noUserSpecifiedNumPartition = conf.coalesceShufflePartitionsEnabled
&&
+ r.optNumPartitions.isEmpty) :: Nil
Review comment:
@viirya Thanks for the good questions.
This solution is not ideal till we find a way not to apply local shuffle
reader if the partitioning doesn't match that of dynamic partition. It's not
ideal either that any AQE config is global.
With `coalesceShufflePartitionsEnabled` and `localShuffleReaderEnabled`, the
output partitioning of shuffle is uncertain which arguably contradicts the
purpose of `RepartitionByExpression`.
> If the user needs to coalesce shuffle partition in the query, but also
needs dynamic partition overwrite?
Another option is to introduce a new config specific for repartition, e.g.
`spark.sql.adaptive.repartition.canChangeNumPartitions`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
