maryannxue commented on a change in pull request #33541:
URL: https://github.com/apache/spark/pull/33541#discussion_r678793485
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -32,8 +32,14 @@ import
org.apache.spark.sql.execution.joins.{ShuffledHashJoinExec, SortMergeJoin
* [[org.apache.spark.sql.catalyst.plans.physical.Distribution Distribution]]
requirements for
* each operator by inserting [[ShuffleExchangeExec]] Operators where
required. Also ensure that
* the input partition ordering requirements are met.
+ *
+ * @param optimizeOutRepartitionByCol A flag to indicate that if this rule
should optimize out
+ * user-specified repartition-by-col
shuffles or not. This is
+ * mostly true, but can be false in AQE
when AQE optimization
+ * may change the plan output partitioning
and need to retain
+ * the user-specified repartition-by-col
shuffles in the plan.
*/
-object EnsureRequirements extends Rule[SparkPlan] {
+case class EnsureRequirements(optimizeOutRepartitionByCol: Boolean = true)
extends Rule[SparkPlan] {
Review comment:
nit: `optimizeOutRepartitionShuffle`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
