maropu commented on a change in pull request #30373:
URL: https://github.com/apache/spark/pull/30373#discussion_r523421228
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
##########
@@ -343,8 +343,8 @@ object QueryExecution {
PlanDynamicPruningFilters,
PlanSubqueries,
RemoveRedundantProjects,
- RemoveRedundantSorts,
EnsureRequirements,
+ RemoveRedundantSorts,
Review comment:
Could you leave some comments here about why we need to put this rule
after `EnsureRequirements`?
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala
##########
@@ -135,6 +135,26 @@ abstract class RemoveRedundantSortsSuiteBase
}
}
}
+
+ test("shuffled join with different left and right side partition numbers") {
Review comment:
nit: could you add the prefix: `SPARK-33183: `
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/RemoveRedundantSortsSuite.scala
##########
@@ -135,6 +135,26 @@ abstract class RemoveRedundantSortsSuiteBase
}
}
}
+
+ test("shuffled join with different left and right side partition numbers") {
+ withTempView("t1", "t2") {
+ spark.range(0, 100, 1, 2).select('id as
"key").createOrReplaceTempView("t1")
+ (0 to 100).toDF("key").createOrReplaceTempView("t2")
+
+ // left side partitioning: RangePartitioning(key ASC, 2)
+ // right side partitioning: UnknownPartitioning(0)
Review comment:
Could you add `assert` to check if the query below has the output
partitions above?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
