pan3793 commented on code in PR #57762:
URL: https://github.com/apache/spark/pull/57762#discussion_r3718382676
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/exchange/EnsureRequirementsSuite.scala:
##########
@@ -1016,6 +1001,32 @@ class EnsureRequirementsSuite extends SharedSparkSession
{
}
}
+ test("KeyedPartitioning: duplicated join keys do not block SPJ") {
+ // The coverage check of requireAllClusterKeysForCoPartition ignores key
order and
+ // duplicated cluster keys: join keys [a, b, b] are fully covered by
partition keys
+ // on [a, b], so SPJ is allowed with either config value.
+ val plan1 = new DummySparkPlanWithBatchScanChild(
+ outputPartitioning =
+ KeyedPartitioning(bucket(4, exprA) :: bucket(4, exprB) :: Nil,
Seq.empty))
+ val plan2 = new DummySparkPlanWithBatchScanChild(
+ outputPartitioning =
+ KeyedPartitioning(bucket(4, exprA) :: bucket(4, exprC) :: Nil,
Seq.empty))
+ val smjExec = SortMergeJoinExec(
+ exprA :: exprB :: exprB :: Nil, exprA :: exprC :: exprC :: Nil, Inner,
None, plan1, plan2)
Review Comment:
Added both tests in 66ae049fc14: the `[a, b]` / `[a, c]` join against the
`(years(a), bucket(4, b), days(a))` partitionings in `EnsureRequirementsSuite`,
and the end-to-end `PARTITIONED BY (bucket(8, id), identity(id))` join in
`KeyGroupedPartitioningSuite`, both with default configs. Verified both fail
against the base gate and pass with the coverage check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
