szehon-ho commented on code in PR #45267:
URL: https://github.com/apache/spark/pull/45267#discussion_r1529562057
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:
##########
@@ -505,11 +506,28 @@ case class EnsureRequirements(
}
}
- // Now we need to push-down the common partition key to the scan in
each child
- newLeft = populatePartitionValues(left, mergedPartValues,
leftSpec.joinKeyPositions,
- applyPartialClustering, replicateLeftSide)
- newRight = populatePartitionValues(right, mergedPartValues,
rightSpec.joinKeyPositions,
- applyPartialClustering, replicateRightSide)
+ // in case of compatible but not identical partition expressions, we
apply 'reduce'
+ // transforms to group one side's partitions as well as the common
partition values
+ val leftReducers = leftSpec.reducers(rightSpec)
+ val rightReducers = rightSpec.reducers(leftSpec)
+
+ if (leftReducers.isDefined || rightReducers.isDefined) {
Review Comment:
Yea, or in other words both sides mod the gcd. Ie bucket(16, x) % 4 and
bucket(12, y) % 4.
Implemented and added test case in latest patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
