sunchao commented on a change in pull request #35574:
URL: https://github.com/apache/spark/pull/35574#discussion_r812511093
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -56,7 +57,23 @@ case class EnsureRequirements(
// Ensure that the operator's children satisfy their output distribution
requirements.
var children = originalChildren.zip(requiredChildDistributions).map {
case (child, distribution) if
child.outputPartitioning.satisfies(distribution) =>
- child
+ (child.outputPartitioning, distribution) match {
+ case (p: HashPartitioning, d: ClusteredDistribution) =>
+ if
(conf.getConf(SQLConf.REQUIRE_ALL_CLUSTER_KEYS_FOR_SOLE_PARTITION) &&
+ requiredChildDistributions.size == 1 &&
!p.isPartitionedOnFullKeys(d)) {
+ // Add an extra shuffle for `ClusteredDistribution` even though
its child
Review comment:
```
if (partitionSpec.exists(_.dataType.isInstanceOf[BooleanType])) {
HashClusteredDistribution(partitionSpec) :: Nil
} else {
ClusteredDistribution(partitionSpec) :: Nil
}
```
I'm not sure if this is a good example, since you also need to check
`child.outputPartitioning`, right? otherwise, the partition keys in the child
output partitioning may not contain boolean at all but you'd still require a
full partition key match, which may not be optimal. For instance:
```
WindowExec(partitionSpec=(x: bool, y: int, z: long))
- child partitioned by y, z
```
So in the end, it seems like this is not only expressing requirements but
also doing some sort of matching similar to what's done in `EnsureRequirements`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
