c21 commented on a change in pull request #35574:
URL: https://github.com/apache/spark/pull/35574#discussion_r810572744
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
##########
@@ -271,6 +279,17 @@ case class HashPartitioning(expressions: Seq[Expression],
numPartitions: Int)
override def createShuffleSpec(distribution: ClusteredDistribution):
ShuffleSpec =
HashShuffleSpec(this, distribution)
+ /**
+ * Checks if [[HashPartitioning]] is partitioned on exactly same full
`clustering` keys of
+ * [[ClusteredDistribution]].
+ */
+ def isPartitionedOnFullKeys(distribution: ClusteredDistribution): Boolean = {
+ expressions.length == distribution.clustering.length &&
Review comment:
@HeartSaVioR - this is a good point. I am more inclined to more
restricted condition, as `hash(x1, x2)` will send rows in different partitions
compared to `hash(x2, x1)`, and potentially it can have data skew in one case,
but not in the other. Before me update the doc, cc more folks for commenting,
@cloud-fan and @sunchao.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
