Hisoka-X commented on code in PR #42194:
URL: https://github.com/apache/spark/pull/42194#discussion_r1287859433
##########
core/src/main/scala/org/apache/spark/Partitioner.scala:
##########
@@ -137,6 +137,18 @@ private[spark] class PartitionIdPassthrough(override val
numPartitions: Int) ext
override def getPartition(key: Any): Int = key.asInstanceOf[Int]
}
+/**
+ * A [[org.apache.spark.Partitioner]] that partitions all records use
partition value map
+ */
+private[spark] class PartitionValueMapPartitioner(
+ valueMap: mutable.Map[Seq[Any], Int],
+ override val numPartitions: Int) extends Partitioner {
+ override def getPartition(key: Any): Int = {
+ val keys = key.asInstanceOf[Seq[Any]]
+ valueMap.getOrElseUpdate(keys, Utils.nonNegativeMod(keys.hashCode,
numPartitions))
Review Comment:
> When do we need to update this map?
When the shuffle side (without KeyGroupedPartitioning side) have some
partition value not in KeyGroupedPartitioning. This is a fallback operation to
ensure that all data will be shuffled. In fact we can add Filter to avoid this
kind data to be shuffled. We can do it later.
> will this collide with one of the existing mappings?
no, just update not existed key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
