Hisoka-X commented on code in PR #42194:
URL: https://github.com/apache/spark/pull/42194#discussion_r1291988730
##########
core/src/main/scala/org/apache/spark/Partitioner.scala:
##########
@@ -137,6 +137,18 @@ private[spark] class PartitionIdPassthrough(override val
numPartitions: Int) ext
override def getPartition(key: Any): Int = key.asInstanceOf[Int]
}
+/**
+ * A [[org.apache.spark.Partitioner]] that partitions all records use
partition value map
+ */
+private[spark] class PartitionValueMapPartitioner(
+ valueMap: mutable.Map[Seq[Any], Int],
Review Comment:
At now `InternalRowComparableWrapper` not supported `Serializable` , so
can't use it as `KeyGroupedPartitioner` fields. This is the only way until
there is a better solution.
```scala
Serialization stack:
- object not serializable (class:
org.apache.spark.sql.catalyst.expressions.GeneratedClass, value:
org.apache.spark.sql.catalyst.expressions.GeneratedClass@560b0bb7)
- field (class:
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering,
name: this$0, type: class
org.apache.spark.sql.catalyst.expressions.GeneratedClass)
- object (class
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering,
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering@1eff706)
- field (class:
org.apache.spark.sql.catalyst.util.InternalRowComparableWrapper, name:
ordering, type: class org.apache.spark.sql.catalyst.expressions.BaseOrdering)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
