[
https://issues.apache.org/jira/browse/SPARK-18051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin resolved SPARK-18051.
---------------------------------
Resolution: Fixed
Assignee: Weichen Xu
Fix Version/s: 2.1.0
> Custom PartitionCoalescer cause serialization exception
> -------------------------------------------------------
>
> Key: SPARK-18051
> URL: https://issues.apache.org/jira/browse/SPARK-18051
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Weichen Xu
> Assignee: Weichen Xu
> Fix For: 2.1.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> for example, the following code cause exception:
> {code: title=code}
> class MyCoalescer extends PartitionCoalescer{
> override def coalesce(maxPartitions: Int, parent: RDD[_]):
> Array[PartitionGroup] = {
> val pglist = Array.fill(2)(new PartitionGroup())
> pglist(0).partitions.append(parent.partitions(0), parent.partitions(1),
> parent.partitions(2))
> pglist(1).partitions.append(parent.partitions(3), parent.partitions(4),
> parent.partitions(5))
> pglist
> }
> }
> object Test1 {
> def main(args: Array[String]) = {
> val spark = SparkSession.builder().appName("test").getOrCreate()
> val sc = spark.sparkContext
> val rdd = sc.parallelize(1 to 6, 6)
> rdd.coalesce(2, false, Some(new MyCoalescer)).count()
> spark.stop()
> }
> }
> {code}
> it throws exception:
> Exception in thread "dag-scheduler-event-loop" java.lang.StackOverflowError
> at java.lang.Exception.<init>(Exception.java:102)
> ....
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:230)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visit(SerializationDebugger.scala:108)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializableWithWriteObjectMethod(SerializationDebugger.scala:243)
> at
> org.apache.spark.serializer.SerializationDebugger$SerializationDebugger.visitSerializable(SerializationDebugger.scala:189)
> ....
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
