[
https://issues.apache.org/jira/browse/SPARK-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-7180:
-----------------------------------
Assignee: Apache Spark
> SerializationDebugger fails with ArrayOutOfBoundsException
> ----------------------------------------------------------
>
> Key: SPARK-7180
> URL: https://issues.apache.org/jira/browse/SPARK-7180
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Andrew Or
> Assignee: Apache Spark
>
> Simple reproduction:
> {code}
> class Parent extends Serializable {
> val a = "a"
> val b = "b"
> }
> class Child extends Parent with Serializable {
> val c = Array(1)
> val d = Array(2)
> val e = Array(3)
> val f = Array(4)
> val g = Array(5)
> val o = new Object
> }
> // ArrayOutOfBoundsException
> SparkEnv.get.closureSerializer.newInstance().serialize(new Child)
> {code}
> I dug into this a little and found that we are trying to fill the fields of
> `Parent` with the values of `Child`. See the following output I generated by
> adding println's everywhere:
> {code}
> * Visiting object org.apache.spark.serializer.Child@2c3299f6 of type
> org.apache.spark.serializer.Child
> - Found 2 class data slot descriptions
> - Looking at desc #1: org.apache.spark.serializer.Parent: static final long
> serialVersionUID = 3254964199136071914L;
> - Found 2 fields
> - Ljava/lang/String; a
> - Ljava/lang/String; b
> - getObjFieldValues:
> - [I@23faa614
> - [I@1cad7d80
> - [I@420a6d35
> - [I@3a87d472
> - [I@2b8ca663
> - java.lang.Object@1effc3eb
> {code}
> SerializationDebugger#visitSerializable found two fields that belong to the
> parents, but it tried to cram the child's values into these two fields. The
> mismatch of number of fields here throws the ArrayOutOfBoundExceptions as a
> result. The culprit is this line:
> https://github.com/apache/spark/blob/4d9e560b5470029143926827b1cb9d72a0bfbeff/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala#L150,
> which runs reflection on the object `Child` even when it's considering the
> description for `Parent`.
> I ran into this when trying to serialize a test suite that extends `FunSuite`
> (don't ask why).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
