[
https://issues.apache.org/jira/browse/SPARK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038223#comment-14038223
]
Neville Li commented on SPARK-2200:
-----------------------------------
With 0.7 the error went away when reference tracking is set to true.
With 0.8.1 it works either way.
Turns out in 0.7 the recursive references was caused by this:
private final val innerUpdate: ((Int,E) => Unit) = if ((offset == 0) &&
(stride == 1)) { (i:Int,v:E) => {data(i) = v} } else {(i:Int,v:E) =>
{data(offset+i*stride)=v} }
The function val has an closure $outer that references itself. It was removed
in 0.8.1.
> breeze DenseVector not serializable with KryoSerializer
> -------------------------------------------------------
>
> Key: SPARK-2200
> URL: https://issues.apache.org/jira/browse/SPARK-2200
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.0.0
> Reporter: Neville Li
> Priority: Minor
>
> Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector
> with KryoSerializer throws the following stack trace. Looks like some
> recursive field in the object. Upgrading to 0.8.1 solved this.
> {code}
> java.lang.StackOverflowError
> at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
> at
> sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
> at
> sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
> at java.lang.reflect.Field.get(Field.java:379)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> ...
> {code}
> Code to reproduce:
> {code}
> import breeze.linalg.DenseVector
> import org.apache.spark.SparkConf
> import org.apache.spark.serializer.KryoSerializer
> object SerializerTest {
> def main(args: Array[String]) {
> val conf = new SparkConf()
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
> .set("spark.kryo.referenceTracking", "false")
> .set("spark.kryoserializer.buffer.mb", "8")
> val serializer = new KryoSerializer(conf).newInstance()
> serializer.serialize(DenseVector.rand(10))
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
