[
https://issues.apache.org/jira/browse/SPARK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiangrui Meng resolved SPARK-2200.
----------------------------------
Resolution: Later
Close this JIRA since this is caused by a non-default kryo setting.
> breeze DenseVector not serializable with KryoSerializer
> -------------------------------------------------------
>
> Key: SPARK-2200
> URL: https://issues.apache.org/jira/browse/SPARK-2200
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 1.0.0
> Reporter: Neville Li
> Priority: Minor
>
> Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector
> with KryoSerializer throws the following stack trace. Looks like some
> recursive field in the object. Upgrading to 0.8.1 solved this.
> {code}
> java.lang.StackOverflowError
> at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
> at
> sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
> at
> sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
> at java.lang.reflect.Field.get(Field.java:379)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> ...
> {code}
> Code to reproduce:
> {code}
> import breeze.linalg.DenseVector
> import org.apache.spark.SparkConf
> import org.apache.spark.serializer.KryoSerializer
> object SerializerTest {
> def main(args: Array[String]) {
> val conf = new SparkConf()
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
> .set("spark.kryo.referenceTracking", "false")
> .set("spark.kryoserializer.buffer.mb", "8")
> val serializer = new KryoSerializer(conf).newInstance()
> serializer.serialize(DenseVector.rand(10))
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
