[jira] [Created] (SPARK-2200) breeze DenseVector not serializable with KryoSerializer

[Neville Li (JIRA)]

Neville Li created SPARK-2200:
---------------------------------

             Summary: breeze DenseVector not serializable with KryoSerializer
                 Key: SPARK-2200
                 URL: https://issues.apache.org/jira/browse/SPARK-2200
             Project: Spark
          Issue Type: Bug
          Components: MLlib
    Affects Versions: 1.0.0
            Reporter: Neville Li
            Priority: Minor


Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector 
with KryoSerializer throws the following stack trace. Looks like some recursive 
field in the object. Upgrading to 0.8.1 solved this.
{code}
java.lang.StackOverflowError
        at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
        at 
sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
        at 
sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
        at java.lang.reflect.Field.get(Field.java:379)
        at 
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
        at 
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
        at 
com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
        at 
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
...
{code}

Code to reproduce:
{code}
import breeze.linalg.DenseVector
import org.apache.spark.SparkConf
import org.apache.spark.serializer.KryoSerializer

object SerializerTest {
  def main(args: Array[String]) {
    val conf = new SparkConf()
      .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
      .set("spark.kryo.referenceTracking", "false")
      .set("spark.kryoserializer.buffer.mb", "8")

    val serializer = new KryoSerializer(conf).newInstance()
    serializer.serialize(DenseVector.rand(10))
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)