I wrapped the object like done here: https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution/serialization/KryoSerializationWrapper.scala
If you find a better solution, let me know! On 14.12.2013 15:53, Jianmin Wu wrote: > Thanks for the response, sebastian. > > Yes, the class of instance to distribute does not implement the > java.io.Serializable interface. According to description in > http://spark.incubator.apache.org/docs/latest/tuning.html#data-serializatio > n , it seems that implementing the serializable interface is not required > in Kryo way? > > How your problem get resolved? :-) > > Thanks, > Jianmin > > > On 12/14/13 10:39 PM, "Sebastian Schelter" <[email protected]> wrote: > >> I've seen similar problems, have you tried to broadcast an instance of a >> non-serializable class? >> >> --sebastian >> >> On 14.12.2013 15:32, Jianmin Wu wrote: >>> Hi, all >>> >>> We are using Java API in Spark and tried to use KryoSerializer as the >>> method >>> for Java object >>> >>> serialization(http://spark.incubator.apache.org/docs/latest/tuning.html#d >>> ata >>> -serialization). Before creating SparkContext, the following code lines >>> are >>> inserted: >>> >>> System.setProperty("spark.serializer", >>> "org.apache.spark.serializer.KryoSerializer"); >>> >>> System.setProperty("spark.kryo.registrator", >>> "com.zyz.spark.bi.log.BiKryoRegistrator"); >>> >>> >>> >>> In the implementation of KryoRegistrator in class BiKryoRegistrator, >>> all the >>> referred classes are registered in the way like >>> >>> kryo.register(IpLocationDict.class); >>> >>> >>> >>> And we printed a log in the beginning of registration: >>> >>> System.err.println("enter BiKryoRegistrator.registerClasses"); >>> >>> >>> >>> With these setting, we run the package in mesos, but still get the >>> following >>> error message: >>> >>> Exception in thread "main" org.apache.spark.SparkException: Job failed: >>> java.io.NotSerializableException: com.zyz.spark.bi.log.IpLocationDict >>> >>> >>> >>> The log message "enter BiKryoRegistrator.registerClasses" shows that >>> all the >>> registration of class are executed indeed. We googled the error >>> message, but >>> failed to get the helpful information. >>> >>> >>> >>> So any other steps we missed for using KryoSerializer? Any hints or >>> suggestions are appreciated. >>> >>> >>> >>> Thanks in advance, >>> >>> Jianmin >>> >>> >>> >> > >
