Got it, thanks a lot. -Jianmin
On 12/14/13 11:03 PM, "Sebastian Schelter" <[email protected]> wrote: >I wrapped the object like done here: > >https://github.com/amplab/shark/blob/master/src/main/scala/shark/execution >/serialization/KryoSerializationWrapper.scala > >If you find a better solution, let me know! > >On 14.12.2013 15:53, Jianmin Wu wrote: >> Thanks for the response, sebastian. >> >> Yes, the class of instance to distribute does not implement the >> java.io.Serializable interface. According to description in >> >>http://spark.incubator.apache.org/docs/latest/tuning.html#data-serializat >>io >> n , it seems that implementing the serializable interface is not >>required >> in Kryo way? >> >> How your problem get resolved? :-) >> >> Thanks, >> Jianmin >> >> >> On 12/14/13 10:39 PM, "Sebastian Schelter" <[email protected]> wrote: >> >>> I've seen similar problems, have you tried to broadcast an instance of >>>a >>> non-serializable class? >>> >>> --sebastian >>> >>> On 14.12.2013 15:32, Jianmin Wu wrote: >>>> Hi, all >>>> >>>> We are using Java API in Spark and tried to use KryoSerializer as the >>>> method >>>> for Java object >>>> >>>> >>>>serialization(http://spark.incubator.apache.org/docs/latest/tuning.html >>>>#d >>>> ata >>>> -serialization). Before creating SparkContext, the following code >>>>lines >>>> are >>>> inserted: >>>> >>>> System.setProperty("spark.serializer", >>>> "org.apache.spark.serializer.KryoSerializer"); >>>> >>>> System.setProperty("spark.kryo.registrator", >>>> "com.zyz.spark.bi.log.BiKryoRegistrator"); >>>> >>>> >>>> >>>> In the implementation of KryoRegistrator in class BiKryoRegistrator, >>>> all the >>>> referred classes are registered in the way like >>>> >>>> kryo.register(IpLocationDict.class); >>>> >>>> >>>> >>>> And we printed a log in the beginning of registration: >>>> >>>> System.err.println("enter BiKryoRegistrator.registerClasses"); >>>> >>>> >>>> >>>> With these setting, we run the package in mesos, but still get the >>>> following >>>> error message: >>>> >>>> Exception in thread "main" org.apache.spark.SparkException: Job >>>>failed: >>>> java.io.NotSerializableException: com.zyz.spark.bi.log.IpLocationDict >>>> >>>> >>>> >>>> The log message "enter BiKryoRegistrator.registerClasses" shows that >>>> all the >>>> registration of class are executed indeed. We googled the error >>>> message, but >>>> failed to get the helpful information. >>>> >>>> >>>> >>>> So any other steps we missed for using KryoSerializer? Any hints or >>>> suggestions are appreciated. >>>> >>>> >>>> >>>> Thanks in advance, >>>> >>>> Jianmin >>>> >>>> >>>> >>> >> >> >
