Thanks for the response, sebastian. Yes, the class of instance to distribute does not implement the java.io.Serializable interface. According to description in http://spark.incubator.apache.org/docs/latest/tuning.html#data-serializatio n , it seems that implementing the serializable interface is not required in Kryo way?
How your problem get resolved? :-) Thanks, Jianmin On 12/14/13 10:39 PM, "Sebastian Schelter" <[email protected]> wrote: >I've seen similar problems, have you tried to broadcast an instance of a >non-serializable class? > >--sebastian > >On 14.12.2013 15:32, Jianmin Wu wrote: >> Hi, all >> >> We are using Java API in Spark and tried to use KryoSerializer as the >>method >> for Java object >> >>serialization(http://spark.incubator.apache.org/docs/latest/tuning.html#d >>ata >> -serialization). Before creating SparkContext, the following code lines >>are >> inserted: >> >> System.setProperty("spark.serializer", >> "org.apache.spark.serializer.KryoSerializer"); >> >> System.setProperty("spark.kryo.registrator", >> "com.zyz.spark.bi.log.BiKryoRegistrator"); >> >> >> >> In the implementation of KryoRegistrator in class BiKryoRegistrator, >>all the >> referred classes are registered in the way like >> >> kryo.register(IpLocationDict.class); >> >> >> >> And we printed a log in the beginning of registration: >> >> System.err.println("enter BiKryoRegistrator.registerClasses"); >> >> >> >> With these setting, we run the package in mesos, but still get the >>following >> error message: >> >> Exception in thread "main" org.apache.spark.SparkException: Job failed: >> java.io.NotSerializableException: com.zyz.spark.bi.log.IpLocationDict >> >> >> >> The log message "enter BiKryoRegistrator.registerClasses" shows that >>all the >> registration of class are executed indeed. We googled the error >>message, but >> failed to get the helpful information. >> >> >> >> So any other steps we missed for using KryoSerializer? Any hints or >> suggestions are appreciated. >> >> >> >> Thanks in advance, >> >> Jianmin >> >> >> >
