I've seen similar problems, have you tried to broadcast an instance of a non-serializable class?
--sebastian On 14.12.2013 15:32, Jianmin Wu wrote: > Hi, all > > We are using Java API in Spark and tried to use KryoSerializer as the method > for Java object > serialization(http://spark.incubator.apache.org/docs/latest/tuning.html#data > -serialization). Before creating SparkContext, the following code lines are > inserted: > > System.setProperty("spark.serializer", > "org.apache.spark.serializer.KryoSerializer"); > > System.setProperty("spark.kryo.registrator", > "com.zyz.spark.bi.log.BiKryoRegistrator"); > > > > In the implementation of KryoRegistrator in class BiKryoRegistrator, all the > referred classes are registered in the way like > > kryo.register(IpLocationDict.class); > > > > And we printed a log in the beginning of registration: > > System.err.println("enter BiKryoRegistrator.registerClasses"); > > > > With these setting, we run the package in mesos, but still get the following > error message: > > Exception in thread "main" org.apache.spark.SparkException: Job failed: > java.io.NotSerializableException: com.zyz.spark.bi.log.IpLocationDict > > > > The log message "enter BiKryoRegistrator.registerClasses" shows that all the > registration of class are executed indeed. We googled the error message, but > failed to get the helpful information. > > > > So any other steps we missed for using KryoSerializer? Any hints or > suggestions are appreciated. > > > > Thanks in advance, > > Jianmin > > >
