Hi, all
We are using Java API in Spark and tried to use KryoSerializer as the method
for Java object
serialization(http://spark.incubator.apache.org/docs/latest/tuning.html#data
-serialization). Before creating SparkContext, the following code lines are
inserted:
System.setProperty("spark.serializer",
"org.apache.spark.serializer.KryoSerializer");
System.setProperty("spark.kryo.registrator",
"com.zyz.spark.bi.log.BiKryoRegistrator");
In the implementation of KryoRegistrator in class BiKryoRegistrator, all the
referred classes are registered in the way like
kryo.register(IpLocationDict.class);
And we printed a log in the beginning of registration:
System.err.println("enter BiKryoRegistrator.registerClasses");
With these setting, we run the package in mesos, but still get the following
error message:
Exception in thread "main" org.apache.spark.SparkException: Job failed:
java.io.NotSerializableException: com.zyz.spark.bi.log.IpLocationDict
The log message "enter BiKryoRegistrator.registerClasses" shows that all the
registration of class are executed indeed. We googled the error message, but
failed to get the helpful information.
So any other steps we missed for using KryoSerializer? Any hints or
suggestions are appreciated.
Thanks in advance,
Jianmin
