Hi,
I'm trying to call saveAsObjectFile() on an RDD[*(Int, Int, Double Double)*],
expecting the output binary to be smaller, but it is exactly the same size
of when kryo is not on.
I've checked the log, and there is no trace of kryo related errors.
The code looks something like:
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.setRegistrationRequired(true)
kryo.register(classOf[*(Int, Int, Double Double)*])
}
}
System.setProperty("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
System.setProperty("spark.kryo.registrator", "MyRegistrator")
At the end, I tried to call:
kryo.setRegistrationRequired(*true*)
to make sure my class gets registered. But I found errors like:
Exception in thread "DAGScheduler" com.esotericsoftware.kryo.KryoException:
java.lang.IllegalArgumentException: Class is not registered:
*scala.math.Numeric$IntIsIntegral$*
Note: To register this class use:
kryo.register(scala.math.Numeric$IntIsIntegral$.class);
It appears many scala internal types have to be registered in order to have
full kryo support.
Any idea why my simple tuple type should not get kryo benefits?
