There is also this quote from the Tuning guide
(http://spark.incubator.apache.org/docs/latest/tuning.html):
" Finally, if you don't register your classes, Kryo will still work, but
it will have to store the full class name with each object, which is
wasteful."
It implies that you don't really have to register your classes with
Kryo. However, what kind of waste are we talking about? :)
Ognen
On 3/28/14, 12:10 PM, Debasish Das wrote:
Classes are serialized and sent to all the workers as akka msgs....
singletons and case classes I am not sure if they are javaserialized
or kryoserialized by default....
But definitely your own classes if serialized by kryo will be much
efficient.....there is an comparison that Matei did for all the
serialization options and kryo was fastest at that time....
Hi,
I am sorry if this has been asked before. I found that if I wrapped up
some
methods in a class with parameters, spark will throw "Task
Nonserializable"
exception; however if wrapped up in an object or case class without
parameters, it will work fine. Is it true that all classes involving RDD
operation should be registered so that SparkContext could recognize them?
Thanks a lot!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Do-all-classes-involving-RDD-operation-need-to-be-registered-tp3439.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
