How do you know Spark doesn't also use Kryo for shuffled files? Are there metrics or logs somewhere that make you believe it's normal Java serialization?
On Mon, Nov 25, 2013 at 4:46 PM, Mayuresh Kunjir <[email protected]>wrote: > This shows how to serialize user classes. I wanted Spark to serialize all > shuffle files and object files using Kryo. How can I specify that? Or would > that be done by default if I just set spark.serializer to kryo? > > > > > On Mon, Nov 25, 2013 at 7:42 PM, Matei Zaharia <[email protected]>wrote: > >> Did you look through >> http://spark.incubator.apache.org/docs/latest/tuning.html#data-serialization?It >> shows an example of how to register classes with Kryo. In particular, in >> your Registrator, you can use kryo.register(yourClass, new YourSerializer) >> to pass a custom serializer too. >> >> Matei >> >> On Nov 25, 2013, at 4:25 PM, Mayuresh Kunjir <[email protected]> >> wrote: >> >> Hi Spark users, >> >> This has probably been answered before, but I could not locate it. I >> understand from the tuning guide that using Kryo serialization for shuffles >> improves the performance. I would like to know how to register the Kryo >> serializer. Apart from the shuffles, my standalone application needs to >> store and retrieve a few object files as well. I would really appreciate >> any pointers on registering Kryo serializer for both these serialization >> tasks. >> >> Thanks and regards, >> ~Mayuresh >> >> >> >> >
