Yeah, if you just say spark.serializer to Kryo, it will use it for all these things.
Matei On Nov 25, 2013, at 4:59 PM, Andrew Ash <[email protected]> wrote: > How do you know Spark doesn't also use Kryo for shuffled files? Are there > metrics or logs somewhere that make you believe it's normal Java > serialization? > > > On Mon, Nov 25, 2013 at 4:46 PM, Mayuresh Kunjir <[email protected]> > wrote: > This shows how to serialize user classes. I wanted Spark to serialize all > shuffle files and object files using Kryo. How can I specify that? Or would > that be done by default if I just set spark.serializer to kryo? > > > > > On Mon, Nov 25, 2013 at 7:42 PM, Matei Zaharia <[email protected]> > wrote: > Did you look through > http://spark.incubator.apache.org/docs/latest/tuning.html#data-serialization? > It shows an example of how to register classes with Kryo. In particular, in > your Registrator, you can use kryo.register(yourClass, new YourSerializer) to > pass a custom serializer too. > > Matei > > On Nov 25, 2013, at 4:25 PM, Mayuresh Kunjir <[email protected]> > wrote: > >> Hi Spark users, >> >> This has probably been answered before, but I could not locate it. I >> understand from the tuning guide that using Kryo serialization for shuffles >> improves the performance. I would like to know how to register the Kryo >> serializer. Apart from the shuffles, my standalone application needs to >> store and retrieve a few object files as well. I would really appreciate any >> pointers on registering Kryo serializer for both these serialization tasks. >> >> Thanks and regards, >> ~Mayuresh >> >> > > >
