bq. it uses Utils.deserialize, which is always using Java serialization. I agree with your finding.
On Mon, Mar 23, 2015 at 1:14 PM, Koert Kuipers <[email protected]> wrote: > in the comments on SparkContext.objectFile it says: > "It will also be pretty slow if you use the default serializer (Java > serialization)" > > this suggests the spark.serializer is used, which means i can switch to > the much faster kryo serializer. however when i look at the code it uses > Utils.deserialize, which is always using Java serialization. > > did i get that right? and is this desired? > it seems straightforward to switch objectFile to use the serializer as > specified by spark.serializer (although it might being in new classloader > issues). >
