It's because sorting serializes the data during the shuffle phase.
On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <[email protected]> wrote: > Hi, > > When I did > > sc.sequenceFile(file, classOf[Text], > classOf[Text]).flatMap(map_func).count() > It gave me result of 365. > > However, when I did > sc.sequenceFile(file, classOf[Text], > classOf[Text]).flatMap(map_func).sortByKey().count(), > > It threw java.io.NotSerializableException for Key Class returned by flapMap. > My question is > Why does sortByKey require the Key/Value Classes to be Serialized.? > > Thanks and Regards, > Archit Thakur. > > -- > You received this message because you are subscribed to the Google Groups > "Spark Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out.
