I did make the classes Serialized. But now running the same command
sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
For the Collection class which I made Serialized accesses one static
variable that
static com.xyz.logging.Logger Logger = new
com.xyz.logging.Logger(Collection.class) and It throws
java.lang.NoSuchMethodError:
com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
at
com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
but it doesn't do that when I don't sortByKey, ie when I run
sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
func).count() it doesn't throw the error.
Thanks and Regards,
Archit Thakur.
On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <[email protected]> wrote:
> It's because sorting serializes the data during the shuffle phase.
>
> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <[email protected]>
> wrote:
> > Hi,
> >
> > When I did
> >
> > sc.sequenceFile(file, classOf[Text],
> > classOf[Text]).flatMap(map_func).count()
> > It gave me result of 365.
> >
> > However, when I did
> > sc.sequenceFile(file, classOf[Text],
> > classOf[Text]).flatMap(map_func).sortByKey().count(),
> >
> > It threw java.io.NotSerializableException for Key Class returned by
> flapMap.
> > My question is
> > Why does sortByKey require the Key/Value Classes to be Serialized.?
> >
> > Thanks and Regards,
> > Archit Thakur.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Spark Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
