1 update:
Now both of the commands with and without sortByKey have started throwing
an error.
Loss was due to java.lang.NoSuchMethodError
java.lang.NoSuchMethodError:
com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
at
com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
On Mon, Dec 9, 2013 at 11:21 AM, Archit Thakur <[email protected]>wrote:
> And Since sortByKey serializes the classes, I guess it has something to do
> with Serialization thing.
>
>
> On Mon, Dec 9, 2013 at 11:19 AM, Archit Thakur
> <[email protected]>wrote:
>
>> I did make the classes Serialized. But now running the same command
>> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
>> func).sortByKey().count(), gives me java.lang.NoSuchMethodError.
>>
>> For the Collection class which I made Serialized accesses one static
>> variable that
>>
>> static com.xyz.logging.Logger Logger = new
>> com.xyz.logging.Logger(Collection.class) and It throws
>>
>> java.lang.NoSuchMethodError:
>> com.guavus.logging.Logger.<init>(Ljava/lang/Class;)V
>> at
>> com.guavus.mapred.common.collection.Collection.<clinit>(Collection.java:17)
>>
>> but it doesn't do that when I don't sortByKey, ie when I run
>> sc.sequenceFile(file, classOf[Text], classOf[Text]).flatMap(map_
>> func).count() it doesn't throw the error.
>>
>> Thanks and Regards,
>> Archit Thakur.
>>
>>
>>
>> On Mon, Dec 9, 2013 at 10:48 AM, Patrick Wendell <[email protected]>wrote:
>>
>>> It's because sorting serializes the data during the shuffle phase.
>>>
>>> On Sun, Dec 8, 2013 at 8:58 PM, Archit Thakur <[email protected]>
>>> wrote:
>>> > Hi,
>>> >
>>> > When I did
>>> >
>>> > sc.sequenceFile(file, classOf[Text],
>>> > classOf[Text]).flatMap(map_func).count()
>>> > It gave me result of 365.
>>> >
>>> > However, when I did
>>> > sc.sequenceFile(file, classOf[Text],
>>> > classOf[Text]).flatMap(map_func).sortByKey().count(),
>>> >
>>> > It threw java.io.NotSerializableException for Key Class returned by
>>> flapMap.
>>> > My question is
>>> > Why does sortByKey require the Key/Value Classes to be Serialized.?
>>> >
>>> > Thanks and Regards,
>>> > Archit Thakur.
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "Spark Users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> an
>>> > email to [email protected].
>>> > For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Spark Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>
>>
>>
>