One 'rule of thumbs' is to use rdd.toDebugString and check the lineage for
ShuffleRDD. As long as there's no need for restructuring the RDD,
operations can be pipelined on each partition.

"rdd.toDebugString" is your friend :-)

-kr, Gerard.


On Mon, Nov 17, 2014 at 7:37 AM, Mukesh Jha <[email protected]> wrote:

> Thanks I did go through the video it was very informative, but I think I's
> looking for the Transformations section @ page
> https://spark.apache.org/docs/0.9.1/scala-programming-guide.html.
>
>
> On Mon, Nov 17, 2014 at 10:31 AM, Samarth Mailinglist <
> [email protected]> wrote:
>
>> Check this video out:
>> https://www.youtube.com/watch?v=dmL0N3qfSc8&list=UURzsq7k4-kT-h3TDUBQ82-w
>>
>> On Mon, Nov 17, 2014 at 9:43 AM, Deep Pradhan <[email protected]>
>> wrote:
>>
>>> Hi,
>>> Is there any way to know which of my functions perform better in Spark?
>>> In other words, say I have achieved same thing using two different
>>> implementations. How do I judge as to which implementation is better than
>>> the other. Is processing time the only metric that we can use to claim the
>>> goodness of one implementation to the other?
>>> Can anyone please share some thoughts on this?
>>>
>>> Thank You
>>>
>>
>>
>
>
> --
>
>
> Thanks & Regards,
>
> *Mukesh Jha <[email protected]>*
>

Reply via email to