Thank you, Nitin. This does explain the problem. It seems that UI should make
this more clear to the user, otherwise it is simply misleading if you read it
as it.
From: Nitin Goyal [mailto:nitin2go...@gmail.com]
Sent: Sunday, October 11, 2015 5:57 AM
To: Ulanov, Alexander
Cc: dev@spark.apache.org
Subject: Re: Operations with cached RDD
The problem is not that zipWithIndex is executed again. "groupBy" triggered
hash partitioning on your keys and a shuffle happened due to that and that's
why you are seeing 2 stages. You can confirm this by clicking on latter
"zipWithIndex" stage and input data has "(memory)" written which means input
data has been fetched from memory (your cached RDD).
As far as lineage/call site is concerned, I think there was a change in spark
1.3 which excluded some classes from appearing in call site (I know that some
Spark SQL related were removed for sure).
Thanks
-Nitin
On Sat, Oct 10, 2015 at 5:05 AM, Ulanov, Alexander
mailto:alexander.ula...@hpe.com>> wrote:
Dear Spark developers,
I am trying to understand how Spark UI displays operation with the cached RDD.
For example, the following code caches an rdd:
>> val rdd = sc.parallelize(1 to 5, 5).zipWithIndex.cache
>> rdd.count
The Jobs tab shows me that the RDD is evaluated:
: 1 count at :24 2015/10/09 16:15:430.4 s
1/1
: 0 zipWithIndex at :21 2015/10/09 16:15:380.6 s
1/1
An I can observe this rdd in the Storage tab of Spark UI:
: ZippedWithIndexRDD Memory Deserialized 1x Replicated
Then I want to make an operation over the cached RDD. I run the following code:
>> val g = rdd.groupByKey()
>> g.count
The Jobs tab shows me a new Job:
: 2 count at :26
Inside this Job there are two stages:
: 3 count at :26 +details 2015/10/09 16:16:18 0.2 s 5/5
: 2 zipWithIndex at :21
It shows that zipWithIndex is executed again. It does not seem to be
reasonable, because the rdd is cached, and zipWithIndex is already executed
previously.
Could you explain why if I perform an operation followed by an action on a
cached RDD, then the last operation in the lineage of the cached RDD is shown
to be executed in the Spark UI?
Best regards, Alexander
--
Regards
Nitin Goyal