Re: Why spark history server does not show RDD even if it is persisted?

Thanks! *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* *Mobile : (213)-572-7858* *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254 * On Tue, Feb 28, 2017 at 12:53 PM, Shixiong(Ryan) Zhu < shixi...@databricks.com> wrote:

Re: Why spark history server does not show RDD even if it is persisted?

The REST APIs are not just for Spark history server. When an application is running, you can use the REST APIs to talk to Spark UI HTTP server as well. On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari wrote: > ping... > > > > *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* > *Mobile : (213)-

Re: Why spark history server does not show RDD even if it is persisted?

ping... *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)* *Mobile : (213)-572-7858* *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254 * On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari wrote: > Thanks! > > If spark does

Re: Why spark history server does not show RDD even if it is persisted?

Thanks! If spark does not log these events in event log then why spark history server provides an API to get RDD information? >From the documentation, /applications/[app-id]/storage/rdd A list of stored RDDs for the given application. /applications/[app-id]/storage/rdd/[rdd-id] Details for

Re: Why spark history server does not show RDD even if it is persisted?

It is too verbose, and will significantly increase the size event log. Here is the comment in the code: // No-op because logging every update would be overkill > override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {} > > On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari wrote:

Re: Why spark history server does not show RDD even if it is persisted?

Thanks a lot the information! Is there any reason why EventLoggingListener ignore this event? *Thanks,* *​Parag​* On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao wrote: > AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will > not be written into event-log, I think that's w

Re: Why spark history server does not show RDD even if it is persisted?

AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will not be written into event-log, I think that's why you cannot get such info in history server. On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari wrote: > Hi, > > I am running spark shell in spark version 2.0.2. Here is my p

Why spark history server does not show RDD even if it is persisted?

Hi, I am running spark shell in spark version 2.0.2. Here is my program, var myrdd = sc.parallelize(Array.range(1, 10)) myrdd.setName("test") myrdd.cache myrdd.collect But I am not able to see any RDD info in "storage" tab in spark history server. I looked at this