Re: Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari]

Thanks!



*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
*


On Tue, Feb 28, 2017 at 12:53 PM, Shixiong(Ryan) Zhu <
shixi...@databricks.com> wrote:

> The REST APIs are not just for Spark history server. When an application
> is running, you can use the REST APIs to talk to Spark UI HTTP server as
> well.
>
> On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari 
> wrote:
>
>> ping...
>>
>>
>>
>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>> *
>>
>>
>> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari 
>> wrote:
>>
>>> Thanks!
>>>
>>> If spark does not log these events in event log then why spark history
>>> server provides an API to get RDD information?
>>>
>>> From the documentation,
>>>
>>> /applications/[app-id]/storage/rdd   A list of stored RDDs for the
>>> given application.
>>>
>>> /applications/[app-id]/storage/rdd/[rdd-id]   Details for the storage
>>> status of a given RDD.
>>>
>>>
>>>
>>>
>>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>>> *
>>>
>>>
>>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao 
>>> wrote:
>>>
 It is too verbose, and will significantly increase the size event log.

 Here is the comment in the code:

 // No-op because logging every update would be overkill
> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>
>
 On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari 
 wrote:

> Thanks a lot the information!
>
> Is there any reason why EventLoggingListener ignore this event?
>
> *Thanks,*
>
>
> *​Parag​*
>
> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao 
> wrote:
>
>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>> will not be written into event-log, I think that's why you cannot get 
>> such
>> info in history server.
>>
>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari > > wrote:
>>
>>> Hi,
>>>
>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>
>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>> myrdd.setName("test")
>>> myrdd.cache
>>> myrdd.collect
>>>
>>> But I am not able to see any RDD info in "storage" tab in spark
>>> history server.
>>>
>>> I looked at this
>>> 
>>> but it is not helping as I have exact similar program mentioned there. 
>>> Can
>>> anyone help?
>>>
>>>
>>> *Thanks,*
>>>
>>> *​Parag​*
>>>
>>
>>
>

>>>
>>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Shixiong(Ryan) Zhu]

The REST APIs are not just for Spark history server. When an application is
running, you can use the REST APIs to talk to Spark UI HTTP server as well.

On Tue, Feb 28, 2017 at 10:46 AM, Parag Chaudhari 
wrote:

> ping...
>
>
>
> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
> *Mobile : (213)-572-7858 <(213)%20572-7858>*
> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
> *
>
>
> On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari 
> wrote:
>
>> Thanks!
>>
>> If spark does not log these events in event log then why spark history
>> server provides an API to get RDD information?
>>
>> From the documentation,
>>
>> /applications/[app-id]/storage/rdd   A list of stored RDDs for the given
>> application.
>>
>> /applications/[app-id]/storage/rdd/[rdd-id]   Details for the storage
>> status of a given RDD.
>>
>>
>>
>>
>> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
>> *Mobile : (213)-572-7858 <(213)%20572-7858>*
>> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
>> *
>>
>>
>> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao 
>> wrote:
>>
>>> It is too verbose, and will significantly increase the size event log.
>>>
>>> Here is the comment in the code:
>>>
>>> // No-op because logging every update would be overkill
 override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}


>>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari 
>>> wrote:
>>>
 Thanks a lot the information!

 Is there any reason why EventLoggingListener ignore this event?

 *Thanks,*


 *​Parag​*

 On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao 
 wrote:

> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
> will not be written into event-log, I think that's why you cannot get such
> info in history server.
>
> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
> wrote:
>
>> Hi,
>>
>> I am running spark shell in spark version 2.0.2. Here is my program,
>>
>> var myrdd = sc.parallelize(Array.range(1, 10))
>> myrdd.setName("test")
>> myrdd.cache
>> myrdd.collect
>>
>> But I am not able to see any RDD info in "storage" tab in spark
>> history server.
>>
>> I looked at this
>> 
>> but it is not helping as I have exact similar program mentioned there. 
>> Can
>> anyone help?
>>
>>
>> *Thanks,*
>>
>> *​Parag​*
>>
>
>

>>>
>>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari]

ping...



*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
*


On Wed, Feb 22, 2017 at 7:54 PM, Parag Chaudhari 
wrote:

> Thanks!
>
> If spark does not log these events in event log then why spark history
> server provides an API to get RDD information?
>
> From the documentation,
>
> /applications/[app-id]/storage/rdd   A list of stored RDDs for the given
> application.
>
> /applications/[app-id]/storage/rdd/[rdd-id]   Details for the storage
> status of a given RDD.
>
>
>
>
> *Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
> *Mobile : (213)-572-7858 <(213)%20572-7858>*
> *Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
> *
>
>
> On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao 
> wrote:
>
>> It is too verbose, and will significantly increase the size event log.
>>
>> Here is the comment in the code:
>>
>> // No-op because logging every update would be overkill
>>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>>
>>>
>> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari 
>> wrote:
>>
>>> Thanks a lot the information!
>>>
>>> Is there any reason why EventLoggingListener ignore this event?
>>>
>>> *Thanks,*
>>>
>>>
>>> *​Parag​*
>>>
>>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao 
>>> wrote:
>>>
 AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
 will not be written into event-log, I think that's why you cannot get such
 info in history server.

 On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
 wrote:

> Hi,
>
> I am running spark shell in spark version 2.0.2. Here is my program,
>
> var myrdd = sc.parallelize(Array.range(1, 10))
> myrdd.setName("test")
> myrdd.cache
> myrdd.collect
>
> But I am not able to see any RDD info in "storage" tab in spark
> history server.
>
> I looked at this
> 
> but it is not helping as I have exact similar program mentioned there. Can
> anyone help?
>
>
> *Thanks,*
>
> *​Parag​*
>


>>>
>>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari]

Thanks!

If spark does not log these events in event log then why spark history
server provides an API to get RDD information?

>From the documentation,

/applications/[app-id]/storage/rdd   A list of stored RDDs for the given
application.

/applications/[app-id]/storage/rdd/[rdd-id]   Details for the storage
status of a given RDD.




*Thanks,Parag Chaudhari,**USC Alumnus (Fight On!)*
*Mobile : (213)-572-7858*
*Profile: http://www.linkedin.com/pub/parag-chaudhari/28/a55/254
*


On Wed, Feb 22, 2017 at 7:44 PM, Saisai Shao  wrote:

> It is too verbose, and will significantly increase the size event log.
>
> Here is the comment in the code:
>
> // No-op because logging every update would be overkill
>> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>>
>>
> On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari 
> wrote:
>
>> Thanks a lot the information!
>>
>> Is there any reason why EventLoggingListener ignore this event?
>>
>> *Thanks,*
>>
>>
>> *​Parag​*
>>
>> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao 
>> wrote:
>>
>>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>>> will not be written into event-log, I think that's why you cannot get such
>>> info in history server.
>>>
>>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
>>> wrote:
>>>
 Hi,

 I am running spark shell in spark version 2.0.2. Here is my program,

 var myrdd = sc.parallelize(Array.range(1, 10))
 myrdd.setName("test")
 myrdd.cache
 myrdd.collect

 But I am not able to see any RDD info in "storage" tab in spark history
 server.

 I looked at this
 
 but it is not helping as I have exact similar program mentioned there. Can
 anyone help?


 *Thanks,*

 *​Parag​*

>>>
>>>
>>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Saisai Shao]

It is too verbose, and will significantly increase the size event log.

Here is the comment in the code:

// No-op because logging every update would be overkill
> override def onBlockUpdated(event: SparkListenerBlockUpdated): Unit = {}
>
>
On Thu, Feb 23, 2017 at 11:42 AM, Parag Chaudhari 
wrote:

> Thanks a lot the information!
>
> Is there any reason why EventLoggingListener ignore this event?
>
> *Thanks,*
>
>
> *​Parag​*
>
> On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao 
> wrote:
>
>> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it
>> will not be written into event-log, I think that's why you cannot get such
>> info in history server.
>>
>> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
>> wrote:
>>
>>> Hi,
>>>
>>> I am running spark shell in spark version 2.0.2. Here is my program,
>>>
>>> var myrdd = sc.parallelize(Array.range(1, 10))
>>> myrdd.setName("test")
>>> myrdd.cache
>>> myrdd.collect
>>>
>>> But I am not able to see any RDD info in "storage" tab in spark history
>>> server.
>>>
>>> I looked at this
>>> 
>>> but it is not helping as I have exact similar program mentioned there. Can
>>> anyone help?
>>>
>>>
>>> *Thanks,*
>>>
>>> *​Parag​*
>>>
>>
>>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari]

Thanks a lot the information!

Is there any reason why EventLoggingListener ignore this event?

*Thanks,*


*​Parag​*

On Wed, Feb 22, 2017 at 7:11 PM, Saisai Shao  wrote:

> AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will
> not be written into event-log, I think that's why you cannot get such info
> in history server.
>
> On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
> wrote:
>
>> Hi,
>>
>> I am running spark shell in spark version 2.0.2. Here is my program,
>>
>> var myrdd = sc.parallelize(Array.range(1, 10))
>> myrdd.setName("test")
>> myrdd.cache
>> myrdd.collect
>>
>> But I am not able to see any RDD info in "storage" tab in spark history
>> server.
>>
>> I looked at this
>> 
>> but it is not helping as I have exact similar program mentioned there. Can
>> anyone help?
>>
>>
>> *Thanks,*
>>
>> *​Parag​*
>>
>
>

Re: Why spark history server does not show RDD even if it is persisted?

[Saisai Shao]

AFAIK, Spark's EventLoggingListerner ignores BlockUpdate event, so it will
not be written into event-log, I think that's why you cannot get such info
in history server.

On Thu, Feb 23, 2017 at 9:51 AM, Parag Chaudhari 
wrote:

> Hi,
>
> I am running spark shell in spark version 2.0.2. Here is my program,
>
> var myrdd = sc.parallelize(Array.range(1, 10))
> myrdd.setName("test")
> myrdd.cache
> myrdd.collect
>
> But I am not able to see any RDD info in "storage" tab in spark history
> server.
>
> I looked at this
> 
> but it is not helping as I have exact similar program mentioned there. Can
> anyone help?
>
>
> *Thanks,*
>
> *​Parag​*
>

Why spark history server does not show RDD even if it is persisted?

[Parag Chaudhari]

Hi,

I am running spark shell in spark version 2.0.2. Here is my program,

var myrdd = sc.parallelize(Array.range(1, 10))
myrdd.setName("test")
myrdd.cache
myrdd.collect

But I am not able to see any RDD info in "storage" tab in spark history
server.

I looked at this

but it is not helping as I have exact similar program mentioned there. Can
anyone help?


*Thanks,*

*​Parag​*