Re: Spark UI Source Code

[Anshi Shrivastava]

Hi Marcelo, Dev,

Thanks for your response.
I have used SparkListeners to fetch the metrics (the public REST API uses
the same) but to monitor these metrics over time, I have to persist them
(using KVStore library of spark).  Is there a way to fetch data from this
KVStore (which uses levelDb for storage) and filter it on basis on
timestamp?

Thanks,
Anshi

On Mon, May 7, 2018 at 9:51 PM, Marcelo Vanzin [via Apache Spark User List]
<ml+s1001560n32114...@n3.nabble.com> wrote:

> On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava
> <[hidden email] <http:///user/SendEmail.jtp?type=node=32114=0>>
> wrote:
> > I've found a KVStore wrapper which stores all the metrics in a LevelDb
> > store. This KVStore wrapper is available as a spark-dependency but we
> cannot
> > access the metrics directly from spark since they are all private.
>
> I'm not sure what it is you're trying to do exactly, but there's a
> public REST API that exposes all the data Spark keeps about
> applications. There's also a programmatic status tracker
> (SparkContext.statusTracker) that's easier to use from within the
> running Spark app, but has a lot less info.
>
> > Can we use this store to store our own metrics?
>
> No.
>
> > Also can we retrieve these metrics based on timestamp?
>
> Only if the REST API has that feature, don't remember off the top of my
> head.
>
>
> --
> Marcelo
>
> -
> To unsubscribe e-mail: [hidden email]
> <http:///user/SendEmail.jtp?type=node=32114=1>
>
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Re-
> Spark-UI-Source-Code-tp32114.html
> To start a new topic under Apache Spark User List, email
> ml+s1001560n1...@n3.nabble.com
> To unsubscribe from Apache Spark User List, click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=1=YW5zaGkuc2hyaXZhc3RhdmFAZXhhZGF0dW0uY29tfDF8MTU2MDE5MjUz>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

-- 
*
*
*DISCLAIMER:*
All the content in email is intended for the recipient 
and not to be published elsewhere without Exadatum consent. And attachments 
shall be send only if required and with ownership of the sender. This 
message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete 
this email from your system. Email transmission cannot be guaranteed to be 
secure or error-free, as information could be intercepted, corrupted, lost, 
destroyed, arrive late or incomplete, or contain viruses. The sender, 
therefore, does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of email transmission. If 
verification is required, please request a hard-copy version.

Re: Spark UI Source Code

[Marcelo Vanzin]

On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava
 wrote:
> I've found a KVStore wrapper which stores all the metrics in a LevelDb
> store. This KVStore wrapper is available as a spark-dependency but we cannot
> access the metrics directly from spark since they are all private.

I'm not sure what it is you're trying to do exactly, but there's a
public REST API that exposes all the data Spark keeps about
applications. There's also a programmatic status tracker
(SparkContext.statusTracker) that's easier to use from within the
running Spark app, but has a lot less info.

> Can we use this store to store our own metrics?

No.

> Also can we retrieve these metrics based on timestamp?

Only if the REST API has that feature, don't remember off the top of my head.


-- 
Marcelo

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Spark UI Source Code

[Anshi Shrivastava]

Hi All,

I've been trying to debug the Spark UI source code to replicate the same
Metric monitoring mechanism in my application.

I've found a KVStore wrapper which stores all the metrics in a LevelDb
store. This KVStore wrapper is available as a spark-dependency but we
cannot access the metrics directly from spark since they are all private.
Can we use this store to store our own metrics? Also can we retrieve these
metrics based on timestamp?

Thanks and Regards,
Anshi

-- 
*
*
*DISCLAIMER:*
All the content in email is intended for the recipient 
and not to be published elsewhere without Exadatum consent. And attachments 
shall be send only if required and with ownership of the sender. This 
message contains confidential information and is intended only for the 
individual named. If you are not the named addressee, you should not 
disseminate, distribute or copy this email. Please notify the sender 
immediately by email if you have received this email by mistake and delete 
this email from your system. Email transmission cannot be guaranteed to be 
secure or error-free, as information could be intercepted, corrupted, lost, 
destroyed, arrive late or incomplete, or contain viruses. The sender, 
therefore, does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of email transmission. If 
verification is required, please request a hard-copy version.