Sure, you can request it by filing an issue here: https://issues.apache.org/jira/browse/SPARK
2014-09-05 6:50 GMT-07:00 Ruebenacker, Oliver A < oliver.ruebenac...@altisource.com>: > > > Hello, > > > > Thanks for the explanation. So events are stored internally as JSON, but > there is no official support for having Spark serve that JSON via HTTP? So > if I wanted to write an app that monitors Spark, I would either have to > scrape the web UI in HTML or rely on unofficial JSON features? That is > quite surprising, because I would expect dumping out the JSON would be > easier for Spark developers to implement than converting it to HTML. > > > > Do I get that right? Should I make a feature request? Thanks! > > > > Best, Oliver > > > > *From:* Andrew Or [mailto:and...@databricks.com] > *Sent:* Thursday, September 04, 2014 2:11 PM > *To:* Ruebenacker, Oliver A > *Cc:* Akhil Das; Wonha Ryu; user@spark.apache.org > *Subject:* Re: Web UI > > > > Hi all, > > > > The JSON version of the web UI is not officially supported; I don't > believe this is documented anywhere. > > > > The alternative is to set `spark.eventLog.enabled` to true before running > your application. This will create JSON SparkListenerEvents with details > about each task and stage as a log file. Then you can easily reconstruct > the web UI after the application has exited. This is what the standalone > Master and the History Server does, actually. For local mode, you can use > the latter to generate your UI after the fact. (This is documented here: > http://spark.apache.org/docs/latest/monitoring.html). > > > > -Andrew > > > > 2014-09-04 5:28 GMT-07:00 Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com>: > > > > Hello, > > > > Thanks for the link – this is for standalone, though, and most URLs > don’t work for local. > > I will look into deploying as standalone on a single node for testing > and development. > > > > Best, Oliver > > > > *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com] > *Sent:* Thursday, September 04, 2014 3:09 AM > *To:* Ruebenacker, Oliver A > *Cc:* Wonha Ryu; user@spark.apache.org > *Subject:* Re: Web UI > > > > Hi > > > > You can see this doc > <https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security> > for all the available webUI ports. > > > > Yes there are ways to get the data metrics in Json format, One of them is > below: > > > > ** > > *http://webUI:8080/json/ <http://webUI:8080/json/>* Or simply > > ** > > *curl webUI:8080/json/* > > > > There are some PRs about it you can read it over here > https://github.com/apache/spark/pull/1682 > > > Thanks > > Best Regards > > > > On Thu, Sep 4, 2014 at 2:24 AM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > Interestingly, http://localhost:4040/metrics/json/ gives some numbers, > but only a few which never seem to change during the application’s lifetime. > > > > Either the web UI has some very strange limitations, or there are some > URLs yet to be discovered that do something interesting. > > > > Best, > > Oliver > > > > > > *From:* Wonha Ryu [mailto:wonha....@gmail.com] > *Sent:* Wednesday, September 03, 2014 4:27 PM > > > *To:* Ruebenacker, Oliver A > *Cc:* user@spark.apache.org > *Subject:* Re: Web UI > > > > Hey Oliver, > > > > IIRC there's no JSON endpoint for application web UI. They only exist for > cluster master and worker. > > > > - Wonha > > > > > > On Wed, Sep 3, 2014 at 12:58 PM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > Thanks for the help! But I tried starting with “–master local[4]” and > when I load http://localhost:4040/json I just get forwarded to > http://localhost:4040/stages/, and it’s all human-readable HTML, no JSON. > > > > Best, > > Oliver > > > > > > *From:* Wonha Ryu [mailto:wonha....@gmail.com] > *Sent:* Wednesday, September 03, 2014 3:36 PM > *To:* Ruebenacker, Oliver A > *Cc:* user@spark.apache.org > *Subject:* Re: Web UI > > > > Hi Oliver, > > > > Spark standalone master and worker support '/json' endpoint in web UI, > which returns some of the information in JSON format. > > I wasn't able to find relevant documentation, though. > > > > - Wonha > > > > On Wed, Sep 3, 2014 at 12:12 PM, Ruebenacker, Oliver A < > oliver.ruebenac...@altisource.com> wrote: > > > > Hello, > > > > What is included in the Spark web UI? What are the available URLs? Can > the information be obtained in a machine-readable way (e.g. JSON, XML, etc)? > > > > Thanks! > > > > Best, > > Oliver > > > > Oliver Ruebenacker | Solutions Architect > > > > Altisource™ > > 290 Congress St, 7th Floor | Boston, Massachusetts 02210 > > P: (617) 728-5582 | ext: 275585 > > oliver.ruebenac...@altisource.com | www.Altisource.com > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** > > > > > *********************************************************************************************************************** > > This email message and any attachments are intended solely for the use of > the addressee. If you are not the intended recipient, you are prohibited > from reading, disclosing, reproducing, distributing, disseminating or > otherwise using this transmission. If you have received this message in > error, please promptly notify the sender by reply email and immediately > delete this message from your system. > This message and any attachments may contain information that is > confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > > *********************************************************************************************************************** >
