I am working on a PR that allows one to send the same spark listener event message back to the application in yarn cluster mode.
So far I have put this function in our application, our UI will receive and display the same spark job event message such as progress, job start, completed etc Essentially, it establish a communication channel , you can send over progress, messages and detailed exceptions from spark job inside yarn to your application, on you application side , you can display , or log, make use it in other ways. You can send send message to the running spark job via the channel. I will cleanup the code and send PR soon Chester Alpine Data Lab Sent from my iPhone > On Sep 26, 2014, at 7:38 AM, Matt Narrell <matt.narr...@gmail.com> wrote: > > Yes, I’m running Hadoop’s Timeline server that does this for the YARN/Hadoop > logs (and works very nicely btw). Are you saying I can do the same for the > SparkUI as well? Also, where do I set these Spark configurations since this > will be executed inside a YARN container? On the “client” machine via > spark-env.sh? Do I pass these as command line arguments to spark-submit? Do > I set them explicitly on my SparkConf? > > Thanks in advance. > > mn > >> On Sep 25, 2014, at 9:13 PM, Andrew Ash <and...@andrewash.com> wrote: >> >> Matt you should be able to set an HDFS path so you'll get logs written to a >> unified place instead of to local disk on a random box on the cluster. >> >>> On Thu, Sep 25, 2014 at 1:38 PM, Matt Narrell <matt.narr...@gmail.com> >>> wrote: >>> How does this work with a cluster manager like YARN? >>> >>> mn >>> >>>> On Sep 25, 2014, at 2:23 PM, Andrew Or <and...@databricks.com> wrote: >>>> >>>> Hi Harsha, >>>> >>>> You can turn on `spark.eventLog.enabled` as documented here: >>>> http://spark.apache.org/docs/latest/monitoring.html. Then, if you are >>>> running standalone mode, you can access the finished SparkUI through the >>>> Master UI. Otherwise, you can start a HistoryServer to display finished >>>> UIs. >>>> >>>> -Andrew >>>> >>>> 2014-09-25 12:55 GMT-07:00 Harsha HN <99harsha.h....@gmail.com>: >>>>> Hi, >>>>> >>>>> Details laid out in Spark UI for the job in progress is really >>>>> interesting and very useful. >>>>> But this gets vanished once the job is done. >>>>> Is there a way to get job details post processing? >>>>> >>>>> Looking for Spark UI data, not standard input,output and error info. >>>>> >>>>> Thanks, >>>>> Harsha >
