Yeah, actually for that I used directly codahale with my own stuffs using the Akka system from within Spark itself.
So the workers send messages back to a bunch of actors on the driver which are using codahale metrics. This way I can collect what/how an executor do/did, but also I can aggregate all executors metrics at once (via dedicated aggregation purposed codahale metrics). However, I didn't had time to dig enough in Spark to see with I could reuse the SparkListener system itself -- which is kind-of doing the same thing, but w/o akka AFAICT => where I can see that TaskMetrics are collected by task within the context/granularity of a Stage. Than aggregation looks like being done in a built-in (Queued) Bus. So I'll let someone else report how this could be extended, but my gut feeling that it won't be straightforward. hth (with respect of my limited knowledge of these internals ^^) cheers, aℕdy ℙetrella about.me/noootsab [image: aℕdy ℙetrella on about.me] <http://about.me/noootsab> On Thu, May 22, 2014 at 5:02 PM, Pierre B < pierre.borckm...@realimpactanalytics.com> wrote: > Hi Andy! > > Yes Spark UI provides a lot of interesting informations for debugging > purposes. > > Here I’m trying to integrate a simple progress monitoring in my app ui. > > I’m typically running a few “jobs” (or rather actions), and I’d like to be > able to display the progress of each of those in my ui. > > I don’t really see how i could do that using SparkListener for the moment … > > Thanks for your help! > > Cheers! > > > > > *Pierre Borckmans* > Software team > > *Real**Impact* Analytics *| *Brussels Office > www.realimpactanalytics.com *| *[hidden > email]<http://user/SendEmail.jtp?type=node&node=6259&i=0> > > *FR *+32 485 91 87 31 *| **Skype* pierre.borckmans > > > > > > > On 22 May 2014, at 16:58, andy petrella [via Apache Spark User List] <[hidden > email] <http://user/SendEmail.jtp?type=node&node=6259&i=1>> wrote: > > SparkListener offers good stuffs. > But I also completed it with another metrics stuffs on my own that use > Akka to aggregate metrics from anywhere I'd like to collect them (without > any deps on ganglia yet on Codahale). > However, this was useful to gather some custom metrics (from within the > tasks then) not really to collect overall monitoring information about the > spark thingies themselves. > For that Spark UI offers already a pretty good insight no? > > Cheers, > > aℕdy ℙetrella > about.me/noootsab > [image: aℕdy ℙetrella on about.me] > > <http://about.me/noootsab> > > > On Thu, May 22, 2014 at 4:51 PM, Pierre B <<a > href="x-msg://7/user/SendEmail.jtp?type=node&node=6258&i=0" > target="_top" rel="nofollow" link="external">[hidden email]> wrote: > >> Is there a simple way to monitor the overall progress of an action using >> SparkListener or anything else? >> >> I see that one can name an RDD... Could that be used to determine which >> action triggered a stage, ... ? >> >> >> Thanks >> >> Pierre >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html > To unsubscribe from Use SparkListener to get overall progress of an > action, click here. > NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > > ------------------------------ > View this message in context: Re: Use SparkListener to get overall > progress of an > action<http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6259.html> > > Sent from the Apache Spark User List mailing list > archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com. >
