That would be great, Mayur, thanks! Anyhow, to be more specific, my question really was the following:
Is there any way to link events in the SparkListener to an action triggered in your code? Cheers Pierre Borckmans Software team RealImpact Analytics | Brussels Office www.realimpactanalytics.com | pierre.borckm...@realimpactanalytics.com FR +32 485 91 87 31 | Skype pierre.borckmans On 23 May 2014, at 10:17, Mayur Rustagi <mayur.rust...@gmail.com> wrote: > We have an internal patched version of Spark webUI which exports application > related data as Json. We use monitoring systems as well as alternate UI for > that json data for our specific application. Found it much cleaner. Can > provide 0.9.1 version. > Would submit as a pull request soon. > > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi > > > > On Fri, May 23, 2014 at 10:57 AM, Chester <chesterxgc...@yahoo.com> wrote: > This is something we are interested as well. We are planning to investigate > more on this. If someone has suggestions, we would love to hear. > > Chester > > Sent from my iPad > > On May 22, 2014, at 8:02 AM, Pierre B > <pierre.borckm...@realimpactanalytics.com> wrote: > >> Hi Andy! >> >> Yes Spark UI provides a lot of interesting informations for debugging >> purposes. >> >> Here I’m trying to integrate a simple progress monitoring in my app ui. >> >> I’m typically running a few “jobs” (or rather actions), and I’d like to be >> able to display the progress of each of those in my ui. >> >> I don’t really see how i could do that using SparkListener for the moment … >> >> Thanks for your help! >> >> Cheers! >> >> >> >> >> Pierre Borckmans >> Software team >> >> RealImpact Analytics | Brussels Office >> www.realimpactanalytics.com | [hidden email] >> >> FR +32 485 91 87 31 | Skype pierre.borckmans >> >> >> >> >> >> >> On 22 May 2014, at 16:58, andy petrella [via Apache Spark User List] >> <[hidden email]> wrote: >> >>> SparkListener offers good stuffs. >>> But I also completed it with another metrics stuffs on my own that use Akka >>> to aggregate metrics from anywhere I'd like to collect them (without any >>> deps on ganglia yet on Codahale). >>> However, this was useful to gather some custom metrics (from within the >>> tasks then) not really to collect overall monitoring information about the >>> spark thingies themselves. >>> For that Spark UI offers already a pretty good insight no? >>> >>> Cheers, >>> >>> aℕdy ℙetrella >>> about.me/noootsab >>> >>> >>> >>> >>> On Thu, May 22, 2014 at 4:51 PM, Pierre B <<a >>> href="x-msg://7/user/SendEmail.jtp?type=node&node=6258&i=0" >>> target="_top" rel="nofollow" link="external">[hidden email]> wrote: >>> Is there a simple way to monitor the overall progress of an action using >>> SparkListener or anything else? >>> >>> I see that one can name an RDD... Could that be used to determine which >>> action triggered a stage, ... ? >>> >>> >>> Thanks >>> >>> Pierre >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> >>> >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6258.html >>> To unsubscribe from Use SparkListener to get overall progress of an action, >>> click here. >>> NAML >> >> >> View this message in context: Re: Use SparkListener to get overall progress >> of an action >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >
