Re: Tracking / estimating job progress

2016-05-13 Thread Dood
On 5/13/2016 10:39 AM, Anthony May wrote: It looks like it might only be available via REST, http://spark.apache.org/docs/latest/monitoring.html#rest-api Nice, thanks! On Fri, 13 May 2016 at 11:24 Dood@ODDO > wrote: On 5/13/2016

Re: Tracking / estimating job progress

2016-05-13 Thread Anthony May
It looks like it might only be available via REST, http://spark.apache.org/docs/latest/monitoring.html#rest-api On Fri, 13 May 2016 at 11:24 Dood@ODDO wrote: > On 5/13/2016 10:16 AM, Anthony May wrote: > > >

Re: Tracking / estimating job progress

2016-05-13 Thread Dood
On 5/13/2016 10:16 AM, Anthony May wrote: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker Might be useful How do you use it? You cannot instantiate the class - is the constructor private? Thanks! On Fri, 13 May 2016 at 11:11 Ted Yu

Re: Tracking / estimating job progress

2016-05-13 Thread Anthony May
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkStatusTracker Might be useful On Fri, 13 May 2016 at 11:11 Ted Yu wrote: > Have you looked > at core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ? > > Cheers > > On Fri,

Re: Tracking / estimating job progress

2016-05-13 Thread Ted Yu
Have you looked at core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ? Cheers On Fri, May 13, 2016 at 10:05 AM, Dood@ODDO wrote: > I provide a RESTful API interface from scalatra for launching Spark jobs - > part of the functionality is tracking these

Tracking / estimating job progress

2016-05-13 Thread Dood
I provide a RESTful API interface from scalatra for launching Spark jobs - part of the functionality is tracking these jobs. What API is available to track the progress of a particular spark application? How about estimating where in the total job progress the job is? Thanks!