Thanks Philip, I don’t want to go the JobLogger way (too hacky ;) )
In version 1.0, if i’m not mistaken, you can even do what I’m asking for, since they removed the “private” for TaskInfo and such and replaced it with the “@DeveloperApi” annotation. I was looking for a simple way to do this in 0.9.1, but thanks anyway! Pierre On 23 May 2014, at 17:41, Philip Ogren [via Apache Spark User List] <ml-node+s1001560n6326...@n3.nabble.com> wrote: > Hi Pierre, > > I asked a similar question on this list about 6 weeks ago. Here is one > answer I got that is of particular note: > > In the upcoming release of Spark 1.0 there will be a feature that provides > for exactly what you describe: capturing the information displayed on the UI > in JSON. More details will be provided in the documentation, but for now, > anything before 0.9.1 can only go through JobLogger.scala, which outputs > information in a somewhat arbitrary format and will be deprecated soon. If > you find this feature useful, you can test it out by building the master > branch of Spark yourself, following the instructions in > https://github.com/apache/spark/pull/42. > > > > On 05/22/2014 08:51 AM, Pierre B wrote: >> Is there a simple way to monitor the overall progress of an action using >> SparkListener or anything else? >> >> I see that one can name an RDD... Could that be used to determine which >> action triggered a stage, ... ? >> >> >> Thanks >> >> Pierre >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > If you reply to this email, your message will be added to the discussion > below: > http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6326.html > To unsubscribe from Use SparkListener to get overall progress of an action, > click here. > NAML -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkListener-to-get-overall-progress-of-an-action-tp6256p6327.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
