Sorry, meant to include the link: http://spark.incubator.apache.org/docs/latest/scala-programming-guide.html#accumulators
On Mon, Nov 4, 2013 at 1:15 PM, Mark Hamstra <[email protected]>wrote: > You probably want to be looking at accumulators. > > > On Mon, Nov 4, 2013 at 12:22 PM, Markus Losoi <[email protected]>wrote: > >> Hi >> >> Is it possible for a driver program to receive intermediary results of a >> Spark operation? If, e.g., a long map() operation is in progress, can the >> driver become aware of some of the (key, value) pairs before all of them >> are >> computed? >> >> There seems to be SparkListener interface that has an onTaskEnd() event >> [1]. >> However, the documentation is somewhat sparse on what kind of information >> is >> included in a SparkListenerTaskEnd object [2]. >> >> [1] >> >> http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched >> uler/SparkListener.html<http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/scheduler/SparkListener.html> >> [2] >> >> http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched >> uler/SparkListenerTaskEnd.html<http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/scheduler/SparkListenerTaskEnd.html> >> >> Best regards, >> Markus Losoi ([email protected]) >> >> >
