You probably want to be looking at accumulators.
On Mon, Nov 4, 2013 at 12:22 PM, Markus Losoi <[email protected]>wrote: > Hi > > Is it possible for a driver program to receive intermediary results of a > Spark operation? If, e.g., a long map() operation is in progress, can the > driver become aware of some of the (key, value) pairs before all of them > are > computed? > > There seems to be SparkListener interface that has an onTaskEnd() event > [1]. > However, the documentation is somewhat sparse on what kind of information > is > included in a SparkListenerTaskEnd object [2]. > > [1] > > http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched > uler/SparkListener.html > [2] > > http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched > uler/SparkListenerTaskEnd.html > > Best regards, > Markus Losoi ([email protected]) > >
