Hi Is it possible for a driver program to receive intermediary results of a Spark operation? If, e.g., a long map() operation is in progress, can the driver become aware of some of the (key, value) pairs before all of them are computed?
There seems to be SparkListener interface that has an onTaskEnd() event [1]. However, the documentation is somewhat sparse on what kind of information is included in a SparkListenerTaskEnd object [2]. [1] http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched uler/SparkListener.html [2] http://spark.incubator.apache.org/docs/0.8.0/api/core/org/apache/spark/sched uler/SparkListenerTaskEnd.html Best regards, Markus Losoi ([email protected])
