There is the Async API ( https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala), which makes use of FutureAction ( https://github.com/clearstorydata/spark/blob/master/core/src/main/scala/org/apache/spark/FutureAction.scala). You could also wrap up your Jobs in Futures on your own.
On Mon, Sep 14, 2015 at 11:37 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > As of now i think its a no. Not sure if its a naive approach, but yes you > can have a separate program to keep an eye in the webui (possibly parsing > the content) and make it trigger the kill task/job once it detects a lag. > (Again you will have to figure out the correct numbers before killing any > job) > > Thanks > Best Regards > > On Mon, Sep 14, 2015 at 10:40 PM, Dmitry Goldenberg < > dgoldenberg...@gmail.com> wrote: > >> Is there a way in Spark to automatically terminate laggard "stage's", >> ones that appear to be hanging? In other words, is there a timeout for >> processing of a given RDD? >> >> In the Spark GUI, I see the "kill" function for a given Stage under >> 'Details for Job <...>". >> >> Is there something in Spark that would identify and kill laggards >> proactively? >> >> Thanks. >> > >
