Driver timing out laggards seems like a reasonable way of handling laggards. Are there any challenges because of which driver does not do it today? Is there a JIRA for this? I couldn't find one.
On Tue, Sep 15, 2015 at 12:07 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > As of now i think its a no. Not sure if its a naive approach, but yes you > can have a separate program to keep an eye in the webui (possibly parsing > the content) and make it trigger the kill task/job once it detects a lag. > (Again you will have to figure out the correct numbers before killing any > job) > > Thanks > Best Regards > > On Mon, Sep 14, 2015 at 10:40 PM, Dmitry Goldenberg < > dgoldenberg...@gmail.com> wrote: > >> Is there a way in Spark to automatically terminate laggard "stage's", >> ones that appear to be hanging? In other words, is there a timeout for >> processing of a given RDD? >> >> In the Spark GUI, I see the "kill" function for a given Stage under >> 'Details for Job <...>". >> >> Is there something in Spark that would identify and kill laggards >> proactively? >> >> Thanks. >> > >
