Hello
We're running some experiments with Spark (v1.4) and have some questions
about its scheduling behavior. I am hoping someone can answer the
following questions.
What is a task set? It is mentioned in the Spark logs we get from our runs
but we can't seem to find a definition and how it
Task set is a set of tasks within one stage.
Executor will be killed when it is idle for a period of time (default is
60s). The problem you mentioned is bug, scheduler should not allocate tasks
on this to-be killed executors. I think it is fixed in 1.5.
Thanks
Saisai
On Thu, Sep 17, 2015 at
