Hi,
One of the rational behind killing the app can be to avoid skewness in
data.
I have created this issue (https://issues.apache.org/jira/browse/SPARK-6735)
to provide options for disabling this behaviour, as well as making the
number of executor's failure to be relative with respect to a
What's the advantage of killing an application for lack of resources?
I think the rationale behind killing an app based on executor failures is
that, if we see a lot of them in a short span of time, it means there's
probably something going wrong in the app or on the cluster.
On Wed, Apr 1, 2015
Hi,
In spark over YARN, there is a property spark.yarn.max.executor.failures
which controls the maximum number of executor's failure an application will
survive.
If number of executor's failures ( due to any reason like OOM or machine
failure etc ), increases this value then applications quits.
Hi,
Thanks Sandy.
Another way to look at this is that would we like to have our long running
application to die?
So let's say, we create a window of around 10 batches, and we are using
incremental kind of operations inside our application, as restart here is a
relatively more costlier, so
That's a good question, Twinkle.
One solution could be to allow a maximum number of failures within any
given time span. E.g. a max failures per hour property.
-Sandy
On Tue, Mar 31, 2015 at 11:52 PM, twinkle sachdeva
twinkle.sachd...@gmail.com wrote:
Hi,
In spark over YARN, there is a
