GitHub user nyogesh opened a pull request:
https://github.com/apache/spark/pull/21150
[SPARK-24075][MESOS]
## What changes were proposed in this pull request?
* Spark drivers run on mesos with supervise enabled will try the driver
indefinitely on failures. This PR is to optionally limit the number of retries
to a configurable number.
* Introducing sparkConf "spark.mesos.driver.supervise.maxRetries" which
limits the number of times the driver can be retried. The default behavior is
for the driver to be retried indefinitely.
* When the check is made to see if supervise is enabled and the job has to
be retried, an additional check is made to see if the allowable retries have
been exceeded.
* Added unit tests for method hasDriverExceededRetries.
* Added documentation for "spark.mesos.driver.supervise.maxRetries".
## How was this patch tested?
Added unit tests
Built spark package, dockerized it and deployed it as a service on Mesos.
Once spark service was running on mesos, a series of drivers were submitted
1. With supervise disabled, ran a successful driver
2. With supervise disabled, ran a failure driver
3. With supervise enabled, ran a successful driver
4. With supervise enabled and without setting
"spark.mesos.driver.supervise.maxRetries", ran failure driver
5. With supervise enabled and setting
"spark.mesos.driver.supervise.maxRetries=3", ran a failure driver
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nyogesh/spark SPARK-24075
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21150.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21150
----
commit fcfe1e80842f758b4ee0004b7ce5730bca40f543
Author: Yogesh Natarajan <ynataraj@...>
Date: 2018-04-25T07:08:55Z
[SPARK-24075][MESOS]
* Spark drivers run on mesos with supervise enabled will try to run the
driver indefinitely on failures. This PR is to optionally limit the number of
retries to a configurable number.
* Introducing sparkConf "spark.mesos.driver.supervise.maxRetries" which
limits the number of times the driver can be retried. The default behavior is
for the driver to be retried indefinitely.
* When a check is made to see if supervise is enabled, an additional check
is made to see if the allowable retries have been exceeded.
* Added unit tests for method hasDriverExceededRetries.
* Added documentation for "spark.mesos.driver.supervise.maxRetries".
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
