GitHub user CrazyJvm opened a pull request:
https://github.com/apache/spark/pull/1282
fix spark.yarn.max.executor.failures explaination
According to
'''scala
private val maxNumExecutorFailures =
sparkConf.getInt("spark.yarn.max.executor.failures",
sparkConf.getInt("spark.yarn.max.worker.failures",
math.max(args.numExecutors * 2, 3)))
''''
default value should be numExecutors * 2, with minimum of 3, and it's same
to the config
`spark.yarn.max.worker.failures`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/CrazyJvm/spark yarn-doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1282.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1282
----
commit a4b2e27b0c2d2345a60ba66943b219968465b48a
Author: CrazyJvm <[email protected]>
Date: 2014-07-02T06:59:48Z
fix configuration spark.yarn.max.executor.failures
commit 2900d234c6ebb90a5c4601083ddf8d329a2ee99d
Author: CrazyJvm <[email protected]>
Date: 2014-07-02T07:04:51Z
fix style
commit 211f1302aa6d57b07a7b2d3b7cd4ab21e6d50bbd
Author: CrazyJvm <[email protected]>
Date: 2014-07-02T07:06:28Z
fix html tag
commit 86effa612d2ec9ae991b43e229c4ed266e6605a6
Author: CrazyJvm <[email protected]>
Date: 2014-07-02T07:15:08Z
change expression
commit c438aecdec8ce90cb839b7c9aa8260ff4d3c62ba
Author: CrazyJvm <[email protected]>
Date: 2014-07-02T07:18:18Z
fix style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
