GitHub user zhonghaihua opened a pull request:
https://github.com/apache/spark/pull/10794
initialize executorIdCounter after ApplicationMaster killed for max nâ¦
Currently, when max number of executor failures reached the
`maxNumExecutorFailures`, `ApplicationMaster` will be killed and re-register
another one.This time, `YarnAllocator` will be created a new instance.
But, the value of property `executorIdCounter` in `YarnAllocator` will
reset to `0`. Then the Id of new executor will starting from `1`. This will
confuse with the executor has already created before, which will cause
FetchFailedException.
This PR introduce a mechanism to initialize `executorIdCounter` after
`ApplicationMaster` killed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhonghaihua/spark
initExecutorIdCounterAfterAMKilled
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10794.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10794
----
commit 30048ac7ac9fc95edc1b936076415dea335848ef
Author: zhonghaihua <[email protected]>
Date: 2016-01-17T12:46:44Z
initialize executorIdCounter after ApplicationMaster killed for max number
of executor failures reached
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
