GitHub user XuTingjun opened a pull request:
https://github.com/apache/spark/pull/8741
[SPARK-10586] fix bug: BlockManager ca't be removed when it is
re-registered, then disassociats
Scene: When the executor has been removed, but it still exists on the
SparkUI web.
Process:
1. Driver Lost executor because heartbeat timed out;
2. Executor received SIGNAL 15: SIGTERM and **re-registered** the
blockmanager;
3. Driver Lost executor because remote Rpc client disassociated and attempt
to remove blockmanager but not success;
Reason:
The first time lost executor, the [SchedulerBackend
reviveOffers](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L481),
when the host also has other executors, it won't [add this
executor](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L288).
So the second time lost executor, DAGScheduler won't remove BlockManager
because the executorId has been in ```failedEpoch```.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/XuTingjun/spark SPARK-10586
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8741.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8741
----
commit bb562a269b954639a8f254d79782a2f7d0e209c9
Author: xutingjun <[email protected]>
Date: 2015-09-14T07:43:29Z
when executor adds, delete it from failedEpoch
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
