GitHub user KaiXinXiaoLei opened a pull request:
https://github.com/apache/spark/pull/7559
[SPARK-9209] Using executor allocation, a executor is removed but it exists
in ExecutorsPage of the web ui
I set "spark.dynamicAllocation.enabled = trueâ, and run a big job.
After some minutes, in driver, a executor is asked to remove. Then it's
removed successfully, and the process of this executor is not exist. But it
exists in ExecutorPage of the web ui.
The log in driver :
2015-07-17 11:48:14,543 | INFO |
[sparkDriver-akka.actor.default-dispatcher-3] | Removing block manager
BlockManagerId(264, 172.1.1.8, 23811)
2015-07-17 11:48:14,543 | INFO | [dag-scheduler-event-loop] | Removed 264
successfully in removeExecutor
2015-07-17 11:48:21,226 | INFO |
[sparkDriver-akka.actor.default-dispatcher-3] | Registering block manager
172.1.1.8:23811 with 10.4 GB RAM, BlockManagerId(264, 172.1.1.8, 23811)
2015-07-17 11:48:21,228 | INFO |
[sparkDriver-akka.actor.default-dispatcher-3] | Added broadcast_781_piece0 in
memory on 172.1.1.8:23811 (size: 38.6 KB, free: 10.4 GB)
2015-07-17 11:48:35,277 | ERROR |
[sparkDriver-akka.actor.default-dispatcher-16] | Lost executor 264 on
datasight-195: remote Rpc client disassociated
2015-07-17 11:48:35,277 | WARN |
[sparkDriver-akka.actor.default-dispatcher-4] | Association with remote system
[akka.tcp://sparkExecutor@datasight-195:23929] has failed, address is now gated
for [5000] ms. Reason is: [Disassociated].
2015-07-17 11:48:35,277 | INFO |
[sparkDriver-akka.actor.default-dispatcher-16] | Re-queueing tasks for 264 from
TaskSet 415.0
2015-07-17 11:48:35,804 | INFO | [SparkListenerBus] | Existing executor
264 has been removed (new total is 10)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/KaiXinXiaoLei/spark executorpageError
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7559.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7559
----
commit cc2b0a790fa64c67c647f5ff27f8050acee5c409
Author: KaiXinXiaoLei <[email protected]>
Date: 2015-07-21T06:16:08Z
change file
commit f1a20cbb488057e9368285ba6dfdc35647f1369c
Author: KaiXinXiaoLei <[email protected]>
Date: 2015-07-21T06:23:24Z
change file
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
