Andrew Or created SPARK-4020:
--------------------------------
Summary: Failed executor not properly removed if it has not run
tasks
Key: SPARK-4020
URL: https://issues.apache.org/jira/browse/SPARK-4020
Project: Spark
Issue Type: Bug
Reporter: Andrew Or
Assignee: Andrew Or
If an executor has not run tasks, then the DAG scheduler won't remove the block
manager that belongs to this executor. This is because we don't add the
executor to `activeExecutorIds` in `TaskSchedulerImpl` until we have
successfully scheduled a task on the executor.
The motivation for fixing this is to avoid having the `BlockManagerMasterActor`
to remove the associated block manager through a timeout when it can just do so
right away.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
