Eric Vandenberg created SPARK-21219:
---------------------------------------
Summary: Task retry occurs on same executor due to race condition
with blacklisting
Key: SPARK-21219
URL: https://issues.apache.org/jira/browse/SPARK-21219
Project: Spark
Issue Type: Bug
Components: Scheduler
Affects Versions: 2.1.1
Reporter: Eric Vandenberg
Priority: Minor
When a task fails it is added into the pending task list and corresponding
black list policy is enforced (ie, specifying if it can/can't run on a
particular node/executor/etc.) Unfortunately the ordering is such that
retrying the task could assign the task to the same executor, which,
incidentally could be shutting down and immediately fail the retry. Instead
the black list state should be updated and then the task assigned, ensuring
that the black list policy is properly enforced.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
