[
https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Saisai Shao resolved SPARK-22123.
---------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
Issue resolved by pull request 19338
[https://github.com/apache/spark/pull/19338]
> Add latest failure reason for task set blacklist
> ------------------------------------------------
>
> Key: SPARK-22123
> URL: https://issues.apache.org/jira/browse/SPARK-22123
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
> Affects Versions: 2.1.0, 2.2.0
> Reporter: zhoukang
> Fix For: 2.3.0
>
>
> Till now , every job which aborted by completed blacklist just show log like
> below which has no more information:
> {code:java}
> Aborting $taskSet because task $indexInTaskSet (partition $partition) cannot
> run anywhere due to node and executor blacklist. Blacklisting behavior cannot
> run anywhere due to node and executor blacklist.Blacklisting behavior can be
> configured via spark.blacklist.*."
> {code}
> We could add most recent failure reason for taskset blacklist which can be
> showed on spark ui to let user know failure reason directly.
> An example after modifying:
> {code:java}
> Aborting TaskSet 0.0 because task 0 (partition 0) cannot run anywhere due to
> node and executor blacklist.
> Most recent failure:
> Some(Lost task 0.1 in stage 0.0 (TID 3,xxx, executor 1):
> java.lang.Exception: Fake error!
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:73)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:305)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ).
> Blacklisting behavior can be configured via spark.blacklist.*.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
