Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/22288#discussion_r226478269
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -679,20 +679,28 @@ private[spark] class TaskSetManager(
}
}
}
+
if (blacklistedEverywhere) {
- val partition = tasks(indexInTaskSet).partitionId
- abort(s"""
- |Aborting $taskSet because task $indexInTaskSet (partition
$partition)
- |cannot run anywhere due to node and executor blacklist.
- |Most recent failure:
- |${taskSetBlacklist.getLatestFailureReason}
- |
- |Blacklisting behavior can be configured via
spark.blacklist.*.
- |""".stripMargin)
+ return Some(indexInTaskSet)
--- End diff --
not a big deal, but I think you can get rid of the early return and make
this more idiomatic by changing it to
```scala
taskSetBlacklistHelperOpt.flatMap { taskSetBlacklist =>
...
pendingTask.find { indexInTaskSet =>
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
