Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/13603#discussion_r67740258
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -96,22 +120,13 @@ class TaskSchedulerImplSuite extends SparkFunSuite
with LocalSparkContext with L
taskDescriptions =
taskScheduler.resourceOffers(multiCoreWorkerOffers).flatten
assert(1 === taskDescriptions.length)
assert("executor0" === taskDescriptions(0).executorId)
+ assert(!failedTaskSet)
}
test("Scheduler does not crash when tasks are not serializable") {
- sc = new SparkContext("local", "TaskSchedulerImplSuite")
--- End diff --
Unfortunately, this case will also trigger a failure with the msg "Aborting
TaskSet ... due to blacklist". I'm pretty sure this is a bug in
`TaskSchedulerImpl`, but I am so shocked by it I'd like a quick sanity check.
What's going on here is that no tasks have been accepted, so `executorsByHost`
never actually adds the new executor in
[`resourceOfferSingleTaskSet`](https://github.com/apache/spark/blob/9613424898fd2a586156bc4eb48e255749774f20/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L266).
But `executorsByHost` has already added the host at the [beginning of
`resourceOffers`, just not the
executor](https://github.com/apache/spark/blob/9613424898fd2a586156bc4eb48e255749774f20/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L298).
But isn't there a bug in `resourceOffers` -- shouldn't that loop be
updating `newExecAvailable` even if an executor is added to an already existing
host? I expect this to actually be quite common under dynamic allocation. The
end result is that locality preferences aren't properly updated, and
`failedEpochs` aren't updated correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
