squito commented on a change in pull request #22806: [SPARK-25250][CORE] : Late
zombie task completions handled correctly even before new taskset launched
URL: https://github.com/apache/spark/pull/22806#discussion_r251947631
##########
File path:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
##########
@@ -1102,110 +1102,6 @@ class TaskSchedulerImplSuite extends SparkFunSuite
with LocalSparkContext with B
}
}
- test("Completions in zombie tasksets update status of non-zombie taskset") {
Review comment:
I think adding this
[here](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala#L118),
to the mocked DAGScheduler impl inside TaskSchedulerImplSuite, is what is
needed to get this to pass (all tests within spark.scheduler pass for me with
this change):
```scala
override def taskEnded(
task: Task[_],
reason: TaskEndReason,
result: Any,
accumUpdates: Seq[AccumulatorV2[_, _]],
taskInfo: TaskInfo): Unit = {
if (reason == Success) {
// For SPARK-23433 / SPARK-25250, need to make DAGScheduler lets
all tasksets know
// about complete partitions. Super implementation is not enough,
because we've mocked
// out too much of the rest of the DAGScheduler.
taskScheduler.completeTasks(task.partitionId, task.stageId,
taskInfo)
}
super.taskEnded(task, reason, result, accumUpdates, taskInfo)
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
