Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21656#discussion_r200231460
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -772,6 +772,12 @@ private[spark] class TaskSetManager(
private[scheduler] def markPartitionCompleted(partitionId: Int): Unit = {
partitionToIndex.get(partitionId).foreach { index =>
if (!successful(index)) {
+ if (speculationEnabled) {
+ taskAttempts(index).headOption.map { info =>
+ info.markFinished(TaskState.FINISHED, clock.getTimeMillis())
+ successfulTaskDurations.insert(info.duration)
--- End diff --
what's the normal code path to update task durations?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
