Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20493#discussion_r166181455
--- Diff: core/src/main/scala/org/apache/spark/status/AppStatusUtils.scala
---
@@ -17,16 +17,23 @@
package org.apache.spark.status
-import org.apache.spark.status.api.v1.{TaskData, TaskMetrics}
+import org.apache.spark.status.api.v1.TaskData
private[spark] object AppStatusUtils {
+ private val TASK_FINISHED_STATES = Set("FAILED", "KILLED", "SUCCESS")
+
+ private def isTaskFinished(task: TaskData): Boolean = {
+ TASK_FINISHED_STATES.contains(task.status)
+ }
+
def schedulerDelay(task: TaskData): Long = {
- if (task.taskMetrics.isDefined && task.duration.isDefined) {
+ if (isTaskFinished(task) && task.taskMetrics.isDefined &&
task.duration.isDefined) {
--- End diff --
Logically `duration` should be set for running tasks, to indicate how long
a task has been run.
I feel it's safer to keep `task.duration.isDefined`, as we call
`task.duration.get` below.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
