Repository: spark
Updated Branches:
  refs/heads/branch-2.2 e87ffcaa3 -> d02331452


[SPARK-21503][UI] Spark UI shows incorrect task status for a killed Executor 
Process

The executor tab on Spark UI page shows task as completed when an executor 
process that is running that task is killed using the kill command.
Added the case ExecutorLostFailure which was previously not there, thus, the 
default case would be executed in which case, task would be marked as 
completed. This case will consider all those cases where executor connection to 
Spark Driver was lost due to killing the executor process, network connection 
etc.

## How was this patch tested?
Manually Tested the fix by observing the UI change before and after.
Before:
<img width="1398" alt="screen shot-before" 
src="https://user-images.githubusercontent.com/22228190/28482929-571c9cea-6e30-11e7-93dd-728de5cdea95.png";>
After:
<img width="1385" alt="screen shot-after" 
src="https://user-images.githubusercontent.com/22228190/28482964-8649f5ee-6e30-11e7-91bd-2eb2089c61cc.png";>

Please review http://spark.apache.org/contributing.html before opening a pull 
request.

Author: pgandhi <pgan...@yahoo-inc.com>
Author: pgandhi999 <parthkgand...@gmail.com>

Closes #18707 from pgandhi999/master.

(cherry picked from commit f016f5c8f6c6aae674e9905a5c0b0bede09163a4)
Signed-off-by: Wenchen Fan <wenc...@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d0233145
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d0233145
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d0233145

Branch: refs/heads/branch-2.2
Commit: d0233145208eb6afcd9fe0c1c3a9dbbd35d7727e
Parents: e87ffca
Author: pgandhi <pgan...@yahoo-inc.com>
Authored: Wed Aug 9 13:46:06 2017 +0800
Committer: Wenchen Fan <wenc...@databricks.com>
Committed: Wed Aug 9 13:46:25 2017 +0800

----------------------------------------------------------------------
 .../org/apache/spark/ui/exec/ExecutorsTab.scala | 24 ++++++++++----------
 1 file changed, 12 insertions(+), 12 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d0233145/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala 
b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
index aabf6e0..770da22 100644
--- a/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
+++ b/core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
@@ -19,7 +19,7 @@ package org.apache.spark.ui.exec
 
 import scala.collection.mutable.{LinkedHashMap, ListBuffer}
 
-import org.apache.spark.{ExceptionFailure, Resubmitted, SparkConf, 
SparkContext}
+import org.apache.spark.{Resubmitted, SparkConf, SparkContext}
 import org.apache.spark.annotation.DeveloperApi
 import org.apache.spark.scheduler._
 import org.apache.spark.storage.{StorageStatus, StorageStatusListener}
@@ -131,17 +131,17 @@ class ExecutorsListener(storageStatusListener: 
StorageStatusListener, conf: Spar
     if (info != null) {
       val eid = info.executorId
       val taskSummary = executorToTaskSummary.getOrElseUpdate(eid, 
ExecutorTaskSummary(eid))
-      taskEnd.reason match {
-        case Resubmitted =>
-          // Note: For resubmitted tasks, we continue to use the metrics that 
belong to the
-          // first attempt of this task. This may not be 100% accurate because 
the first attempt
-          // could have failed half-way through. The correct fix would be to 
keep track of the
-          // metrics added by each attempt, but this is much more complicated.
-          return
-        case _: ExceptionFailure =>
-          taskSummary.tasksFailed += 1
-        case _ =>
-          taskSummary.tasksComplete += 1
+      // Note: For resubmitted tasks, we continue to use the metrics that 
belong to the
+      // first attempt of this task. This may not be 100% accurate because the 
first attempt
+      // could have failed half-way through. The correct fix would be to keep 
track of the
+      // metrics added by each attempt, but this is much more complicated.
+      if (taskEnd.reason == Resubmitted) {
+        return
+      }
+      if (info.successful) {
+        taskSummary.tasksComplete += 1
+      } else {
+        taskSummary.tasksFailed += 1
       }
       if (taskSummary.tasksActive >= 1) {
         taskSummary.tasksActive -= 1


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

Reply via email to