[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-17 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234400684
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -565,7 +567,11 @@ private[spark] class AppStatusListener(
   if (metricsDelta != null) {
 esummary.metrics = LiveEntityHelpers.addMetrics(esummary.metrics, 
metricsDelta)
   }
-  conditionalLiveUpdate(esummary, now, removeStage)
+
+  val isLastTask = 
(stage.activeTasksPerExecutor(event.taskInfo.executorId) == 0) &&
+((stage.status == v1.StageStatus.COMPLETE) || (stage.status == 
v1.StageStatus.FAILED))
--- End diff --

This issue occurs, when the taskEvent comes after stageEnd. Because during 
'OnStageCompletd' event, we are writing all the esummary to the store. So, 
'OnTaskEnd' method, we just need to force write only if the stageCompleted 
event already have happened.

Yes. the stageEnd check isn't really required, as we always update on the 
last task of each executors of the particular stage. I updated



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234399022
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(
+  SparkListenerTaskEnd(stage.stageId, stage.attemptId, "taskType",
+ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+stage.failureReason = Some("Failed")
+listener.onStageCompleted(SparkListenerStageCompleted(stage))
+time += 1
+listener.onJobEnd(SparkListenerJobEnd(1, time, JobFailed(new 
RuntimeException("Bad Executor"
+
+tasks.filter(_.index >= 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(SparkListenerTaskEnd(stage.stageId, 
stage.attemptId, "taskType",
+  ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+val esummary = 
store.view(classOf[ExecutorStageSummaryWrapper]).asScala.map(_.info)
+esummary.foreach {
+  execSummary => assert(execSummary.failedTasks == 2)
--- End diff --

Done


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234399018
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(
+  SparkListenerTaskEnd(stage.stageId, stage.attemptId, "taskType",
+ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+stage.failureReason = Some("Failed")
+listener.onStageCompleted(SparkListenerStageCompleted(stage))
+time += 1
+listener.onJobEnd(SparkListenerJobEnd(1, time, JobFailed(new 
RuntimeException("Bad Executor"
+
+tasks.filter(_.index >= 2).foreach { task =>
+time += 1
--- End diff --

Updated


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234399010
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
--- End diff --

Updated the code


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234399006
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
--- End diff --

Done.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234382254
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(
+  SparkListenerTaskEnd(stage.stageId, stage.attemptId, "taskType",
+ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+stage.failureReason = Some("Failed")
+listener.onStageCompleted(SparkListenerStageCompleted(stage))
+time += 1
+listener.onJobEnd(SparkListenerJobEnd(1, time, JobFailed(new 
RuntimeException("Bad Executor"
+
+tasks.filter(_.index >= 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(SparkListenerTaskEnd(stage.stageId, 
stage.attemptId, "taskType",
+  ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+val esummary = 
store.view(classOf[ExecutorStageSummaryWrapper]).asScala.map(_.info)
+esummary.foreach {
+  execSummary => assert(execSummary.failedTasks == 2)
--- End diff --

keep `execSummary =>` in the previous line.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234382145
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
--- End diff --

Use the config constant, like the existing code.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234382489
  
--- Diff: 
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ---
@@ -565,7 +567,11 @@ private[spark] class AppStatusListener(
   if (metricsDelta != null) {
 esummary.metrics = LiveEntityHelpers.addMetrics(esummary.metrics, 
metricsDelta)
   }
-  conditionalLiveUpdate(esummary, now, removeStage)
+
+  val isLastTask = 
(stage.activeTasksPerExecutor(event.taskInfo.executorId) == 0) &&
+((stage.status == v1.StageStatus.COMPLETE) || (stage.status == 
v1.StageStatus.FAILED))
--- End diff --

Not sure why this extra condition is needed?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234382208
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
+var execId = (task.index % 2 + 1).toString
+tasks(task.index).markFinished(TaskState.FAILED, time)
+listener.onTaskEnd(
+  SparkListenerTaskEnd(stage.stageId, stage.attemptId, "taskType",
+ExecutorLostFailure(execId, true, Some("Lost executor")), 
tasks(task.index), null))
+}
+
+stage.failureReason = Some("Failed")
+listener.onStageCompleted(SparkListenerStageCompleted(stage))
+time += 1
+listener.onJobEnd(SparkListenerJobEnd(1, time, JobFailed(new 
RuntimeException("Bad Executor"
+
+tasks.filter(_.index >= 2).foreach { task =>
+time += 1
--- End diff --

Same here.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234382183
  
--- Diff: 
core/src/test/scala/org/apache/spark/status/AppStatusListenerSuite.scala ---
@@ -1275,6 +1275,49 @@ class AppStatusListenerSuite extends SparkFunSuite 
with BeforeAndAfter {
 assert(allJobs.head.numFailedStages == 1)
   }
 
+  test("SPARK-25451: total tasks in the executor summary should match 
total stage tasks") {
+val testConf = conf.clone()
+  .set("spark.ui.liveUpdate.period", s"${Int.MaxValue}s")
+
+val listener = new AppStatusListener(store, testConf, true)
+
+val stage = new StageInfo(1, 0, "stage", 4, Nil, Nil, "details")
+listener.onJobStart(SparkListenerJobStart(1, time, Seq(stage), null))
+listener.onStageSubmitted(SparkListenerStageSubmitted(stage, new 
Properties()))
+
+val tasks = createTasks(4, Array("1", "2"))
+tasks.foreach { task =>
+  listener.onTaskStart(SparkListenerTaskStart(stage.stageId, 
stage.attemptNumber, task))
+}
+
+tasks.filter(_.index < 2).foreach { task =>
+time += 1
--- End diff --

Whole block is indented too far.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234367532
  
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -376,6 +376,8 @@ private class LiveStage extends LiveEntity {
 
   val executorSummaries = new HashMap[String, LiveExecutorStageSummary]()
 
+  val activeTaskPerExecutor = new HashMap[String, 
Int]().withDefaultValue(0)
--- End diff --

Hi @vanzin I have added a UT. Kindly review


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234340680
  
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -376,6 +376,8 @@ private class LiveStage extends LiveEntity {
 
   val executorSummaries = new HashMap[String, LiveExecutorStageSummary]()
 
+  val activeTaskPerExecutor = new HashMap[String, 
Int]().withDefaultValue(0)
--- End diff --

I will add one UT


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234337780
  
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -376,6 +376,8 @@ private class LiveStage extends LiveEntity {
 
   val executorSummaries = new HashMap[String, LiveExecutorStageSummary]()
 
+  val activeTaskPerExecutor = new HashMap[String, 
Int]().withDefaultValue(0)
--- End diff --

@vanzin I tried to add one UT in the AppStatusListenerSuite, but because of 
"spark.ui.liveUpdate.period"=0 in the test conf, everything is written to the 
store without wait.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-16 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234311257
  
--- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala ---
@@ -376,6 +376,8 @@ private class LiveStage extends LiveEntity {
 
   val executorSummaries = new HashMap[String, LiveExecutorStageSummary]()
 
+  val activeTaskPerExecutor = new HashMap[String, 
Int]().withDefaultValue(0)
--- End diff --

`activeTasksPerExecutor`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-15 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234072070
  
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -63,6 +63,7 @@ case class ApplicationAttemptInfo private[spark](
 
 class ExecutorStageSummary private[spark](
 val taskTime : Long,
+val activeTasks: Int,
--- End diff --

Hi @vanzin , I have modified based your comment. Kindly review


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-15 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234034325
  
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -63,6 +63,7 @@ case class ApplicationAttemptInfo private[spark](
 
 class ExecutorStageSummary private[spark](
 val taskTime : Long,
+val activeTasks: Int,
--- End diff --

Okay. I will try without exposing in the public API


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-15 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234032990
  
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -63,6 +63,7 @@ case class ApplicationAttemptInfo private[spark](
 
 class ExecutorStageSummary private[spark](
 val taskTime : Long,
+val activeTasks: Int,
--- End diff --

You didn't answer my question.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-15 Thread shahidki31 
Github user shahidki31 commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234032510
  
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -63,6 +63,7 @@ case class ApplicationAttemptInfo private[spark](
 
 class ExecutorStageSummary private[spark](
 val taskTime : Long,
+val activeTasks: Int,
--- End diff --

Thank you @vanzin for the review.
Actually my objective is to get the last task of the particular executorId 
of the stage. If corresponding activeTasks == 0, then force update in the 
kvstore.

In stages, jobs, exec has "activeTasks" and using the parameter,  it force 
update on the last task.


https://github.com/apache/spark/blob/9a5fda60e532dc7203d21d5fbe385cd561906ccb/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala#L631


https://github.com/apache/spark/blob/9a5fda60e532dc7203d21d5fbe385cd561906ccb/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala#L563



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-15 Thread vanzin 
Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/23038#discussion_r234028545
  
--- Diff: core/src/main/scala/org/apache/spark/status/api/v1/api.scala ---
@@ -63,6 +63,7 @@ case class ApplicationAttemptInfo private[spark](
 
 class ExecutorStageSummary private[spark](
 val taskTime : Long,
+val activeTasks: Int,
--- End diff --

You don't need to expose this in the public API to fix the bug, do you?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #23038: [SPARK-25451][CORE][WEBUI]Aggregated metrics tabl...

2018-11-14 Thread shahidki31 
GitHub user shahidki31 opened a pull request:

https://github.com/apache/spark/pull/23038

[SPARK-25451][CORE][WEBUI]Aggregated metrics table doesn't show the right 
number of the total tasks

## What changes were proposed in this pull request?
Total tasks in the aggregated table and the tasks table are not matching 
some times in the WEBUI.
We need to update the executor summary of particular executor, when ever 
last task of that executor has reached. Currently it update based on last task, 
of all the executor. So, some particular executor task might miss, because for 
live application we update after some period.

## How was this patch tested?
Tests to reproduce:
```
bin/spark-shell --master yarn --conf spark.executor.instances=3
sc.parallelize(1 to 1, 10).map{ x => throw new RuntimeException("Bad 
executor")}.collect() 
```
Before patch:
![screenshot from 2018-11-15 
02-24-05](https://user-images.githubusercontent.com/23054875/48511776-b0d36480-e87d-11e8-89a8-ab97216e2c21.png)

After patch:
![screenshot from 2018-11-15 
02-32-38](https://user-images.githubusercontent.com/23054875/48512141-c39a6900-e87e-11e8-8535-903e1d11d13e.png)




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shahidki31/spark SPARK-25451

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/23038.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #23038


commit ed9895827b69e5f47e4fbe93de479663c47a4a37
Author: Shahid 
Date:   2018-11-13T22:24:04Z

Stages page doesn't show the right number of the total tasks




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org