git commit: [SPARK-611] Display executor thread dumps in web UI
Repository: spark
Updated Branches:
refs/heads/branch-1.2 e7f735637 - 866c7bbe5
[SPARK-611] Display executor thread dumps in web UI
This patch allows executor thread dumps to be collected on-demand and viewed in
the Spark web UI.
The thread dumps are collected using Thread.getAllStackTraces(). To allow
remote thread dumps to be triggered from the web UI, I added a new
`ExecutorActor` that runs inside of the Executor actor system and responds to
RPCs from the driver. The driver's mechanism for obtaining a reference to this
actor is a little bit hacky: it uses the block manager master actor to
determine the host/port of the executor actor systems in order to construct
ActorRefs to ExecutorActor. Unfortunately, I couldn't find a much cleaner way
to do this without a big refactoring of the executor - driver communication.
Screenshots:
Author: Josh Rosen joshro...@databricks.com
Closes #2944 from JoshRosen/jstack-in-web-ui and squashes the following commits:
3c21a5d [Josh Rosen] Address review comments:
880f7f7 [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
f719266 [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
19707b0 [Josh Rosen] Add one comment.
127a130 [Josh Rosen] Update to use SparkContext.DRIVER_IDENTIFIER
b8e69aa [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
3dfc2d4 [Josh Rosen] Add missing file.
bc1e675 [Josh Rosen] Undo some leftover changes from the earlier approach.
f4ac1c1 [Josh Rosen] Switch to on-demand collection of thread dumps
dfec08b [Josh Rosen] Add option to disable thread dumps in UI.
4c87d7f [Josh Rosen] Use separate RPC for sending thread dumps.
2b8bdf3 [Josh Rosen] Enable thread dumps from the driver when running in
non-local mode.
cc3e6b3 [Josh Rosen] Fix test code in DAGSchedulerSuite.
87b8b65 [Josh Rosen] Add new listener event for thread dumps.
8c10216 [Josh Rosen] Add missing file.
0f198ac [Josh Rosen] [SPARK-611] Display executor thread dumps in web UI
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/866c7bbe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/866c7bbe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/866c7bbe
Branch: refs/heads/branch-1.2
Commit: 866c7bbe56f9c7fd96d3f4afe8a76405dc877a6e
Parents: e7f7356
Author: Josh Rosen joshro...@databricks.com
Authored: Mon Nov 3 18:18:47 2014 -0800
Committer: Patrick Wendell pwend...@gmail.com
Committed: Wed Nov 5 14:44:24 2014 -0800
--
.../scala/org/apache/spark/SparkContext.scala | 29 +++-
.../executor/CoarseGrainedExecutorBackend.scala | 3 +-
.../org/apache/spark/executor/Executor.scala| 7 +-
.../apache/spark/executor/ExecutorActor.scala | 41 +++
.../spark/storage/BlockManagerMaster.scala | 4 ++
.../spark/storage/BlockManagerMasterActor.scala | 18 +
.../spark/storage/BlockManagerMessages.scala| 2 +
.../spark/ui/exec/ExecutorThreadDumpPage.scala | 73
.../apache/spark/ui/exec/ExecutorsPage.scala| 15 +++-
.../org/apache/spark/ui/exec/ExecutorsTab.scala | 8 ++-
.../scala/org/apache/spark/util/AkkaUtils.scala | 14
.../apache/spark/util/ThreadStackTrace.scala| 27
.../scala/org/apache/spark/util/Utils.scala | 13
13 files changed, 247 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/866c7bbe/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index d65027d..3cdaa6a 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -21,9 +21,8 @@ import scala.language.implicitConversions
import java.io._
import java.net.URI
-import java.util.Arrays
+import java.util.{Arrays, Properties, UUID}
import java.util.concurrent.atomic.AtomicInteger
-import java.util.{Properties, UUID}
import java.util.UUID.randomUUID
import scala.collection.{Map, Set}
import scala.collection.generic.Growable
@@ -41,6 +40,7 @@ import akka.actor.Props
import org.apache.spark.annotation.{DeveloperApi, Experimental}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.deploy.{LocalSparkCluster, SparkHadoopUtil}
+import org.apache.spark.executor.TriggerThreadDump
import
git commit: [SPARK-611] Display executor thread dumps in web UI
Repository: spark
Updated Branches:
refs/heads/master 97a466eca - 4f035dd2c
[SPARK-611] Display executor thread dumps in web UI
This patch allows executor thread dumps to be collected on-demand and viewed in
the Spark web UI.
The thread dumps are collected using Thread.getAllStackTraces(). To allow
remote thread dumps to be triggered from the web UI, I added a new
`ExecutorActor` that runs inside of the Executor actor system and responds to
RPCs from the driver. The driver's mechanism for obtaining a reference to this
actor is a little bit hacky: it uses the block manager master actor to
determine the host/port of the executor actor systems in order to construct
ActorRefs to ExecutorActor. Unfortunately, I couldn't find a much cleaner way
to do this without a big refactoring of the executor - driver communication.
Screenshots:
Author: Josh Rosen joshro...@databricks.com
Closes #2944 from JoshRosen/jstack-in-web-ui and squashes the following commits:
3c21a5d [Josh Rosen] Address review comments:
880f7f7 [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
f719266 [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
19707b0 [Josh Rosen] Add one comment.
127a130 [Josh Rosen] Update to use SparkContext.DRIVER_IDENTIFIER
b8e69aa [Josh Rosen] Merge remote-tracking branch 'origin/master' into
jstack-in-web-ui
3dfc2d4 [Josh Rosen] Add missing file.
bc1e675 [Josh Rosen] Undo some leftover changes from the earlier approach.
f4ac1c1 [Josh Rosen] Switch to on-demand collection of thread dumps
dfec08b [Josh Rosen] Add option to disable thread dumps in UI.
4c87d7f [Josh Rosen] Use separate RPC for sending thread dumps.
2b8bdf3 [Josh Rosen] Enable thread dumps from the driver when running in
non-local mode.
cc3e6b3 [Josh Rosen] Fix test code in DAGSchedulerSuite.
87b8b65 [Josh Rosen] Add new listener event for thread dumps.
8c10216 [Josh Rosen] Add missing file.
0f198ac [Josh Rosen] [SPARK-611] Display executor thread dumps in web UI
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4f035dd2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4f035dd2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4f035dd2
Branch: refs/heads/master
Commit: 4f035dd2cd6f1ec9059811f3495f3e0a8ec5fb84
Parents: 97a466e
Author: Josh Rosen joshro...@databricks.com
Authored: Mon Nov 3 18:18:47 2014 -0800
Committer: Andrew Or and...@databricks.com
Committed: Mon Nov 3 18:18:47 2014 -0800
--
.../scala/org/apache/spark/SparkContext.scala | 29 +++-
.../executor/CoarseGrainedExecutorBackend.scala | 3 +-
.../org/apache/spark/executor/Executor.scala| 7 +-
.../apache/spark/executor/ExecutorActor.scala | 41 +++
.../spark/storage/BlockManagerMaster.scala | 4 ++
.../spark/storage/BlockManagerMasterActor.scala | 18 +
.../spark/storage/BlockManagerMessages.scala| 2 +
.../spark/ui/exec/ExecutorThreadDumpPage.scala | 73
.../apache/spark/ui/exec/ExecutorsPage.scala| 15 +++-
.../org/apache/spark/ui/exec/ExecutorsTab.scala | 8 ++-
.../scala/org/apache/spark/util/AkkaUtils.scala | 14
.../apache/spark/util/ThreadStackTrace.scala| 27
.../scala/org/apache/spark/util/Utils.scala | 13
13 files changed, 247 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/4f035dd2/core/src/main/scala/org/apache/spark/SparkContext.scala
--
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 8b4db78..40444c2 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -21,9 +21,8 @@ import scala.language.implicitConversions
import java.io._
import java.net.URI
-import java.util.Arrays
+import java.util.{Arrays, Properties, UUID}
import java.util.concurrent.atomic.AtomicInteger
-import java.util.{Properties, UUID}
import java.util.UUID.randomUUID
import scala.collection.{Map, Set}
import scala.collection.generic.Growable
@@ -41,6 +40,7 @@ import akka.actor.Props
import org.apache.spark.annotation.{DeveloperApi, Experimental}
import org.apache.spark.broadcast.Broadcast
import org.apache.spark.deploy.{LocalSparkCluster, SparkHadoopUtil}
+import org.apache.spark.executor.TriggerThreadDump
import
