Repository: spark
Updated Branches:
  refs/heads/master 534a063f7 -> 663f30d14

[SPARK-13041][MESOS] Adds sandbox uri to spark dispatcher ui

## What changes were proposed in this pull request?
Adds a sandbox link per driver in the dispatcher ui with minimal changes after 
a bug was fixed here:
The sandbox uri has the following format:
http://<proxy_uri>/#/slaves/\<agent-id\>/ frameworks/ 

For dc/os the proxy uri is <dc/os uri>/mesos. For the dc/os deployment scenario 
and to make things easier I introduced a new config property named 
`spark.mesos.proxy.baseURL` which should be passed to the dispatcher when 
launched using --conf. If no such configuration is detected then no sandbox uri 
is depicted, and there is an empty column with a header (this can be changed so 
nothing is shown).

Within dc/os the base url must be a property for the dispatcher that we should 
add in the future here:
It is not easy to detect in different environments what is that uri so user 
should pass it.

## How was this patch tested?
Tested with the mesos test suite here:
Attached image shows the ui modification where the sandbox header is added.

Tested the uri redirection the way it was suggested here:

Built mesos 1.4 from the master branch and started the mesos dispatcher with 
the command:

`./sbin/ --conf 
spark.mesos.proxy.baseURL=http://localhost:5050 -m mesos://`

Run a spark example:

`./bin/spark-submit   --class org.apache.spark.examples.SparkPi   --master 
mesos://   --deploy-mode cluster   --executor-memory 2G   
--total-executor-cores 2     http://<path>/spark-examples_2.11-2.1.1.jar  10`

Sandbox uri is shown at the bottom of the page:


Redirection works as expected:

Author: Stavros Kontopoulos <>

Closes #18528 from skonto/adds_the_sandbox_uri.


Branch: refs/heads/master
Commit: 663f30d14a0c9219e07697af1ab56e11a714d9a6
Parents: 534a063
Author: Stavros Kontopoulos <>
Authored: Mon Aug 7 10:32:19 2017 +0100
Committer: Sean Owen <>
Committed: Mon Aug 7 10:32:19 2017 +0100

 docs/                              |  2 ++
 .../spark/deploy/mesos/ui/MesosClusterPage.scala      | 14 +++++++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/docs/ b/docs/
index cf257c0..ae38550 100644
--- a/docs/
+++ b/docs/
@@ -153,6 +153,8 @@ can find the results of the driver from the Mesos Web UI.
 To use cluster mode, you must start the `MesosClusterDispatcher` in your 
cluster via the `sbin/` script,
 passing in the Mesos master URL (e.g: mesos://host:5050). This starts the 
`MesosClusterDispatcher` as a daemon running on the host.
+By setting the Mesos proxy config property (requires mesos version >= 1.4), 
`--conf spark.mesos.proxy.baseURL=http://localhost:5050` when launching the 
dispacther, the mesos sandbox URI for each driver is added to the mesos 
dispatcher UI.
 If you like to run the `MesosClusterDispatcher` with Marathon, you need to run 
the `MesosClusterDispatcher` in the foreground (i.e: `bin/spark-class 
org.apache.spark.deploy.mesos.MesosClusterDispatcher`). Note that the 
`MesosClusterDispatcher` not yet supports multiple instances for HA.
 The `MesosClusterDispatcher` also supports writing recovery state into 
Zookeeper. This will allow the `MesosClusterDispatcher` to be able to recover 
all submitted and running containers on relaunch.   In order to enable this 
recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spark-env by configuring 
`spark.deploy.recoveryMode` and related spark.deploy.zookeeper.* configurations.
diff --git 
index c9107c3..219b52e 100644
@@ -37,10 +37,11 @@ private[mesos] class MesosClusterPage(parent: 
MesosClusterUI) extends WebUIPage(
     val driverHeader = Seq("Driver ID")
     val historyHeader = => 
     val submissionHeader = Seq("Submit Date", "Main Class", "Driver Resources")
+    val sandboxHeader = Seq("Sandbox")
     val queuedHeaders = driverHeader ++ submissionHeader
     val driverHeaders = driverHeader ++ historyHeader ++ submissionHeader ++
-      Seq("Start Date", "Mesos Slave ID", "State")
+      Seq("Start Date", "Mesos Slave ID", "State") ++ sandboxHeader
     val retryHeaders = Seq("Driver ID", "Submit Date", "Description") ++
       Seq("Last Failed Status", "Next Retry Time", "Attempt Count")
     val queuedTable = UIUtils.listingTable(queuedHeaders, queuedRow, 
@@ -76,6 +77,16 @@ private[mesos] class MesosClusterPage(parent: 
MesosClusterUI) extends WebUIPage(
   private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
     val id = state.driverDescription.submissionId
+    val proxy = parent.conf.getOption("spark.mesos.proxy.baseURL")
+    val sandboxCol = if (proxy.isDefined) {
+      val clusterSchedulerId = parent.scheduler.getSchedulerState().frameworkId
+      val sandBoxUri = 
s"${proxy.get}/#/agents/${state.slaveId.getValue}/frameworks/" +
+        s"${clusterSchedulerId}/executors/${id}/browse"
+      <a href={sandBoxUri}>Sandbox</a>
+    } else {
+      " "
+    }
     val historyCol = if (historyServerURL.isDefined) {
@@ -94,6 +105,7 @@ private[mesos] class MesosClusterPage(parent: 
MesosClusterUI) extends WebUIPage(
+      <td>{sandboxCol}</td>

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to