GitHub user tnachen opened a pull request:
https://github.com/apache/spark/pull/5144
[SPARK-][MESOS] Add cluster mode support for Mesos
This patch adds the support for cluster mode to run on Mesos.
It introduces a new Mesos framework dedicated to launch new apps/drivers,
and can be called with the spark-submit script and specifying --master flag to
the cluster mode REST interface instead of Mesos master.
Example:
./bin/spark-submit --deploy-mode cluster --class
org.apache.spark.examples.SparkPi --master mesos://10.0.0.206:8077
--executor-memory 1G --total-executor-cores 100
examples/target/spark-examples_2.10-1.3.0-SNAPSHOT.jar 30
Features of the cluster mode in this PR:
- Supports supervise mode where scheduler will keep trying to reschedule
exited job.
- Adds a new UI for the cluster mode scheduler to see all the running jobs,
finished jobs, and supervise jobs waiting to be retried
- Supports state persistence to ZK, so when the cluster scheduler fails
over it can pick up all the queued and running jobs
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tnachen/spark mesos_cluster_mode
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5144.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5144
----
commit f7757a9382559cc051bfd8cf90a3d551c71652f5
Author: Timothy Chen <[email protected]>
Date: 2015-02-20T08:17:11Z
Add Mesos Cluster dispatcher
commit 1e24e4ee26818d9d3237e17fc60e05ed15744d71
Author: Timothy Chen <[email protected]>
Date: 2015-02-24T08:46:21Z
Rename StandaloneRestClient to RestClient and add sbin scripts
commit 29d20cca65a50cfc947eb6764ca4eda21d1cd534
Author: Timothy Chen <[email protected]>
Date: 2015-02-24T22:50:25Z
Kill drivers when shutdown
commit 32b584387490163b7d8d4a2c25e1c5f2d3445c4f
Author: Timothy Chen <[email protected]>
Date: 2015-02-26T08:20:16Z
Add Mesos Cluster UI to display driver results
commit 97e6fcf5dc126bfe56cd88f001c9a10edc85f0d2
Author: Timothy Chen <[email protected]>
Date: 2015-02-27T04:48:01Z
Change Driver page output and add logging
commit 3b873724cf1edb8bab0bc002aa19192b09b99088
Author: Timothy Chen <[email protected]>
Date: 2015-02-27T18:22:17Z
Support fetching remote uris in driver runner.
commit af6e2908bb2b147f9d46b1c597d8bc57ea730e39
Author: Timothy Chen <[email protected]>
Date: 2015-02-28T09:09:29Z
Specify user jar in command to be replaced with local.
commit 156dbca857c705e0b3804f1e86ac83c2f0223a8a
Author: Timothy Chen <[email protected]>
Date: 2015-03-10T00:11:01Z
WIP: Making a cluster mode a mesos framework.
commit 4483aa80f4b0093b1cf900cd9db9629fcdf088fb
Author: Luc Bourlier <[email protected]>
Date: 2015-03-10T23:49:37Z
Launch task through the mesos scheduler
commit ed48e1d7adcf0fccf378634f7ce93c8c7f07c50f
Author: Luc Bourlier <[email protected]>
Date: 2015-03-11T00:41:11Z
Adds a shutdown latch to keep the deamon running
commit 0d05bb5556ad30c7d64b735b7b1f78708082904c
Author: Timothy Chen <[email protected]>
Date: 2015-03-11T20:20:15Z
Add documentation
commit 5e1f091226eaf891dfd13627a181479c80efdac1
Author: Luc Bourlier <[email protected]>
Date: 2015-03-11T18:40:40Z
Supports more spark-submit parameters
commit 180d0726e9f02da8999fb96419e628860fb25476
Author: Timothy Chen <[email protected]>
Date: 2015-03-11T18:41:08Z
Fix Mesos dispatcher UI.
commit 468b89b96da195508ab3bec79c9e384de6b89798
Author: Timothy Chen <[email protected]>
Date: 2015-03-12T00:48:19Z
Support looking at SPARK_EXECUTOR_URI env variable in schedulers
commit 6952da8b493634871a789469e81496bcd1871fe5
Author: Luc Bourlier <[email protected]>
Date: 2015-03-11T23:05:04Z
Adds support to kill submissions
commit 0f011ac31b1dc05fdf879d242866c9764db88d4b
Author: Timothy Chen <[email protected]>
Date: 2015-03-12T08:01:05Z
Schedule multiple jobs
commit adba6b90a07626d3347d05a08567e397871e8033
Author: Timothy Chen <[email protected]>
Date: 2015-03-12T20:59:52Z
Bound the finished drivers in memory
commit 913ac4e9451bd2b2281b1857dccab9e46f55637e
Author: Timothy Chen <[email protected]>
Date: 2015-03-18T22:01:38Z
Handle cluster mode recovery and state persistence.
commit 2b5e23c2402c8fbee73c49f1780c3219da1188fa
Author: Timothy Chen <[email protected]>
Date: 2015-03-23T21:49:21Z
Add supervise support and persist retries.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
