GitHub user barnardb opened a pull request:
https://github.com/apache/spark/pull/18209
[SPARK-20992][Scheduler] Add support for Nomad as a scheduler backend
## What changes were proposed in this pull request?
Adds support for [Nomad](https://github.com/hashicorp/nomad) as a scheduler
backend. Nomad is a cluster manager designed for both long lived services and
short lived batch processing workloads.
The integration supports client and cluster mode, dynamic allocation
(increasing only), has basic support for python and R applications, and works
with applications packaged either as JARs or as docker images.
Documentation is in
[docs/running-on-nomad.md](https://github.com/barnardb/spark/blob/nomad/docs/running-on-nomad.md).
This will be [presented at Spark Summit
2017](https://spark-summit.org/2017/events/homologous-apache-spark-clusters-using-nomad/).
A build of the pull request with Nomad support is at available
[here](https://www.dropbox.com/s/llcv388yl5hweje/spark-2.3.0-SNAPSHOT-bin-nomad.tgz?dl=0).
Feedback would be much appreciated.
## How was this patch tested?
This patch was tested with Integration and manual tests, and a load test
was performed to ensure it doesn't have worse performance than the YARN
integration.
The feature was developed and tested against Nomad 0.5.6 (current stable
version)
on Spark 2.1.0, rebased to 2.1.1 and retested, and finally rebased to
master and retested.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/barnardb/spark nomad
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18209.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18209
----
commit c762194188e64cccff8a9758885b45f9d395cced
Author: Ben Barnard <[email protected]>
Date: 2017-06-06T01:19:35Z
Add support for Nomad as a scheduler backend
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
