Github user maasg commented on a diff in the pull request:
https://github.com/apache/spark/pull/756#discussion_r12578665
--- Diff: docs/running-on-mesos.md ---
@@ -3,19 +3,109 @@ layout: global
title: Running Spark on Mesos
---
-Spark can run on clusters managed by [Apache
Mesos](http://mesos.apache.org/). Follow the steps below to install Mesos and
Spark:
-
-1. Download and build Spark using the instructions [here](index.html).
**Note:** Don't forget to consider what version of HDFS you might want to use!
-2. Download, build, install, and start Mesos {{site.MESOS_VERSION}} on
your cluster. You can download the Mesos distribution from a
[mirror](http://www.apache.org/dyn/closer.cgi/mesos/{{site.MESOS_VERSION}}/).
See the Mesos [Getting Started](http://mesos.apache.org/gettingstarted) page
for more information. **Note:** If you want to run Mesos without installing it
into the default paths on your system (e.g., if you don't have administrative
privileges to install it), you should also pass the `--prefix` option to
`configure` to tell it where to install. For example, pass
`--prefix=/home/user/mesos`. By default the prefix is `/usr/local`.
-3. Create a Spark "distribution" using `make-distribution.sh`.
-4. Rename the `dist` directory created from `make-distribution.sh` to
`spark-{{site.SPARK_VERSION}}`.
-5. Create a `tar` archive: `tar czf spark-{{site.SPARK_VERSION}}.tar.gz
spark-{{site.SPARK_VERSION}}`
-6. Upload this archive to HDFS or another place accessible from Mesos via
`http://`, e.g., [Amazon Simple Storage Service](http://aws.amazon.com/s3):
`hadoop fs -put spark-{{site.SPARK_VERSION}}.tar.gz
/path/to/spark-{{site.SPARK_VERSION}}.tar.gz`
-7. Create a file called `spark-env.sh` in Spark's `conf` directory, by
copying `conf/spark-env.sh.template`, and add the following lines to it:
- * `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is
usually `<prefix>/lib/libmesos.so` (where the prefix is `/usr/local` by
default, see above). Also, on Mac OS X, the library is called `libmesos.dylib`
instead of `libmesos.so`.
- * `export SPARK_EXECUTOR_URI=<path to
spark-{{site.SPARK_VERSION}}.tar.gz uploaded above>`.
- * `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and
port (default: 5050) of your Mesos master (or `zk://...` if using Mesos with
ZooKeeper).
-8. To run a Spark application against the cluster, when you create your
`SparkContext`, pass the string `mesos://HOST:PORT` as the master URL. In
addition, you'll need to set the `spark.executor.uri` property. For example:
+# Why Mesos
--- End diff --
I miss a chapter "How Spark works on Mesos" that helps understanding the
need of the assembly in a reachable location by the slaves and how the spark
driver program becomes the 'scheduler' in Mesos terms and the tasks get
delivered to the executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
