Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/6924#discussion_r32914556
--- Diff: docs/running-on-yarn.md ---
@@ -7,6 +7,53 @@ Support for running on [YARN (Hadoop
NextGen)](http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html)
was added to Spark in version 0.6.0, and improved in subsequent releases.
+# Launching Spark on YARN
+
+Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory
which contains the (client side) configuration files for the Hadoop cluster.
+These configs are used to write to HDFS and connect to the YARN
ResourceManager. The
+configuration contained in this directory will be distributed to the YARN
cluster so that all
+containers used by the application use the same configuration. If the
configuration references
+Java system properties or environment variables not managed by YARN, they
should also be set in the
+Spark application's configuration (driver, executors, and the AM when
running in client mode).
+
+There are two deploy modes that can be used to launch Spark applications
on YARN. In yarn-cluster mode, the Spark driver runs inside an application
master process which is managed by YARN on the cluster, and the client can go
away after initiating the application. In yarn-client mode, the driver runs in
the client process, and the application master is only used for requesting
resources from YARN.
+(Default: `--deploy-mode client`)
--- End diff --
`--master yarn-cluster` should be the same as `--master yarn` and
`--deploy-mode cluster`. I thought the latter was less-used, and none of the
examples show it on this page. Therefore how about removing this reference to
`--deploy-mode` here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
