Github user nssalian commented on a diff in the pull request:
https://github.com/apache/spark/pull/6924#discussion_r32897974
--- Diff: docs/running-on-yarn.md ---
@@ -7,6 +7,53 @@ Support for running on [YARN (Hadoop
NextGen)](http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html)
was added to Spark in version 0.6.0, and improved in subsequent releases.
+# Launching Spark on YARN
+
+Ensure that `HADOOP_CONF_DIR` or `YARN_CONF_DIR` points to the directory
which contains the (client side) configuration files for the Hadoop cluster.
+These configs are used to write to the dfs and connect to the YARN
ResourceManager. The
+configuration contained in this directory will be distributed to the YARN
cluster so that all
+containers used by the application use the same configuration. If the
configuration references
+Java system properties or environment variables not managed by YARN, they
should also be set in the
+Spark application's configuration (driver, executors, and the AM when
running in client mode).
+
+There are two deploy modes that can be used to launch Spark applications
on YARN. In yarn-cluster mode, the Spark driver runs inside an application
master process which is managed by YARN on the cluster, and the client can go
away after initiating the application. In yarn-client mode, the driver runs in
the client process, and the application master is only used for requesting
resources from YARN.
+(Default: --deploy-mode client)
+
+Unlike in Spark standalone and Mesos mode, in which the master's address
is specified in the "master" parameter, in YARN mode the ResourceManager's
address is picked up from the Hadoop configuration. Thus, the master parameter
is yarn.
--- End diff --
So, in spark-submit the options:
--master MASTER_URL spark://host:port, mesos://host:port, yarn, or
local.
So just yarn or specifically client and cluster. I would suggest keeping it
as yarn since --deploy-mode covers the client or cluster part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
