Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8385#discussion_r37719449
--- Diff: docs/running-on-yarn.md ---
@@ -386,6 +405,6 @@ If you need a reference to the proper location to put
log files in the YARN so t
# Important notes
- Whether core requests are honored in scheduling decisions depends on
which scheduler is in use and how it is configured.
-- In `yarn-cluster` mode, the local directories used by the Spark
executors and the Spark driver will be the local directories configured for
YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies
`spark.local.dir`, it will be ignored. In `yarn-client` mode, the Spark
executors will use the local directories configured for YARN while the Spark
driver will use those defined in `spark.local.dir`. This is because the Spark
driver does not run on the YARN cluster in `yarn-client` mode, only the Spark
executors do.
+- In `--master yarn --deploy-mode cluster`, the local directories used by
the Spark executors and the Spark driver will be the local directories
configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the
user specifies `spark.local.dir`, it will be ignored. In `yarn-client` mode,
the Spark executors will use the local directories configured for YARN while
the Spark driver will use those defined in `spark.local.dir`. This is because
the Spark driver does not run on the YARN cluster in `yarn-client` mode, only
the Spark executors do.
--- End diff --
I would call the mode "YARN cluster" and "YARN client" mode consistently
when not actually referring to the flags used to specify them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
