This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c040593 [SPARK-36377][DOCS] Re-document "Options read in YARN
client/cluster mode" section in spark-env.sh.template
c040593 is described below
commit c040593fb432f75f7e22e3d32a6979142a125375
Author: Yuto Akutsu <[email protected]>
AuthorDate: Tue Aug 10 11:05:39 2021 +0900
[SPARK-36377][DOCS] Re-document "Options read in YARN client/cluster mode"
section in spark-env.sh.template
### What changes were proposed in this pull request?
Edit spark-env.sh.template for better documentation.
### Why are the changes needed?
Some options (e.g. SPARK_CONF_DIR, SPARK_EXECUTOR_CORES, etc.) are read by
any mode but are in the "Options read in YARN client/cluster mode" that might
confuse users, so we should separate YARN only options from others.
### Does this PR introduce _any_ user-facing change?
Yes, docs changed.
### How was this patch tested?
Just a manual checking.
<img width="336" alt="options"
src="https://user-images.githubusercontent.com/87687356/127804872-02a27337-f304-47df-affa-a8c4e9e98fe5.png";>
Closes #33604 from yutoacts/SPARK-36377.
Authored-by: Yuto Akutsu <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
conf/spark-env.sh.template | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/conf/spark-env.sh.template b/conf/spark-env.sh.template
index 3c003f4..a2f1380 100755
--- a/conf/spark-env.sh.template
+++ b/conf/spark-env.sh.template
@@ -32,14 +32,18 @@
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and
RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos
-# Options read in YARN client/cluster mode
+# Options read in any mode
# - SPARK_CONF_DIR, Alternate conf dir. (Default: ${SPARK_HOME}/conf)
-# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
-# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you
use YARN
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G)
+# Options read in any cluster manager using HDFS
+# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
+
+# Options read in YARN client/cluster mode
+# - YARN_CONF_DIR, to point Spark towards YARN configuration files when you
use YARN
+
# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_HOST, to bind the master to a different IP address or hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for
the master
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/87687356/127804872-02a27337-f304-47df-affa-a8c4e9e98fe5.png"](https://user-images.githubusercontent.com/87687356/127804872-02a27337-f304-47df-affa-a8c4e9e98fe5.png"){kind=link}