Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/299#discussion_r11220148
--- Diff: conf/spark-env.sh.template ---
@@ -1,19 +1,36 @@
#!/usr/bin/env bash
-# This file contains environment variables required to run Spark. Copy it
as
-# spark-env.sh and edit that to configure Spark for your site.
-#
-# The following variables can be set in this file:
+# This file is sourced when running various Spark classes.
+# Copy it as spark-env.sh and edit that to configure Spark for your site.
+
+# Options read when launching programs locally with
+# ./bin/spark-example or ./bin/spark-submit
+# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
+# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
+# - SPARK_CLASSPATH, default classpath entries to append
+
+# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
+# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
+# - SPARK_LOCAL_DIRS, shuffle directories to use on this node
# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos
-# - SPARK_JAVA_OPTS, to set node-specific JVM options for Spark. Note that
-# we recommend setting app-wide options in the application's driver
program.
-# Examples of node-specific options : -Dspark.local.dir, GC options
-# Examples of app-wide options : -Dspark.serializer
-#
-# If using the standalone deploy mode, you can also set variables for it
here:
+# - SPARK_CLASSPATH, default classpath entries to append
+
+# Options read in YARN client mode
+# - SPARK_YARN_APP_JAR, Path to your applicationâs JAR file (required)
+# - SPARK_WORKER_INSTANCES, Number of workers to start (Default: 2)
+# - SPARK_WORKER_CORES, Number of cores for the workers (Default: 1).
+# - SPARK_WORKER_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
+# - SPARK_MASTER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512
Mb)
+# - SPARK_YARN_APP_NAME, The name of your application (Default: Spark)
+# - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests
(Default: âdefaultâ)
+# - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed
with the job.
+# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be
distributed with the job.
+
+# Options for the daemons used in the standalone deploy mode:
# - SPARK_MASTER_IP, to bind the master to a different IP address or
hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports
+# - SPARK_MASTER_OPTS, to set config properties at the master (e.g "-Dx=y")
--- End diff --
SPARK_WORKER_OPTS is not listed here (though note that
SPARK_DAEMON_OPTS/MEMORY is the more general option for both)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
