[
https://issues.apache.org/jira/browse/SPARK-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengzhe Yao resolved SPARK-1048.
---------------------------------
Resolution: Duplicate
Patrick submit a patch which introduces spark-defaults.conf to set system
properties. This seems to achieve our original purpose: allow user to set
configuration values in a well-known config file and Spark will automatically
pick it up.
commit fb98488fc8e68cc84f6e0750fd4e9e29029879d2
Author: Patrick Wendell <[email protected]>
Date: Mon Apr 21 10:26:33 2014 -0700
Clean up and simplify Spark configuration
Over time as we've added more deployment modes, this have gotten a bit
unwieldy with user-facing configuration options in Spark. Going forward we'll
advise all users to run `spark-submit` to launch applications. This is a WIP
patch but it makes the following improvements:
1. Improved `spark-env.sh.template` which was missing a lot of things users
now set in that file.
2. Removes the shipping of SPARK_CLASSPATH, SPARK_JAVA_OPTS, and
SPARK_LIBRARY_PATH to the executors on the cluster. This was an ugly hack.
Instead it introduces config variables spark.executor.extraJavaOpts,
spark.executor.extraLibraryPath, and spark.executor.extraClassPath.
3. Adds ability to set these same variables for the driver using
`spark-submit`.
4. Allows you to load system properties from a `spark-defaults.conf` file
when running `spark-submit`. This will allow setting both SparkConf options and
other system properties utilized by `spark-submit`.
5. Made `SPARK_LOCAL_IP` an environment variable rather than a SparkConf
property. This is more consistent with it being set on each node.
Author: Patrick Wendell <[email protected]>
> Create spark-site.xml or spark-site.yaml for configuration
> ----------------------------------------------------------
>
> Key: SPARK-1048
> URL: https://issues.apache.org/jira/browse/SPARK-1048
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 0.8.1, 0.9.0, 1.0.0
> Reporter: Shengzhe Yao
> Priority: Critical
>
> Currently, spark doesn't provide the ability to pass configuration as a file,
> people have to explicitly specify them on the command line. This becomes a
> bigger issue when deploying spark on a cluster and people want to specify
> parameters other than default value.
> As of now, spark is being a top apache project and we need to pay more
> attention to the configuration. Most apache projects provide a xml
> configuration file (like hdfs, hadoop, hbase, etc.) and probably we want to
> do the same thing for spark. The advantages are obvious, it helps developers
> to specify their own spark configuration for the cluster and add/remove
> configuration parameters will be much easier via file than via system
> property.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
