Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/18668#discussion_r131093892
--- Diff: docs/configuration.md ---
@@ -2335,5 +2335,61 @@ The location of these configuration files varies
across Hadoop versions, but
a common location is inside of `/etc/hadoop/conf`. Some tools create
configurations on-the-fly, but offer a mechanisms to download copies of
them.
-To make these files visible to Spark, set `HADOOP_CONF_DIR` in
`$SPARK_HOME/spark-env.sh`
+To make these files visible to Spark, set `HADOOP_CONF_DIR` in
`$SPARK_HOME/conf/spark-env.sh`
to a location containing the configuration files.
+
+# Custom Hadoop/Hive Configuration
+
+If your Spark applications interacting with Hadoop, Hive, or both, there
are probably Hadoop/Hive
+configuration files in Spark's class path.
+
+Multiple running applications might require different Hadoop/Hive client
side configurations.
+You can copy and modify `hdfs-site.xml`, `core-site.xml`, `yarn-site.xml`,
`hive-site.xml` in
+Spark's class path for each application, but it is not very convenient and
these
+files are best to be shared with common properties to avoid hard-coding
certain configurations.
--- End diff --
"best shared"
You can'd do that anyway on a production Spark on Yarn cluster as if you
did., lots of other things would break. How about
```
In a Spark cluster running on YARN, these configuration files are set
cluster-wide, and cannot safely be changed by the application.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
