GitHub user yaooqinn opened a pull request:
https://github.com/apache/spark/pull/19663
[SPARK-21888][Hive]add hadoop/hive/hdfs configuration files in
SPARK_CONF_DIR to distribute archive
## What changes were proposed in this pull request?
When I ran self contained sql apps, such as
```scala
import org.apache.spark.sql.SparkSession
object ShowHiveTables {
def main(args: Array[String]): Unit = {
val spark = SparkSession
.builder()
.appName("Show Hive Tables")
.enableHiveSupport()
.getOrCreate()
spark.sql("show tables").show()
spark.stop()
}
}
```
with **yarn cluster** mode and `hive-site.xml` correctly within
`$SPARH_HOME/conf`,they failed to connect the right hive metestore for not
seeing hive-site.xml in AM/Driver's classpath.
Although submitting them with `--files/--jars local/path/to/hive-site.xml`
or puting it to `$HADOOP_CONF_DIR/YARN_CONF_DIR` can make these apps works well
in cluster mode as client mode, according to the official doc, see @
http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
> Configuration of Hive is done by placing your hive-site.xml,
core-site.xml (for security configuration), and hdfs-site.xml (for HDFS
configuration) file in conf/.
We may respect these configuration files too or modify the doc for
hive-tables in cluster mode.
## How was this patch tested?
cc @cloud-fan @gatorsmile
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yaooqinn/spark SPARK-21888
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19663.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19663
----
commit 696dbd6e5b5af89752c3869264f70ceddb868baf
Author: Kent Yao <[email protected]>
Date: 2017-11-06T04:56:04Z
add hadoop/hive/hdfs configuration files in SPARK_CONF_DIR to distribute
archive
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
