[
https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudhakar Thota updated SPARK-9600:
----------------------------------
Attachment: SPARK-9600-fl1.txt
my screen content
> DataFrameWriter.saveAsTable always writes data to "/user/hive/warehouse"
> ------------------------------------------------------------------------
>
> Key: SPARK-9600
> URL: https://issues.apache.org/jira/browse/SPARK-9600
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1, 1.5.0
> Reporter: Cheng Lian
> Assignee: Sudhakar Thota
> Priority: Critical
> Attachments: SPARK-9600-fl1.txt
>
>
> Having a {{hive-site.xml}} with a non-default
> {{hive.metastore.warehouse.dir}} value, Spark SQL still writes to the default
> warehouse location {{/user/hive/warehouse}} when saving data source tables
> using {{DataFrameWriter.saveAsTable()}}:
> {noformat}
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/metastore_hive13_hadoop2</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>password</value>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>hdfs://localhost:9000/user/hive/warehouse_hive13</value>
> </property>
> </configuration>
> {noformat}
> Spark shell snippet to reproduce:
> {noformat}
> sqlContext.range(10).write.saveAsTable("xxx")
> {noformat}
> Running {{DESC EXTENDED xxx}} in Hive to check SerDe propertyies:
> {noformat}
> ...
> location:hdfs://localhost:9000/user/hive/warehouse_hive13/xxx
> ...
> parameters:{path=hdfs://localhost:9000/user/hive/warehouse/xxx,
> serialization.format=1})
> ...
> {noformat}
> We are probably using execution Hive configuration when calling
> {{HiveMetastoreCatalog.hiveDefaultTableFilePath()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
