[
https://issues.apache.org/jira/browse/SPARK-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662125#comment-14662125
]
Sudhakar Thota commented on SPARK-9600:
---------------------------------------
Hi,
I tested this on 1.4.1 and 1.5.0 both seems to be working fine. Here are the
steps I followed.
Following steps create table under default directory /user/hive/warehouse
1. val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
2. df.write.saveAsTable("chenglianwarehouse")
3. Check if the table is created in default directory.
Sudhakars-MacBook-Pro-2:warehouse sudhakarthota$ ls -l /user/hive/warehouse |
grep chenglianwarehouse
drwxr-xr-x 10 sudhakarthota wheel 340 Aug 7 08:56 chenglianwarehouse
Following steps create table under directory under a given directory:
/user/hive/warehouse13
1. mkdir /user/hive/warehouse13
2. Copy the hive-site.xml under $SPARK_HOME/conf directory and edit the file to
change the parameter hive.metastore.warehouse.dir=/user/hive/warehouse13
3. Ensure that change is in place
Sudhakars-MacBook-Pro-2:spark-1.4.1 sudhakarthota$ diff -w
sql/hive/src/test/resources/data/conf/hive-site.xml conf/hive-site.xml
90c90
< <value>${test.warehouse.dir}</value>
---
> <value>/user/hive/warehouse13</value>
4. val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
5. val df =
hiveContext.read.format("parquet").load("examples/src/main/resources/users.parquet")
6. df.write.saveAsTable("chenglianwarehouse13")
7. Check if the table is created in right directory:
Sudhakars-MacBook-Pro-2:warehouse13 sudhakarthota$ ls -l /user/hive/warehouse13
| grep chenglianwarehouse
drwxr-xr-x 10 sudhakarthota wheel 340 Aug 7 09:02 chenglianwarehouse13
Sudhakars-MacBook-Pro-2:warehouse13 sudhakarthota$
Verified the code, it is taking this parameter from class path and it seems
right.
Please execute the same steps and verify if the same works on your machine.
Please let me know if it fails to work. I will investigate further.
Thanks
Sudhakar Thota
> DataFrameWriter.saveAsTable always writes data to "/user/hive/warehouse"
> ------------------------------------------------------------------------
>
> Key: SPARK-9600
> URL: https://issues.apache.org/jira/browse/SPARK-9600
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1, 1.5.0
> Reporter: Cheng Lian
> Assignee: Sudhakar Thota
> Priority: Critical
>
> Having a {{hive-site.xml}} with a non-default
> {{hive.metastore.warehouse.dir}} value, Spark SQL still writes to the default
> warehouse location {{/user/hive/warehouse}} when saving data source tables
> using {{DataFrameWriter.saveAsTable()}}:
> {noformat}
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost/metastore_hive13_hadoop2</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>password</value>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>hdfs://localhost:9000/user/hive/warehouse_hive13</value>
> </property>
> </configuration>
> {noformat}
> Spark shell snippet to reproduce:
> {noformat}
> sqlContext.range(10).write.saveAsTable("xxx")
> {noformat}
> Running {{DESC EXTENDED xxx}} in Hive to check SerDe propertyies:
> {noformat}
> ...
> location:hdfs://localhost:9000/user/hive/warehouse_hive13/xxx
> ...
> parameters:{path=hdfs://localhost:9000/user/hive/warehouse/xxx,
> serialization.format=1})
> ...
> {noformat}
> We are probably using execution Hive configuration when calling
> {{HiveMetastoreCatalog.hiveDefaultTableFilePath()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
