GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/13270
[SPARK-15269][SQL] Removes unexpected empty table directories created while
creating external Spark SQL data sourcet tables.
This PR is an alternative to #13120 authored by @xwu0226.
## What changes were proposed in this pull request?
When creating an external Spark SQL data source table and persisting its
metadata to Hive metastore, we don't use the standard Hive `Table.dataLocation`
field because Hive only allows directory paths as data locations while Spark
SQL also allows file paths. However, if we don't set `Table.dataLocation`, Hive
always creates an unexpected empty table directory under database location, but
doesn't remove it while dropping the table (because the table is external).
This PR works around this issue by explicitly setting `Table.dataLocation`
and then manullay removing the created directory after creating the external
table.
Please refer to [this JIRA comment][1] for more details about why we chose
this approach as a workaround.
[1]:
https://issues.apache.org/jira/browse/SPARK-15269?focusedCommentId=15297408&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15297408
## How was this patch tested?
1. A new test case is added in `HiveQuerySuite` for this case
2. Updated `ShowCreateTableSuite` to use the same table name in all test
cases. (This is how I hit this issue at the first place.)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark spark-15269-unpleasant-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13270.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13270
----
commit f376332b3c6eb59ddc4a45e0843285374ceaaace
Author: Cheng Lian <[email protected]>
Date: 2016-05-24T00:30:37Z
Fixes SPARK-15269
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
