GitHub user xwu0226 opened a pull request:
https://github.com/apache/spark/pull/12804
[SPARK-15025][SQL] fix duplicate of PATH key in datasource table options
## What changes were proposed in this pull request?
The issue is that when the user provides the path option with uppercase
"PATH" key, `options` contains `PATH` key and will get into the non-external
case in the following code in `createDataSourceTables.scala`, where a new key
"path" is created with a default path.
```
val optionsWithPath =
if (!options.contains("path")) {
isExternal = false
options + ("path" ->
sessionState.catalog.defaultTablePath(tableIdent))
} else {
options
}
```
So before creating hive table, serdeInfo.parameters will contain both
"PATH" and "path" keys and different directories. and Hive table's dataLocation
contains the value of "path".
The fix in this PR is to convert `options` in the code above to
`CaseInsensitiveMap` before checking for containing "path" key.
## How was this patch tested?
A testcase is added
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xwu0226/spark SPARK-15025
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12804.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12804
----
commit a6e24177fda7650522ef7cc1aabb8676806925a8
Author: xin Wu <[email protected]>
Date: 2016-04-30T03:50:56Z
SPARK-15025 -- make caseinsensitive map for the options before checking for
'path' key when createDataSourceTable
commit 633c8504f94a62bfa5218f7eafd47304718cef6a
Author: xin Wu <[email protected]>
Date: 2016-04-30T05:45:44Z
rebase and fix testcase
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
