GitHub user ericl opened a pull request:
https://github.com/apache/spark/pull/15705
[SPARK-18183] [SPARK-18184] Fix INSERT [INTO|OVERWRITE] TABLE ... PARTITION
for Datasource tables
## What changes were proposed in this pull request?
There are a couple issues with the current 2.1 behavior when inserting into
Datasource tables with partitions managed by Hive.
(1) OVERWRITE TABLE ... PARTITION will actually overwrite the entire table
instead of just the specified partition.
(2) INSERT|OVERWRITE does not work with partitions that have custom
locations.
This PR fixes both of these issues for Datasource tables managed by Hive.
The behavior for legacy tables or when `manageFilesourcePartitions = false` is
unchanged.
## How was this patch tested?
Unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ericl/spark sc-4942
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15705.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15705
----
commit dee4b9e107cc84f67640523fa277b31947661042
Author: Eric Liang <[email protected]>
Date: 2016-10-27T21:44:04Z
Thu Oct 27 14:44:04 PDT 2016
commit d99afacfc80e731d17af4ae25c548e45d948cd11
Author: Eric Liang <[email protected]>
Date: 2016-10-28T21:20:28Z
Fri Oct 28 14:20:28 PDT 2016
commit 137aeb295bc02e26755d7c18c89bd0117371676e
Author: Eric Liang <[email protected]>
Date: 2016-10-31T19:16:44Z
Merge branch 'master' into sc-4942
commit 8da8b0f7f70291cd86bd0a103758c5f88f714a27
Author: Eric Liang <[email protected]>
Date: 2016-10-31T21:29:58Z
wip
commit d5a7bd3931dc6f3f1673d9edff0f0aa77ad9d138
Author: Eric Liang <[email protected]>
Date: 2016-10-31T23:14:56Z
implement custom locations
commit a833a19d20f6b891cf42cf4a0f544c72abb982cd
Author: Eric Liang <[email protected]>
Date: 2016-10-31T23:32:16Z
Mon Oct 31 16:32:15 PDT 2016
commit 587b85e83f569ad8acae3de82c695b0fad4041db
Author: Eric Liang <[email protected]>
Date: 2016-10-31T23:34:47Z
Mon Oct 31 16:34:47 PDT 2016
commit 669e6cce48bfe903f473f14f20e1857ababc27be
Author: Eric Liang <[email protected]>
Date: 2016-10-31T23:41:40Z
Mon Oct 31 16:41:40 PDT 2016
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
