GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/16517
[SPARK-18243][SQL] Implement InsertIntoHiveTable with FileCommitProtocol
and FileFormatWriter
## What changes were proposed in this pull request?
Inserting data into Hive tables has its own implementation that is distinct
from data sources: `InsertIntoHiveTable`, `SparkHiveWriterContainer` and
`SparkHiveDynamicPartitionWriterContainer`.
Note that one other major difference is that data source tables write
directly to the final destination without using some staging directory, and
then Spark itself adds the partitions/tables to the catalog. Hive tables
actually write to some staging directory, and then call Hive metastore's
loadPartition/loadTable function to load those data in. So we still need to
keep `InsertIntoHiveTable` to put this special logic. In the future, we should
think of writing to the hive table location directly, so that we don't need to
call `loadTable`/`loadPartition` at the end and remove `InsertIntoHiveTable`.
This PR removes `SparkHiveWriterContainer` and
`SparkHiveDynamicPartitionWriterContainer`, and create a `HiveFileFormat` to
implement the write logic. In the future, we should also implement the read
logic in `HiveFileFormat`.
## How was this patch tested?
existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark insert-hive
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16517.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16517
----
commit ca67a388d2b26f080b5422d9bb720da63e24686d
Author: Wenchen Fan <[email protected]>
Date: 2017-01-09T14:12:48Z
Implement InsertIntoHiveTable with FileCommitProtocol and FileFormatWriter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
