[
https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Liang updated SPARK-18185:
-------------------------------
Description:
As of current 2.1, INSERT OVERWRITE with dynamic partitions against a
Datasource table will overwrite the entire table instead of only the updated
partitions as in Hive. It also doesn't respect custom partition locations.
We should delete only the proper partitions, scan the metastore for affected
partitions with custom locations, and ensure that deletes/writes go to the
right locations for those as well.
was:
As of current 2.1, INSERT OVERWRITE with dynamic partitions against a
Datasource table will overwrite the entire table instead of only the updated
partitions as in Hive.
This is non-trivial to fix in 2.1, so we should throw an exception here.
> Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions
> ------------------------------------------------------------------------------
>
> Key: SPARK-18185
> URL: https://issues.apache.org/jira/browse/SPARK-18185
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Eric Liang
>
> As of current 2.1, INSERT OVERWRITE with dynamic partitions against a
> Datasource table will overwrite the entire table instead of only the updated
> partitions as in Hive. It also doesn't respect custom partition locations.
> We should delete only the proper partitions, scan the metastore for affected
> partitions with custom locations, and ensure that deletes/writes go to the
> right locations for those as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
