[
https://issues.apache.org/jira/browse/SPARK-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-20236:
--------------------------------
Description:
When we overwrite a partitioned data source table, currently Spark will
truncate the entire table to write new data, or truncate a bunch of partitions
according to the given static partitions.
For example, {{INSERT OVERWRITE tbl ...}} will truncate the entire table,
{{INSERT OVERWRITE tbl PARTITION (a=1, b)}} will truncate all the partitions
that starts with {{a=1}}.
This behavior is kind of reasonable as we can know which partitions will be
overwritten before runtime. However, hive has a different behavior that it only
overwrites related partitions, e.g. {{INSERT OVERWRITE tbl SELECT 1,2,3}} will
only overwrite partition {{a=2, b=3}}, assuming {{tbl}} has only one data
column and is partitioned by {{a}} and {{b}}.
It seems better if we can follow hive's behavior.
was:
When we overwrite a partitioned table, currently Spark will truncate the entire
table to write new data, or truncate a bunch of partitions according to the
given static partitions.
For example, {{INSERT OVERWRITE tbl ...}} will truncate the entire table,
{{INSERT OVERWRITE tbl PARTITION (a=1, b)}} will truncate all the partitions
that starts with {{a=1}}.
This behavior is kind of reasonable as we can know which partitions will be
overwritten before runtime. However, hive has a different behavior that it only
overwrites related partitions, e.g. {{INSERT OVERWRITE tbl SELECT 1,2,3}} will
only overwrite partition {{a=2, b=3}}, assuming {{tbl}} has only one data
column and is partitioned by {{a}} and {{b}}.
It seems better if we can follow hive's behavior.
> Overwrite a partitioned table should only overwrite related partitions
> ----------------------------------------------------------------------
>
> Key: SPARK-20236
> URL: https://issues.apache.org/jira/browse/SPARK-20236
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: Wenchen Fan
>
> When we overwrite a partitioned data source table, currently Spark will
> truncate the entire table to write new data, or truncate a bunch of
> partitions according to the given static partitions.
> For example, {{INSERT OVERWRITE tbl ...}} will truncate the entire table,
> {{INSERT OVERWRITE tbl PARTITION (a=1, b)}} will truncate all the partitions
> that starts with {{a=1}}.
> This behavior is kind of reasonable as we can know which partitions will be
> overwritten before runtime. However, hive has a different behavior that it
> only overwrites related partitions, e.g. {{INSERT OVERWRITE tbl SELECT
> 1,2,3}} will only overwrite partition {{a=2, b=3}}, assuming {{tbl}} has only
> one data column and is partitioned by {{a}} and {{b}}.
> It seems better if we can follow hive's behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
