[
https://issues.apache.org/jira/browse/SPARK-60527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kumar sambhav updated SPARK-60527:
----------------------------------
Description:
h4. Problem
When {{spark.sql.statistics.size.autoUpdate.enabled}} is true and the table has
stats, {{ALTER TABLE .. ADD IF NOT EXISTS PARTITION}} still overwrites existing
partitions named in the command. {{AlterTableAddPartitionCommand}} first calls
{{{}catalog.createPartitions(..., ignoreIfExists = true){}}}, which skips
existing partitions (in both the in-memory and Hive catalogs since
SPARK-60129). However, the stats branch then: * computes {{addedSize}} from the
requested locations of all partitions, including existing ones, which inflates
the table's {{{}sizeInBytes{}}}, and * calls
{{catalog.alterPartitions(table.identifier, parts)}} with all requested
partitions, which replaces the storage (e.g. the {{{}LOCATION{}}}) of existing
partitions.
The issue is at the command level, so it applies to both the in-memory and Hive
catalogs. h2. How to reproduce
{code:sql}
SET spark.sql.statistics.size.autoUpdate.enabled=true;
CREATE TABLE t (id INT, part INT) USING PARQUET PARTITIONED BY (part); INSERT
INTO t PARTITION (part=0) SELECT 0; -- '/other/dir' is any non-empty directory
ALTER TABLE t ADD IF NOT EXISTS PARTITION (part=0) LOCATION '/other/dir';
DESCRIBE TABLE EXTENDED t PARTITION (part=0); {code}
h4. Expected: the location of {{part=0}} and the table size are unchanged,
because the partition already exists. Actual: the location of {{part=0}}
becomes {{{}/other/dir{}}}, and the table size grows by the size of
{{{}/other/dir{}}}. h2. Proposed fix When {{IF NOT EXISTS}} is specified and
the stats are updated incrementally, determine the already-existing partitions
before creating the new ones, and exclude them from both {{addedSize}} and the
{{alterPartitions}} call.
References Reported by [~cloud_fan] in the review of SPARK-60129: [PR #59728
comment|https://github.com/apache/spark/pull/59728#issuecomment-6371577273]
> ALTER TABLE ADD IF NOT EXISTS PARTITION overwrites existing partitions when
> auto size update is enabled
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60527
> URL: https://issues.apache.org/jira/browse/SPARK-60527
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: kumar sambhav
> Priority: Major
>
> h4. Problem
> When {{spark.sql.statistics.size.autoUpdate.enabled}} is true and the table
> has stats, {{ALTER TABLE .. ADD IF NOT EXISTS PARTITION}} still overwrites
> existing partitions named in the command. {{AlterTableAddPartitionCommand}}
> first calls {{{}catalog.createPartitions(..., ignoreIfExists = true){}}},
> which skips existing partitions (in both the in-memory and Hive catalogs
> since SPARK-60129). However, the stats branch then: * computes {{addedSize}}
> from the requested locations of all partitions, including existing ones,
> which inflates the table's {{{}sizeInBytes{}}}, and * calls
> {{catalog.alterPartitions(table.identifier, parts)}} with all requested
> partitions, which replaces the storage (e.g. the {{{}LOCATION{}}}) of
> existing partitions.
> The issue is at the command level, so it applies to both the in-memory and
> Hive catalogs. h2. How to reproduce
> {code:sql}
> SET spark.sql.statistics.size.autoUpdate.enabled=true;
> CREATE TABLE t (id INT, part INT) USING PARQUET PARTITIONED BY (part); INSERT
> INTO t PARTITION (part=0) SELECT 0; -- '/other/dir' is any non-empty
> directory
> ALTER TABLE t ADD IF NOT EXISTS PARTITION (part=0) LOCATION '/other/dir';
> DESCRIBE TABLE EXTENDED t PARTITION (part=0); {code}
> h4. Expected: the location of {{part=0}} and the table size are unchanged,
> because the partition already exists. Actual: the location of {{part=0}}
> becomes {{{}/other/dir{}}}, and the table size grows by the size of
> {{{}/other/dir{}}}. h2. Proposed fix When {{IF NOT EXISTS}} is specified and
> the stats are updated incrementally, determine the already-existing
> partitions before creating the new ones, and exclude them from both
> {{addedSize}} and the {{alterPartitions}} call.
> References Reported by [~cloud_fan] in the review of SPARK-60129: [PR #59728
> comment|https://github.com/apache/spark/pull/59728#issuecomment-6371577273]
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
