[
https://issues.apache.org/jira/browse/SPARK-60527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131374#comment-18131374
]
kumar sambhav commented on SPARK-60527:
---------------------------------------
I'd like to work on this.
> ALTER TABLE ADD IF NOT EXISTS PARTITION overwrites existing partitions when
> auto size update is enabled
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60527
> URL: https://issues.apache.org/jira/browse/SPARK-60527
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: kumar sambhav
> Priority: Major
> Labels: pull-request-available
>
> h4. Problem
> When {{spark.sql.statistics.size.autoUpdate.enabled}} is true and the table
> has stats, {{ALTER TABLE .. ADD IF NOT EXISTS PARTITION}} still overwrites
> existing partitions named in the command. {{AlterTableAddPartitionCommand}}
> first calls {{{}catalog.createPartitions(..., ignoreIfExists = true){}}},
> which skips existing partitions (in both the in-memory and Hive catalogs
> since SPARK-60129). However, the stats branch then: * computes {{addedSize}}
> from the requested locations of all partitions, including existing ones,
> which inflates the table's {{{}sizeInBytes{}}}, and * calls
> {{catalog.alterPartitions(table.identifier, parts)}} with all requested
> partitions, which replaces the storage (e.g. the {{{}LOCATION{}}}) of
> existing partitions.
> The issue is at the command level, so it applies to both the in-memory and
> Hive catalogs. h2. How to reproduce
> {code:sql}
> SET spark.sql.statistics.size.autoUpdate.enabled=true;
> CREATE TABLE t (id INT, part INT) USING PARQUET PARTITIONED BY (part); INSERT
> INTO t PARTITION (part=0) SELECT 0; -- '/other/dir' is any non-empty
> directory
> ALTER TABLE t ADD IF NOT EXISTS PARTITION (part=0) LOCATION '/other/dir';
> DESCRIBE TABLE EXTENDED t PARTITION (part=0); {code}
> h4. Expected: the location of {{part=0}} and the table size are unchanged,
> because the partition already exists. Actual: the location of {{part=0}}
> becomes {{{}/other/dir{}}}, and the table size grows by the size of
> {{{}/other/dir{}}}. h2. Proposed fix When {{IF NOT EXISTS}} is specified and
> the stats are updated incrementally, determine the already-existing
> partitions before creating the new ones, and exclude them from both
> {{addedSize}} and the {{alterPartitions}} call.
> References Reported by [~cloud_fan] in the review of SPARK-60129: [PR #59728
> comment|https://github.com/apache/spark/pull/59728#issuecomment-6371577273]
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
