sambhav37 opened a new pull request, #59765: URL: https://github.com/apache/spark/pull/59765
### What changes were proposed in this pull request? In `AlterTableAddPartitionCommand`, when `IF NOT EXISTS` is specified and the table stats are updated incrementally (`spark.sql.statistics.size.autoUpdate.enabled = true` and the table has stats), exclude partitions that already exist from the stats update: - their locations are no longer counted in the added size, and - they are no longer passed to `catalog.alterPartitions`. The existing partitions are determined before the new partitions are created, and only on this code path, so the common path does not issue extra catalog calls. This is a follow-up to the review comment in #59728 (SPARK-60129). ### Why are the changes needed? Since SPARK-60129, `createPartitions(..., ignoreIfExists = true)` leaves existing partitions untouched in both the in-memory and Hive catalogs. However, the stats branch of `AlterTableAddPartitionCommand` then called `alterPartitions` with all requested partitions, so an existing partition named in `ALTER TABLE .. ADD IF NOT EXISTS PARTITION` still had its storage (e.g. its location) replaced. The size of the requested location was also added to the table's `sizeInBytes`, even though nothing was added: ```sql SET spark.sql.statistics.size.autoUpdate.enabled=true; CREATE TABLE t (id INT, part INT) USING PARQUET PARTITIONED BY (part); INSERT INTO t PARTITION (part=0) SELECT 0; -- '/other/dir' is a non-empty directory ALTER TABLE t ADD IF NOT EXISTS PARTITION (part=0) LOCATION '/other/dir'; -- Before this PR: the location of part=0 becomes '/other/dir', and the table size grows. ``` ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug. `ALTER TABLE .. ADD IF NOT EXISTS PARTITION` no longer changes the metadata of existing partitions and no longer inflates the table size when auto size update is enabled. ### How was this patch tested? Added a new test, "SPARK-60527: keep existing partitions and stats with IF NOT EXISTS", to the v1 `AlterTableAddPartitionSuiteBase`. It runs for both the in-memory and Hive v1 catalogs. The test fails without the fix and passes with it: ``` build/sbt 'sql/testOnly org.apache.spark.sql.execution.command.v1.AlterTableAddPartitionSuite' build/sbt 'hive/testOnly org.apache.spark.sql.hive.execution.command.AlterTableAddPartitionSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cortex Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
