sambhav37 opened a new pull request, #59728:
URL: https://github.com/apache/spark/pull/59728
### What changes were proposed in this pull request?
In `InMemoryCatalog.createPartitions`, skip partitions that already exist
when `ignoreIfExists = true`, instead of overwriting their metadata. Previously
`ignoreIfExists` only disabled the duplicate check, and the subsequent loop
still replaced every existing partition (including its location) with the new
definition.
This aligns `InMemoryCatalog` with `HiveExternalCatalog`, which passes
`ifNotExists` to the Hive metastore and leaves existing partitions untouched.
### Why are the changes needed?
`ALTER TABLE .. RECOVER PARTITIONS` calls `createPartitions(...,
ignoreIfExists = true)` for every partition directory it finds under the table
root. With the in-memory catalog, after the table root is changed via `ALTER
TABLE .. SET LOCATION`, recovering partitions silently repoints
already-registered partitions to the new root, so queries return different data:
```sql
CREATE TABLE t (id INT, value STRING) USING PARQUET PARTITIONED BY (p STRING)
LOCATION '/tmp/spark_t_old';
INSERT INTO t VALUES (1, 'old-a', 'a'), (2, 'old-b', 'b');
CREATE TABLE t_new (id INT, value STRING) USING PARQUET PARTITIONED BY (p
STRING)
LOCATION '/tmp/spark_t_new';
INSERT INTO t_new VALUES (3, 'new-b', 'b');
ALTER TABLE t SET LOCATION '/tmp/spark_t_new';
ALTER TABLE t RECOVER PARTITIONS;
SELECT id, value, p FROM t ORDER BY id;
-- 1 old-a a
-- 3 new-b b <- partition p=b now points to /tmp/spark_t_new/p=b
```
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug. With the in-memory catalog
(`spark.sql.catalogImplementation=in-memory`), `ALTER TABLE .. RECOVER
PARTITIONS` no longer changes the location of existing partitions. For the
example above, the last query now returns:
```
1 old-a a
2 old-b b
```
This matches the behavior of the Hive catalog.
### How was this patch tested?
Added new tests:
- `ExternalCatalogSuite`: "create partitions with ignoreIfExists does not
alter existing partitions". It runs for both `InMemoryCatalogSuite` and
`HiveExternalCatalogSuite`.
- v1 `AlterTableRecoverPartitionsSuiteBase`: "SPARK-60129: keep locations of
existing partitions". It runs for the in-memory (sequential and parallel) and
Hive v1 suites.
Both new tests fail without the fix and pass with it:
```
build/sbt 'catalyst/testOnly *InMemoryCatalogSuite'
build/sbt 'sql/testOnly
org.apache.spark.sql.execution.command.v1.AlterTableRecoverPartitions*'
build/sbt 'hive/testOnly *HiveExternalCatalogSuite
org.apache.spark.sql.hive.execution.command.AlterTableRecoverPartitionsSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cortex Code (claude-opus-5-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
