[
https://issues.apache.org/jira/browse/SPARK-60127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibo Dong updated SPARK-60127:
------------------------------
Labels: sql (was: )
> ADD IF NOT EXISTS PARTITION changes the location of an existing partition
> -------------------------------------------------------------------------
>
> Key: SPARK-60127
> URL: https://issues.apache.org/jira/browse/SPARK-60127
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark: 5.0.0-SNAPSHOT
> Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Reporter: Yibo Dong
> Priority: Major
> Labels: sql
>
> h2. What happened
> {{ALTER TABLE ADD IF NOT EXISTS PARTITION}} changes the location of an
> existing partition when a different {{LOCATION}} is specified. As a result,
> subsequent queries return data from the new location instead of the original
> partition data.
> h2. How to reproduce
> Run the following SQL using {{{}spark-sql{}}}:
> {code:sql}
> DROP TABLE IF EXISTS t_add_partition_if_not_exists;
> DROP TABLE IF EXISTS t_add_partition_if_not_exists_alternate;
> CREATE TABLE t_add_partition_if_not_exists (
> id INT,
> value STRING
> )
> USING PARQUET
> PARTITIONED BY (p STRING)
> LOCATION '/tmp/spark_t_add_partition_if_not_exists_old';
> INSERT INTO t_add_partition_if_not_exists VALUES (1, 'old', 'a');
> CREATE TABLE t_add_partition_if_not_exists_alternate (
> id INT,
> value STRING
> )
> USING PARQUET
> PARTITIONED BY (p STRING)
> LOCATION '/tmp/spark_t_add_partition_if_not_exists_alternate';
> INSERT INTO t_add_partition_if_not_exists_alternate VALUES (2, 'alternate',
> 'a');
> DROP TABLE t_add_partition_if_not_exists_alternate;
> SELECT id, value, p
> FROM t_add_partition_if_not_exists
> ORDER BY id;
> ALTER TABLE t_add_partition_if_not_exists
> ADD IF NOT EXISTS PARTITION (p = 'a')
> LOCATION '/tmp/spark_t_add_partition_if_not_exists_alternate/p=a';
> SHOW PARTITIONS t_add_partition_if_not_exists;
> SELECT id, value, p
> FROM t_add_partition_if_not_exists
> ORDER BY id;
> {code}
> Save the SQL as {{{}repro.sql{}}}. Start with clean directories and run:
> {code:bash}
> rm -rf /tmp/spark_t_add_partition_if_not_exists_old
> /tmp/spark_t_add_partition_if_not_exists_alternate
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> -f repro.sql
> {code}
> h2. Expected result
> Since partition {{p=a}} already exists, {{ADD IF NOT EXISTS PARTITION}}
> should leave its location unchanged.
> Both queries should return:
> {code}
> id | value | p
> ---+-------+---
> 1 | old | a
> {code}
> h2. Actual result
> Before the {{ALTER TABLE}} statement, the query returns:
> {code}
> id | value | p
> ---+-------+---
> 1 | old | a
> {code}
> After {{{}ADD IF NOT EXISTS PARTITION{}}}, the query returns:
> {code}
> id | value | p
> ---+-----------+---
> 2 | alternate | a
> {code}
> {{SHOW PARTITIONS}} still lists only {{{}p=a{}}}, but the existing partition
> now reads data from the alternate location.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
