[
https://issues.apache.org/jira/browse/SPARK-60129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-60129:
-----------------------------------
Assignee: kumar sambhav
> RECOVER PARTITIONS changes an existing partition's location after SET LOCATION
> ------------------------------------------------------------------------------
>
> Key: SPARK-60129
> URL: https://issues.apache.org/jira/browse/SPARK-60129
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark: 5.0.0-SNAPSHOT
> Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Reporter: Yibo Dong
> Assignee: kumar sambhav
> Priority: Major
> Labels: pull-request-available
>
> h2. What happened
> After changing a partitioned table's root location using {{ALTER TABLE ...
> SET LOCATION}}, {{ALTER TABLE ... RECOVER PARTITIONS}} can change the
> location of an already registered partition.
> As a result, subsequent queries return different data for the same partition,
> even though its location was never explicitly changed.
> h2. How to reproduce
> Save the following SQL as {{tmp.sql}}:
> {code:sql}
> DROP TABLE IF EXISTS t_set_location_existing;
> DROP TABLE IF EXISTS t_set_location_new;
> CREATE TABLE t_set_location_existing (
> id INT,
> value STRING
> )
> USING PARQUET
> PARTITIONED BY (p STRING)
> LOCATION '/tmp/spark_t_set_location_old';
> INSERT INTO t_set_location_existing VALUES
> (1, 'old-a', 'a'),
> (2, 'old-b', 'b');
> CREATE TABLE t_set_location_new (
> id INT,
> value STRING
> )
> USING PARQUET
> PARTITIONED BY (p STRING)
> LOCATION '/tmp/spark_t_set_location_new';
> INSERT INTO t_set_location_new VALUES (3, 'new-b', 'b');
> ALTER TABLE t_set_location_existing RECOVER PARTITIONS;
> SELECT id, value, p
> FROM t_set_location_existing
> ORDER BY id;
> ALTER TABLE t_set_location_existing
> SET LOCATION '/tmp/spark_t_set_location_new';
> SELECT id, value, p
> FROM t_set_location_existing
> ORDER BY id;
> ALTER TABLE t_set_location_existing RECOVER PARTITIONS;
> SELECT id, value, p
> FROM t_set_location_existing
> ORDER BY id;
> {code}
> Start with clean directories and run:
> {code:bash}
> rm -rf /tmp/spark_t_set_location_old
> /tmp/spark_t_set_location_new
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.shuffle.partitions=1
> -f tmp.sql
> {code}
> h2. Expected result
> Changing the table's root location should not change the locations of already
> registered partitions. The subsequent {{RECOVER PARTITIONS}} should not
> replace existing partitions.
> All three queries should return:
> {code:none}
> id | value | p
> ---+-------+---
> 1 | old-a | a
> 2 | old-b | b
> {code}
> h2. Actual result
> After the initial {{RECOVER PARTITIONS}}, the query returns:
> {code:none}
> id | value | p
> ---+-------+---
> 1 | old-a | a
> 2 | old-b | b
> {code}
> After {{ALTER TABLE ... SET LOCATION}}, the query still returns:
> {code:none}
> id | value | p
> ---+-------+---
> 1 | old-a | a
> 2 | old-b | b
> {code}
> However, after running {{RECOVER PARTITIONS}} again, the result changes to:
> {code:none}
> id | value | p
> ---+-------+---
> 1 | old-a | a
> 3 | new-b | b
> {code}
> The existing partition {{p=b}} now returns data from the new table root
> instead of its original location, although no partition-level location change
> was requested.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
