dongjoon-hyun opened a new pull request, #59733:
URL: https://github.com/apache/spark/pull/59733
### What changes were proposed in this pull request?
This PR aims to keep the Hive incompatible partition columns of data source
tables when `HiveExternalCatalog.alterTable` and
`HiveExternalCatalog.renameTable` write the raw table metadata back to Hive
metastore.
Since SPARK-51418 (#50182), Hive incompatible partition columns like
`TIMESTAMP_NTZ` and ANSI interval columns are stored in Hive metastore with the
placeholder type, `<derived from deserializer>`, and `HiveClientImpl` skips
them when it reads the raw table metadata. `alterTable` and `renameTable` write
the raw table metadata back, so they drop these partition columns. Hive
metastore rejects this because it does not allow changing the partition
columns. It compares the number and the names of the partition columns
case-sensitively, and it stores the names in lower case.
SPARK-51840 (#50623) restored the partition columns in `alterTable` from the
table properties when the raw table schema is `EMPTY_DATA_SCHEMA`. However,
- it does not cover tables whose raw table schema is not `EMPTY_DATA_SCHEMA`;
- it uses the case-preserving names from the table properties;
- it also adds partition columns that are not in Hive metastore. Data source
tables created by Spark 2.0 or prior have the partition columns only in the
table properties (SPARK-17970 started to store them in Hive metastore in Spark
2.1).
This PR adds
- `RawHiveTable.hivePartitionColumnNames()` to get the names of all
partition columns in Hive metastore, including the Hive incompatible ones.
- `HiveExternalCatalog.getRawTableWithAllPartitionColumns`, which restores
the missing partition columns with the names and the order in Hive metastore
and the data types in the table properties.
`alterTable` uses it instead of the SPARK-51840 restoration, and
`renameTable` uses it instead of `getRawTable`.
### Why are the changes needed?
For data source tables with Hive incompatible partition columns, the
following cases fail. `ALTER TABLE ... SET TBLPROPERTIES`, `ALTER TABLE ...
UNSET TBLPROPERTIES`, and `MSCK REPAIR TABLE` call `alterTable`, and `ALTER
TABLE ... RENAME TO` calls `renameTable`.
```
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
partition keys can not be changed.
```
1. `alterTable` after `ALTER TABLE ... ADD COLUMNS` with Hive compatible new
columns. `alterTableDataSchema` stores the new data columns in Hive metastore,
so the raw table schema is no longer `EMPTY_DATA_SCHEMA`.
```sql
CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(pt);
ALTER TABLE t ADD COLUMNS (c STRING);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
2. `alterTable` for a table with both Hive compatible and Hive incompatible
partition columns.
```sql
CREATE TABLE t (i INT, p1 STRING, p2 TIMESTAMP_NTZ) USING PARQUET
PARTITIONED BY (p1, p2);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
3. `alterTable` for a Hive incompatible partition column whose name has
upper case letters.
```sql
CREATE TABLE t (i INT, PT TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(PT);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
4. `renameTable` for any table with Hive incompatible partition columns.
```sql
CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(pt);
ALTER TABLE t RENAME TO t2;
```
In addition, since SPARK-51840, `alterTable` fails in the same way for
partitioned data source tables created by Spark 2.0 or prior, because it adds
the partition columns that are not in Hive metastore. This PR fixes it too, and
`renameTable` keeps working for these tables.
### Does this PR introduce _any_ user-facing change?
Yes. The above commands work for data source tables with Hive incompatible
partition columns. They have failed since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which added the placeholder type (SPARK-51418). `TIME` partition columns fail
in the same way since SPARK-60482 (#59720), which is not released yet. Apache
Spark 4.0.x does not have SPARK-51840, so `alterTable` fails there even without
`ALTER TABLE ... ADD COLUMNS`. For partitioned data source tables created by
Spark 2.0 or prior, `alterTable` works again as it did before Apache Spark
[v4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveDDLSuite`. Both test
cases fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
