This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new af93ce382fbf [SPARK-60492][SQL] Keep Hive incompatible partition
columns in `HiveExternalCatalog.(alter|rename)Table`
af93ce382fbf is described below
commit af93ce382fbfd6fa3195de5348cadeae66a241e2
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Oct 26 11:00:44 2026 -0700
[SPARK-60492][SQL] Keep Hive incompatible partition columns in
`HiveExternalCatalog.(alter|rename)Table`
### What changes were proposed in this pull request?
This PR aims to keep the Hive incompatible partition columns of data source
tables when `HiveExternalCatalog.alterTable` and
`HiveExternalCatalog.renameTable` write the raw table metadata back to Hive
metastore.
Since SPARK-51418 (#50182), Hive incompatible partition columns like
`TIMESTAMP_NTZ` and ANSI interval columns are stored in Hive metastore with the
placeholder type, `<derived from deserializer>`, and `HiveClientImpl` skips
them when it reads the raw table metadata. `alterTable` and `renameTable` write
the raw table metadata back, so they drop these partition columns. Hive
metastore rejects this because it does not allow changing the partition
columns. It compares the number and the [...]
SPARK-51840 (#50623) restored the partition columns in `alterTable` from
the table properties when the raw table schema is `EMPTY_DATA_SCHEMA`. However,
- it does not cover tables whose raw table schema is not
`EMPTY_DATA_SCHEMA`;
- it uses the case-preserving names from the table properties;
- it also adds partition columns that are not in Hive metastore. Data
source tables created by Spark 2.0 or prior have the partition columns only in
the table properties (SPARK-17970 started to store them in Hive metastore in
Spark 2.1).
This PR adds
- `RawHiveTable.hivePartitionColumnNames()` to get the names of all
partition columns in Hive metastore, including the Hive incompatible ones.
- `HiveExternalCatalog.getRawTableWithAllPartitionColumns`, which restores
the missing partition columns with the names and the order in Hive metastore
and the data types in the table properties.
`alterTable` uses it instead of the SPARK-51840 restoration, and
`renameTable` uses it instead of `getRawTable`.
### Why are the changes needed?
For data source tables with Hive incompatible partition columns, the
following cases fail. `ALTER TABLE ... SET TBLPROPERTIES`, `ALTER TABLE ...
UNSET TBLPROPERTIES`, and `MSCK REPAIR TABLE` call `alterTable`, and `ALTER
TABLE ... RENAME TO` calls `renameTable`.
```
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table.
partition keys can not be changed.
```
1. `alterTable` after `ALTER TABLE ... ADD COLUMNS` with Hive compatible
new columns. `alterTableDataSchema` stores the new data columns in Hive
metastore, so the raw table schema is no longer `EMPTY_DATA_SCHEMA`.
```sql
CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(pt);
ALTER TABLE t ADD COLUMNS (c STRING);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
2. `alterTable` for a table with both Hive compatible and Hive incompatible
partition columns.
```sql
CREATE TABLE t (i INT, p1 STRING, p2 TIMESTAMP_NTZ) USING PARQUET
PARTITIONED BY (p1, p2);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
3. `alterTable` for a Hive incompatible partition column whose name has
upper case letters.
```sql
CREATE TABLE t (i INT, PT TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(PT);
ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
```
4. `renameTable` for any table with Hive incompatible partition columns.
```sql
CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY
(pt);
ALTER TABLE t RENAME TO t2;
```
In addition, since SPARK-51840, `alterTable` fails in the same way for
partitioned data source tables created by Spark 2.0 or prior, because it adds
the partition columns that are not in Hive metastore. This PR fixes it too, and
`renameTable` keeps working for these tables.
### Does this PR introduce _any_ user-facing change?
Yes. The above commands work for data source tables with Hive incompatible
partition columns. They have failed since Apache Spark
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which added the placeholder type (SPARK-51418). `TIME` partition columns fail
in the same way since SPARK-60482 (#59720), which is not released yet. Apache
Spark 4.0.x does not have SPARK-51840, so `alterTable` fails there even without
`ALTER TABLE ... ADD COLUMNS`. For partitioned [...]
### How was this patch tested?
Pass the CIs with the newly added test cases in `HiveDDLSuite`. Both test
cases fail without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59733 from dongjoon-hyun/SPARK-60492.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e9b194c2d8206b40a6cea8994cc913bc06cc8790)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/HiveExternalCatalog.scala | 43 +++++++++----
.../apache/spark/sql/hive/client/HiveClient.scala | 6 ++
.../spark/sql/hive/client/HiveClientImpl.scala | 4 ++
.../spark/sql/hive/execution/HiveDDLSuite.scala | 72 +++++++++++++++++++++-
4 files changed, 113 insertions(+), 12 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 0379fbffebaf..421c4a88280a 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -135,6 +135,34 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
client.getTablesByName(db, tables)
}
+ /**
+ * Get the raw table metadata with all the partition columns in Hive
metastore. The raw table
+ * metadata does not contain the Hive incompatible partition columns, which
are stored in Hive
+ * metastore with a placeholder type (see `HiveClientImpl.toHiveTable`).
Here we restore them
+ * from the table properties, so that we can write the raw table metadata
back to Hive
+ * metastore, which does not allow changing the partition columns. Note that
data source tables
+ * created by Spark 2.0 or prior have the partition columns only in the
table properties.
+ */
+ private def getRawTableWithAllPartitionColumns(db: String, table: String):
CatalogTable = {
+ val rawHiveTable = client.getRawHiveTable(db, table)
+ val rawTable = rawHiveTable.toCatalogTable
+ val partitionColumnNames = rawHiveTable.hivePartitionColumnNames()
+ if (partitionColumnNames.length > rawTable.partitionColumnNames.length) {
+ val partitionSchema =
+ rawTable.partitionSchema ++
restoreTableMetadata(rawTable).partitionSchema
+ val partitionFields = partitionColumnNames.flatMap { name =>
+ // Hive metastore is not case preserving and keeps partition columns
with lower cased
+ // names, while the table properties keep the original names.
+ partitionSchema.find(_.name.equalsIgnoreCase(name)).map(_.copy(name =
name))
+ }
+ rawTable.copy(
+ schema = StructType(rawTable.dataSchema ++ partitionFields),
+ partitionColumnNames = partitionFields.map(_.name))
+ } else {
+ rawTable
+ }
+ }
+
/**
* If the given table properties contains datasource properties, throw an
exception. We will do
* this check when create or alter a table, i.e. when we try to write table
metadata to Hive
@@ -564,7 +592,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
db: String,
oldName: String,
newName: String): Unit = withClient {
- val rawTable = getRawTable(db, oldName)
+ val rawTable = getRawTableWithAllPartitionColumns(db, oldName)
// Note that Hive serde tables don't use path option in storage properties
to store the value
// of table location, but use `locationUri` field to store it directly.
And `locationUri` field
@@ -640,7 +668,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
client.alterTable(newTable.copy(schema = EMPTY_DATA_SCHEMA))
}
} else {
- val oldTableDef = getRawTable(db, tableDefinition.identifier.table)
+ val oldTableDef = getRawTableWithAllPartitionColumns(db,
tableDefinition.identifier.table)
val newStorage = if (DDLUtils.isHiveTable(tableDefinition)) {
tableDefinition.storage
@@ -708,19 +736,12 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
val newTableProps =
propsFromOldTable ++ tableDefinition.properties +
partitionProviderProp ++ newFormatIfExists
- val (newSchema, partitionColumnNames) = if (oldTableDef.schema ==
EMPTY_DATA_SCHEMA) {
- val restoredOldTable = restoreTableMetadata(oldTableDef)
- (StructType(EMPTY_DATA_SCHEMA ++ restoredOldTable.partitionSchema),
- restoredOldTable.partitionColumnNames)
- } else {
- (oldTableDef.schema, oldTableDef.partitionColumnNames)
- }
// // Add old table's owner if we need to restore
val owner =
Option(tableDefinition.owner).filter(_.nonEmpty).getOrElse(oldTableDef.owner)
val newDef = tableDefinition.copy(
storage = newStorage,
- schema = newSchema,
- partitionColumnNames = partitionColumnNames,
+ schema = oldTableDef.schema,
+ partitionColumnNames = oldTableDef.partitionColumnNames,
bucketSpec = oldTableDef.bucketSpec,
properties = newTableProps,
owner = owner)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index 7503d7fdcd14..71e08ea3e5eb 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -33,6 +33,12 @@ private[hive] trait RawHiveTable {
/** Get hive table properties. */
def hiveTableProps(): Map[String, String]
+
+ /**
+ * Get hive partition column names, including the Hive incompatible
partition columns which
+ * `toCatalogTable` does not contain.
+ */
+ def hivePartitionColumnNames(): Seq[String]
}
/**
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 6bb3e15da968..1ad479aed762 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -113,6 +113,10 @@ private[hive] class HiveClientImpl(
override def hiveTableProps(): Map[String, String] = {
rawTable.getParameters.asScala.toMap
}
+
+ override def hivePartitionColumnNames(): Seq[String] = {
+ rawTable.getPartCols.asScala.map(_.getName).toSeq
+ }
}
import HiveClientImpl._
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index 9e0c1b988ac4..54d5cc734fd2 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.hive.execution
import java.io.File
import java.net.URI
-import java.time.LocalDateTime
+import java.time.{LocalDateTime, LocalTime}
import java.util.Locale
import scala.jdk.CollectionConverters._
@@ -629,6 +629,76 @@ class HiveDDLSuite
}
}
+ test("SPARK-60492: Keep Hive incompatible partition columns in alterTable
and renameTable") {
+ Seq(
+ ("TIMESTAMP_NTZ", "TIMESTAMP_NTZ '2018-11-17 13:33:33'",
+ LocalDateTime.of(2018, 11, 17, 13, 33, 33)),
+ ("TIME", "TIME '12:34:56'", LocalTime.of(12, 34, 56))
+ ).foreach { case (dataType, literal, value) =>
+ def checkTable(partitionColumnNames: Seq[String], expected: Row): Unit =
{
+ val table =
spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(table.properties.get("k").contains("v"))
+ assert(table.partitionColumnNames == partitionColumnNames)
+ checkAnswer(spark.table("t2"), expected)
+ }
+ withTable("t", "t2") {
+ sql(s"CREATE TABLE t (i INT, pt $dataType) USING PARQUET PARTITIONED
BY (pt)")
+ // This stores the data columns in Hive metastore as they are Hive
compatible.
+ sql("ALTER TABLE t ADD COLUMNS (c STRING)")
+ sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+ sql(s"INSERT INTO t SELECT 1, 'a', $literal")
+ sql("MSCK REPAIR TABLE t")
+ sql("ALTER TABLE t RENAME TO t2")
+ checkTable(Seq("pt"), Row(1, "a", value))
+ }
+ withTable("t", "t2") {
+ sql(s"CREATE TABLE t (i INT, P1 STRING, P2 $dataType) USING PARQUET " +
+ "PARTITIONED BY (P1, P2)")
+ sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+ sql(s"INSERT INTO t SELECT 1, 'a', $literal")
+ sql("MSCK REPAIR TABLE t")
+ sql("ALTER TABLE t RENAME TO t2")
+ checkTable(Seq("P1", "P2"), Row(1, "a", value))
+ }
+ withTable("t", "t2") {
+ sql(s"CREATE TABLE t (i INT, P1 STRING, P2 $dataType) USING PARQUET " +
+ "PARTITIONED BY (P2, P1)")
+ sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+ sql(s"INSERT INTO t SELECT 1, $literal, 'a'")
+ sql("MSCK REPAIR TABLE t")
+ sql("ALTER TABLE t RENAME TO t2")
+ checkTable(Seq("P2", "P1"), Row(1, value, "a"))
+ }
+ }
+ }
+
+ test("SPARK-60492: alterTable and renameTable of a partitioned table created
by Spark 2.0") {
+ // Spark 2.0 or prior stores the partition columns of a data source table
only in the table
+ // properties, not in Hive metastore.
+ withTable("t", "t2") {
+ val schema = new StructType().add("i", "int").add("p", "int")
+ hiveClient.createTable(CatalogTable(
+ identifier = TableIdentifier("t", Some("default")),
+ tableType = CatalogTableType.MANAGED,
+ storage = CatalogStorageFormat.empty.copy(properties = Map(
+ "path" ->
spark.sessionState.catalog.defaultTablePath(TableIdentifier("t")).toString)),
+ schema = HiveExternalCatalog.EMPTY_DATA_SCHEMA,
+ provider = Some("json"),
+ properties = Map(
+ "spark.sql.sources.provider" -> "json",
+ "spark.sql.sources.schema.numParts" -> "1",
+ "spark.sql.sources.schema.part.0" -> schema.json,
+ "spark.sql.sources.schema.numPartCols" -> "1",
+ "spark.sql.sources.schema.partCol.0" -> "p")), ignoreIfExists =
false)
+ sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+ sql("ALTER TABLE t RENAME TO t2")
+ val table =
spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+ assert(table.properties.get("k").contains("v"))
+ assert(table.schema == schema)
+ assert(table.partitionColumnNames == Seq("p"))
+ }
+ }
+
test("add/drop partitions - external table") {
val catalog = spark.sessionState.catalog
withTempDir { tmpDir =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]