This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 76bb46a017b4 [SPARK-60492][SQL] Keep Hive incompatible partition 
columns in `HiveExternalCatalog.(alter|rename)Table`
76bb46a017b4 is described below

commit 76bb46a017b4e3f3865e0a1f2509863cf73c5f4a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Oct 26 11:00:44 2026 -0700

    [SPARK-60492][SQL] Keep Hive incompatible partition columns in 
`HiveExternalCatalog.(alter|rename)Table`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to keep the Hive incompatible partition columns of data source 
tables when `HiveExternalCatalog.alterTable` and 
`HiveExternalCatalog.renameTable` write the raw table metadata back to Hive 
metastore.
    
    Since SPARK-51418 (#50182), Hive incompatible partition columns like 
`TIMESTAMP_NTZ` and ANSI interval columns are stored in Hive metastore with the 
placeholder type, `<derived from deserializer>`, and `HiveClientImpl` skips 
them when it reads the raw table metadata. `alterTable` and `renameTable` write 
the raw table metadata back, so they drop these partition columns. Hive 
metastore rejects this because it does not allow changing the partition 
columns. It compares the number and the  [...]
    
    SPARK-51840 (#50623) restored the partition columns in `alterTable` from 
the table properties when the raw table schema is `EMPTY_DATA_SCHEMA`. However,
    - it does not cover tables whose raw table schema is not 
`EMPTY_DATA_SCHEMA`;
    - it uses the case-preserving names from the table properties;
    - it also adds partition columns that are not in Hive metastore. Data 
source tables created by Spark 2.0 or prior have the partition columns only in 
the table properties (SPARK-17970 started to store them in Hive metastore in 
Spark 2.1).
    
    This PR adds
    - `RawHiveTable.hivePartitionColumnNames()` to get the names of all 
partition columns in Hive metastore, including the Hive incompatible ones.
    - `HiveExternalCatalog.getRawTableWithAllPartitionColumns`, which restores 
the missing partition columns with the names and the order in Hive metastore 
and the data types in the table properties.
    
    `alterTable` uses it instead of the SPARK-51840 restoration, and 
`renameTable` uses it instead of `getRawTable`.
    
    ### Why are the changes needed?
    
    For data source tables with Hive incompatible partition columns, the 
following cases fail. `ALTER TABLE ... SET TBLPROPERTIES`, `ALTER TABLE ... 
UNSET TBLPROPERTIES`, and `MSCK REPAIR TABLE` call `alterTable`, and `ALTER 
TABLE ... RENAME TO` calls `renameTable`.
    
    ```
    org.apache.spark.sql.AnalysisException: 
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter table. 
partition keys can not be changed.
    ```
    
    1. `alterTable` after `ALTER TABLE ... ADD COLUMNS` with Hive compatible 
new columns. `alterTableDataSchema` stores the new data columns in Hive 
metastore, so the raw table schema is no longer `EMPTY_DATA_SCHEMA`.
        ```sql
        CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY 
(pt);
        ALTER TABLE t ADD COLUMNS (c STRING);
        ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
        ```
    2. `alterTable` for a table with both Hive compatible and Hive incompatible 
partition columns.
        ```sql
        CREATE TABLE t (i INT, p1 STRING, p2 TIMESTAMP_NTZ) USING PARQUET 
PARTITIONED BY (p1, p2);
        ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
        ```
    3. `alterTable` for a Hive incompatible partition column whose name has 
upper case letters.
        ```sql
        CREATE TABLE t (i INT, PT TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY 
(PT);
        ALTER TABLE t SET TBLPROPERTIES ('k' = 'v');
        ```
    4. `renameTable` for any table with Hive incompatible partition columns.
        ```sql
        CREATE TABLE t (i INT, pt TIMESTAMP_NTZ) USING PARQUET PARTITIONED BY 
(pt);
        ALTER TABLE t RENAME TO t2;
        ```
    
    In addition, since SPARK-51840, `alterTable` fails in the same way for 
partitioned data source tables created by Spark 2.0 or prior, because it adds 
the partition columns that are not in Hive metastore. This PR fixes it too, and 
`renameTable` keeps working for these tables.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The above commands work for data source tables with Hive incompatible 
partition columns. They have failed since Apache Spark 
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), 
which added the placeholder type (SPARK-51418). `TIME` partition columns fail 
in the same way since SPARK-60482 (#59720), which is not released yet. Apache 
Spark 4.0.x does not have SPARK-51840, so `alterTable` fails there even without 
`ALTER TABLE ... ADD COLUMNS`. For partitioned  [...]
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases in `HiveDDLSuite`. Both test 
cases fail without this fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59733 from dongjoon-hyun/SPARK-60492.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit e9b194c2d8206b40a6cea8994cc913bc06cc8790)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/HiveExternalCatalog.scala       | 43 +++++++++----
 .../apache/spark/sql/hive/client/HiveClient.scala  |  6 ++
 .../spark/sql/hive/client/HiveClientImpl.scala     |  4 ++
 .../spark/sql/hive/execution/HiveDDLSuite.scala    | 72 +++++++++++++++++++++-
 4 files changed, 113 insertions(+), 12 deletions(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 484bde04aeb3..ddb23b833fa1 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -135,6 +135,34 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, 
hadoopConf: Configurat
     client.getTablesByName(db, tables)
   }
 
+  /**
+   * Get the raw table metadata with all the partition columns in Hive 
metastore. The raw table
+   * metadata does not contain the Hive incompatible partition columns, which 
are stored in Hive
+   * metastore with a placeholder type (see `HiveClientImpl.toHiveTable`). 
Here we restore them
+   * from the table properties, so that we can write the raw table metadata 
back to Hive
+   * metastore, which does not allow changing the partition columns. Note that 
data source tables
+   * created by Spark 2.0 or prior have the partition columns only in the 
table properties.
+   */
+  private def getRawTableWithAllPartitionColumns(db: String, table: String): 
CatalogTable = {
+    val rawHiveTable = client.getRawHiveTable(db, table)
+    val rawTable = rawHiveTable.toCatalogTable
+    val partitionColumnNames = rawHiveTable.hivePartitionColumnNames()
+    if (partitionColumnNames.length > rawTable.partitionColumnNames.length) {
+      val partitionSchema =
+        rawTable.partitionSchema ++ 
restoreTableMetadata(rawTable).partitionSchema
+      val partitionFields = partitionColumnNames.flatMap { name =>
+        // Hive metastore is not case preserving and keeps partition columns 
with lower cased
+        // names, while the table properties keep the original names.
+        partitionSchema.find(_.name.equalsIgnoreCase(name)).map(_.copy(name = 
name))
+      }
+      rawTable.copy(
+        schema = StructType(rawTable.dataSchema ++ partitionFields),
+        partitionColumnNames = partitionFields.map(_.name))
+    } else {
+      rawTable
+    }
+  }
+
   /**
    * If the given table properties contains datasource properties, throw an 
exception. We will do
    * this check when create or alter a table, i.e. when we try to write table 
metadata to Hive
@@ -561,7 +589,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, 
hadoopConf: Configurat
       db: String,
       oldName: String,
       newName: String): Unit = withClient {
-    val rawTable = getRawTable(db, oldName)
+    val rawTable = getRawTableWithAllPartitionColumns(db, oldName)
 
     // Note that Hive serde tables don't use path option in storage properties 
to store the value
     // of table location, but use `locationUri` field to store it directly. 
And `locationUri` field
@@ -637,7 +665,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, 
hadoopConf: Configurat
           client.alterTable(newTable.copy(schema = EMPTY_DATA_SCHEMA))
       }
     } else {
-      val oldTableDef = getRawTable(db, tableDefinition.identifier.table)
+      val oldTableDef = getRawTableWithAllPartitionColumns(db, 
tableDefinition.identifier.table)
 
       val newStorage = if (DDLUtils.isHiveTable(tableDefinition)) {
         tableDefinition.storage
@@ -705,19 +733,12 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, 
hadoopConf: Configurat
       val newTableProps =
         propsFromOldTable ++ tableDefinition.properties + 
partitionProviderProp ++ newFormatIfExists
 
-      val (newSchema, partitionColumnNames) = if (oldTableDef.schema == 
EMPTY_DATA_SCHEMA) {
-        val restoredOldTable = restoreTableMetadata(oldTableDef)
-        (StructType(EMPTY_DATA_SCHEMA ++ restoredOldTable.partitionSchema),
-          restoredOldTable.partitionColumnNames)
-      } else {
-        (oldTableDef.schema, oldTableDef.partitionColumnNames)
-      }
       // // Add old table's owner if we need to restore
       val owner = 
Option(tableDefinition.owner).filter(_.nonEmpty).getOrElse(oldTableDef.owner)
       val newDef = tableDefinition.copy(
         storage = newStorage,
-        schema = newSchema,
-        partitionColumnNames = partitionColumnNames,
+        schema = oldTableDef.schema,
+        partitionColumnNames = oldTableDef.partitionColumnNames,
         bucketSpec = oldTableDef.bucketSpec,
         properties = newTableProps,
         owner = owner)
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index 7503d7fdcd14..71e08ea3e5eb 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -33,6 +33,12 @@ private[hive] trait RawHiveTable {
 
   /** Get hive table properties. */
   def hiveTableProps(): Map[String, String]
+
+  /**
+   * Get hive partition column names, including the Hive incompatible 
partition columns which
+   * `toCatalogTable` does not contain.
+   */
+  def hivePartitionColumnNames(): Seq[String]
 }
 
 /**
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 6bb3e15da968..1ad479aed762 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -113,6 +113,10 @@ private[hive] class HiveClientImpl(
     override def hiveTableProps(): Map[String, String] = {
       rawTable.getParameters.asScala.toMap
     }
+
+    override def hivePartitionColumnNames(): Seq[String] = {
+      rawTable.getPartCols.asScala.map(_.getName).toSeq
+    }
   }
 
   import HiveClientImpl._
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
index ff07dd6b7020..4c6d4fa3b424 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.hive.execution
 
 import java.io.File
 import java.net.URI
-import java.time.LocalDateTime
+import java.time.{LocalDateTime, LocalTime}
 import java.util.Locale
 
 import scala.jdk.CollectionConverters._
@@ -629,6 +629,76 @@ class HiveDDLSuite
     }
   }
 
+  test("SPARK-60492: Keep Hive incompatible partition columns in alterTable 
and renameTable") {
+    Seq(
+      ("TIMESTAMP_NTZ", "TIMESTAMP_NTZ '2018-11-17 13:33:33'",
+        LocalDateTime.of(2018, 11, 17, 13, 33, 33)),
+      ("TIME", "TIME '12:34:56'", LocalTime.of(12, 34, 56))
+    ).foreach { case (dataType, literal, value) =>
+      def checkTable(partitionColumnNames: Seq[String], expected: Row): Unit = 
{
+        val table = 
spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+        assert(table.properties.get("k").contains("v"))
+        assert(table.partitionColumnNames == partitionColumnNames)
+        checkAnswer(spark.table("t2"), expected)
+      }
+      withTable("t", "t2") {
+        sql(s"CREATE TABLE t (i INT, pt $dataType) USING PARQUET PARTITIONED 
BY (pt)")
+        // This stores the data columns in Hive metastore as they are Hive 
compatible.
+        sql("ALTER TABLE t ADD COLUMNS (c STRING)")
+        sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+        sql(s"INSERT INTO t SELECT 1, 'a', $literal")
+        sql("MSCK REPAIR TABLE t")
+        sql("ALTER TABLE t RENAME TO t2")
+        checkTable(Seq("pt"), Row(1, "a", value))
+      }
+      withTable("t", "t2") {
+        sql(s"CREATE TABLE t (i INT, P1 STRING, P2 $dataType) USING PARQUET " +
+          "PARTITIONED BY (P1, P2)")
+        sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+        sql(s"INSERT INTO t SELECT 1, 'a', $literal")
+        sql("MSCK REPAIR TABLE t")
+        sql("ALTER TABLE t RENAME TO t2")
+        checkTable(Seq("P1", "P2"), Row(1, "a", value))
+      }
+      withTable("t", "t2") {
+        sql(s"CREATE TABLE t (i INT, P1 STRING, P2 $dataType) USING PARQUET " +
+          "PARTITIONED BY (P2, P1)")
+        sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+        sql(s"INSERT INTO t SELECT 1, $literal, 'a'")
+        sql("MSCK REPAIR TABLE t")
+        sql("ALTER TABLE t RENAME TO t2")
+        checkTable(Seq("P2", "P1"), Row(1, value, "a"))
+      }
+    }
+  }
+
+  test("SPARK-60492: alterTable and renameTable of a partitioned table created 
by Spark 2.0") {
+    // Spark 2.0 or prior stores the partition columns of a data source table 
only in the table
+    // properties, not in Hive metastore.
+    withTable("t", "t2") {
+      val schema = new StructType().add("i", "int").add("p", "int")
+      hiveClient.createTable(CatalogTable(
+        identifier = TableIdentifier("t", Some("default")),
+        tableType = CatalogTableType.MANAGED,
+        storage = CatalogStorageFormat.empty.copy(properties = Map(
+          "path" -> 
spark.sessionState.catalog.defaultTablePath(TableIdentifier("t")).toString)),
+        schema = HiveExternalCatalog.EMPTY_DATA_SCHEMA,
+        provider = Some("json"),
+        properties = Map(
+          "spark.sql.sources.provider" -> "json",
+          "spark.sql.sources.schema.numParts" -> "1",
+          "spark.sql.sources.schema.part.0" -> schema.json,
+          "spark.sql.sources.schema.numPartCols" -> "1",
+          "spark.sql.sources.schema.partCol.0" -> "p")), ignoreIfExists = 
false)
+      sql("ALTER TABLE t SET TBLPROPERTIES ('k' = 'v')")
+      sql("ALTER TABLE t RENAME TO t2")
+      val table = 
spark.sessionState.catalog.getTableMetadata(TableIdentifier("t2"))
+      assert(table.properties.get("k").contains("v"))
+      assert(table.schema == schema)
+      assert(table.partitionColumnNames == Seq("p"))
+    }
+  }
+
   test("add/drop partitions - external table") {
     val catalog = spark.sessionState.catalog
     withTempDir { tmpDir =>


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to