This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 49f7408ebce1 [SPARK-60515][SQL] Keep the existing table when Hive 
rejects the new table of `saveAsTable` overwrite
49f7408ebce1 is described below

commit 49f7408ebce123dbdb9f54f0fff39a3c7f9628c7
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 07:12:26 2026 -0700

    [SPARK-60515][SQL] Keep the existing table when Hive rejects the new table 
of `saveAsTable` overwrite
    
    ### What changes were proposed in this pull request?
    
    This PR makes `CreateHiveTableAsSelectCommand` run the checks of Hive on 
the new table before it drops the existing table for `saveAsTable` with 
`SaveMode.Overwrite` and `format("hive")`.
    
    - `HiveClient.validateTable` runs the checks that `Hive.createTable` runs 
before calling Hive metastore, i.e., the SerDe initialization with the table 
columns and `Table.checkValidity`, without creating the table. Like 
`Hive.createTable`, it wraps the errors in `HiveException`.
    - `HiveExternalCatalog.validateHiveSerdeTable` calls it after removing the 
collations, like `HiveExternalCatalog.createTable`.
    - `CreateHiveTableAsSelectCommand` calls it before dropping the existing 
table.
    
    ### Why are the changes needed?
    
    SPARK-60387 made `saveAsTable` with `SaveMode.Overwrite` drop the existing 
table as late as it can. `CreateHiveTableAsSelectCommand` drops it after the 
analysis, because it creates the new table before writing. When Hive rejects 
the new table there, the existing table, and the data of a managed table, are 
already gone.
    
    ```scala
    sql("CREATE TABLE t (i INT) USING HIVE")
    sql("INSERT INTO t VALUES (1)")
    sql("SELECT 1 AS i, CAST(NULL AS VARIANT) AS 
c").write.format("hive").mode("overwrite").saveAsTable("t")
    // AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.IllegalArgumentException: Error: type expected at the position 4 of 
'int:variant' but 'variant' is found.
    spark.table("t")
    // AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t` 
cannot be found. Verify the spelling and correctness of the schema and catalog. 
...
    ```
    
    Hive rejects, for example:
    - data column types that the Hive SerDe can't parse: `TIME`, 
`TIMESTAMP_NTZ`, `TIMESTAMP_NTZ(9)`, `VARIANT`, `GEOMETRY`, `GEOGRAPHY`, and 
nested ANSI intervals
    - the void type in Parquet and ORC tables
    - nested field names like `` `a-b` ``, and `` `a b` `` in Avro tables
    - column names that differ only in case when `spark.sql.caseSensitive` is 
`true`
    
    This has been the behavior since Apache Spark 2.2.0 (SPARK-19151).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. When Hive rejects the new table before calling Hive metastore, such an 
overwrite keeps the existing table and its data. The error is the same as 
before. Valid overwrites are unchanged. A rejection by Hive metastore itself, 
e.g., for a `path` that points to a file, or a failure after creating the new 
table, e.g., in the write, still loses the existing table.
    
    ### How was this patch tested?
    
    Added tests to `MetastoreDataSourcesSuite` and `HiveClientSuite`. The test 
of the rejected overwrites in `MetastoreDataSourcesSuite` fails without this 
fix.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59757 from dongjoon-hyun/SPARK-60515.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit cd80bea5a6699c6cc72f4fbf05f0299d0c79b7ec)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/hive/HiveExternalCatalog.scala       | 10 +++++
 .../apache/spark/sql/hive/client/HiveClient.scala  |  6 +++
 .../spark/sql/hive/client/HiveClientImpl.scala     | 15 +++++++
 .../execution/CreateHiveTableAsSelectCommand.scala | 19 +++++----
 .../spark/sql/hive/MetastoreDataSourcesSuite.scala | 47 ++++++++++++++++++++++
 .../spark/sql/hive/client/HiveClientSuite.scala    | 19 ++++++++-
 6 files changed, 108 insertions(+), 8 deletions(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 02dd55f15e15..e7997c7e862f 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -306,6 +306,16 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, 
hadoopConf: Configurat
     }
   }
 
+  /**
+   * Runs the checks that Hive runs on a new Hive serde table before creating 
it in Hive
+   * metastore, e.g., the SerDe initialization with the table columns, without 
creating it.
+   * A command that replaces an existing table calls this before dropping the 
existing table.
+   */
+  private[hive] def validateHiveSerdeTable(tableDefinition: CatalogTable): 
Unit = withClient {
+    // Like `createTable`, remove the collations, which Hive doesn't support.
+    client.validateTable(tableDefinition.copy(schema = 
removeCollation(tableDefinition.schema)))
+  }
+
   private def createDataSourceTable(table: CatalogTable, ignoreIfExists: 
Boolean): Unit = {
     // data source table always have a provider, it's guaranteed by 
`DDLUtils.isDatasourceTable`.
     val provider = table.provider.get
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index ad7648096354..7503d7fdcd14 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -132,6 +132,12 @@ private[hive] trait HiveClient {
   /** Creates a table with the given metadata. */
   def createTable(table: CatalogTable, ignoreIfExists: Boolean): Unit
 
+  /**
+   * Runs the checks that [[createTable]] runs on the given table before 
calling Hive metastore,
+   * e.g., the SerDe initialization with the table columns, without creating 
the table.
+   */
+  def validateTable(table: CatalogTable): Unit
+
   /** Drop the specified table. */
   def dropTable(dbName: String, tableName: String, ignoreIfNotExists: Boolean, 
purge: Boolean): Unit
 
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index a37b33ab49c0..6bb3e15da968 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -584,6 +584,21 @@ private[hive] class HiveClientImpl(
     shim.createTable(client, toHiveTable(table, Some(userName)), 
ignoreIfExists)
   }
 
+  override def validateTable(table: CatalogTable): Unit = withHiveState {
+    val hiveTable = toHiveTable(table, Some(userName))
+    // These are the checks of `Hive.createTable` before it calls Hive 
metastore. Like it, wrap
+    // their errors in `HiveException`.
+    try {
+      if (hiveTable.getCols.isEmpty || hiveTable.getSd.getColsSize == 0) {
+        // `Hive.createTable` gets the columns from the SerDe here, e.g., for 
an Avro table.
+        hiveTable.getDeserializer
+      }
+      hiveTable.checkValidity(conf)
+    } catch {
+      case NonFatal(e) => throw new HiveException(e)
+    }
+  }
+
   override def dropTable(
       dbName: String,
       tableName: String,
diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
index c0f3bab3143b..280978b8543e 100644
--- 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
+++ 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
@@ -25,6 +25,7 @@ import 
org.apache.spark.sql.catalyst.plans.logical.{CTEInChildren, CTERelationDe
 import org.apache.spark.sql.catalyst.util.CharVarcharUtils
 import org.apache.spark.sql.errors.QueryCompilationErrors
 import org.apache.spark.sql.execution.command.{DataWritingCommand, 
LeafRunnableCommand}
+import org.apache.spark.sql.hive.HiveExternalCatalog
 import org.apache.spark.sql.hive.execution.InsertIntoHiveTable.BY_CTAS
 
 /**
@@ -65,20 +66,24 @@ case class CreateHiveTableAsSelectCommand(
       tableDesc.storage.locationUri.foreach { p =>
         DataWritingCommand.assertEmptyRootPath(p, mode, 
sparkSession.sessionState.newHadoopConf())
       }
+      val outputColumns = DataWritingCommand.logicalPlanOutputWithNames(query, 
outputColumnNames)
+      val tableSchema = CharVarcharUtils.getRawSchema(
+        outputColumns.toStructType, sparkSession.sessionState.conf)
+      assert(tableDesc.schema.isEmpty)
+      val newTableDesc = tableDesc.copy(schema = tableSchema)
       if (tableExists) {
         // `SaveAsV1TableCommand` runs this command with `SaveMode.Overwrite` 
to replace the
-        // table. Drop it here, after the analysis, because the new table is 
created before the
-        // write.
+        // table. Drop it here, after the analysis and the checks of the new 
table, because the
+        // new table is created before the write.
+        
sparkSession.sharedState.externalCatalog.unwrapped.asInstanceOf[HiveExternalCatalog]
+          .validateHiveSerdeTable(
+            newTableDesc.copy(identifier = 
catalog.qualifyIdentifier(tableIdentifier)))
         catalog.dropTable(tableIdentifier, ignoreIfNotExists = true, purge = 
false)
       }
       // TODO ideally, we should get the output data ready first and then
       // add the relation into catalog, just in case of failure occurs while 
data
       // processing.
-      val outputColumns = DataWritingCommand.logicalPlanOutputWithNames(query, 
outputColumnNames)
-      val tableSchema = CharVarcharUtils.getRawSchema(
-        outputColumns.toStructType, sparkSession.sessionState.conf)
-      assert(tableDesc.schema.isEmpty)
-      catalog.createTable(tableDesc.copy(schema = tableSchema), ignoreIfExists 
= false)
+      catalog.createTable(newTableDesc, ignoreIfExists = false)
 
       try {
         // Read back the metadata of the table which was created just now.
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 0cdaa2b41642..8730cb178440 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -1324,6 +1324,53 @@ class MetastoreDataSourcesSuite extends QueryTest
     }
   }
 
+  test("SPARK-60515: saveAsTable with mode Overwrite keeps the existing hive 
table when " +
+      "Hive rejects the new table") {
+    def checkKeepsTable(fileFormat: String, columns: String*): Unit = {
+      withTable("t", "t2") {
+        spark.range(3).write.format("hive").saveAsTable("t")
+        def newWriter = spark.range(5).selectExpr(columns: 
_*).write.format("hive")
+          .option("fileFormat", fileFormat)
+        val e = intercept[AnalysisException] {
+          newWriter.mode(SaveMode.Overwrite).saveAsTable("t")
+        }
+        assert(e.getCondition === "_LEGACY_ERROR_TEMP_3065")
+        // The same error as creating a new table.
+        val expected = 
intercept[AnalysisException](newWriter.saveAsTable("t2"))
+        assert(e.getMessage === expected.getMessage)
+        checkAnswer(table("t"), spark.range(3).toDF())
+      }
+    }
+    // The Hive SerDe can't parse these types.
+    Seq("TIME(6)", "TIMESTAMP_NTZ", "TIMESTAMP_NTZ(9)", "VARIANT").foreach { 
dataType =>
+      checkKeepsTable("textfile", "id", s"CAST(NULL AS $dataType) AS c")
+    }
+    // The Parquet Hive SerDe doesn't support the void type.
+    checkKeepsTable("parquet", "id", "CAST(NULL AS VOID) AS c")
+    // The Hive SerDe can't parse the nested field name.
+    checkKeepsTable("textfile", "id", "CAST(NULL AS STRUCT<`a-b`: INT>) AS c")
+    // The Avro Hive SerDe doesn't allow a space in a field name.
+    checkKeepsTable("avro", "id", "CAST(NULL AS STRUCT<`a b`: INT>) AS c")
+    // Hive column names are case insensitive.
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+      checkKeepsTable("textfile", "id", "id AS ID")
+    }
+  }
+
+  test("SPARK-60515: saveAsTable with mode Overwrite replaces the existing 
hive table with " +
+      "a collated string column") {
+    withTable("t") {
+      spark.range(3).write.format("hive").saveAsTable("t")
+      // Like `createTable`, the checks of Hive on the new table remove the 
collation, which Hive
+      // doesn't support.
+      val df = spark.range(5).selectExpr("id", "CAST(id AS STRING COLLATE 
UNICODE_CI) AS c")
+      df.write.format("hive").option("fileFormat", "parquet")
+        .mode(SaveMode.Overwrite).saveAsTable("t")
+      checkAnswer(table("t"), df)
+      assert(table("t").schema("c").dataType === StringType("UNICODE_CI"))
+    }
+  }
+
   test("insertInto - source and target are the same table") {
     val tableName = "tab1"
     withTable(tableName) {
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
index bb701d80bea0..59959366bdda 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
@@ -24,6 +24,7 @@ import org.apache.hadoop.conf.Configuration
 import org.apache.hadoop.fs.Path
 import org.apache.hadoop.hive.common.StatsSetupConst
 import org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+import org.apache.hadoop.hive.ql.metadata.HiveException
 import org.apache.hadoop.hive.serde2.`lazy`.LazySimpleSerDe
 import org.apache.hadoop.mapred.TextInputFormat
 import org.apache.hadoop.security.UserGroupInformation
@@ -36,7 +37,7 @@ import 
org.apache.spark.sql.catalyst.expressions.{AttributeReference, EqualTo, L
 import org.apache.spark.sql.connector.catalog.TableCatalog
 import org.apache.spark.sql.hive.HiveExternalCatalog
 import org.apache.spark.sql.hive.test.TestHiveVersion
-import org.apache.spark.sql.types.{IntegerType, StructType}
+import org.apache.spark.sql.types.{IntegerType, StructType, VariantType}
 import org.apache.spark.tags.SlowHiveTest
 import org.apache.spark.util.{MutableURLClassLoader, Utils}
 
@@ -227,6 +228,22 @@ class HiveClientSuite(version: String) extends 
HiveVersionSuite(version) {
       ignoreIfExists = false)
   }
 
+  test("validateTable") {
+    val tbl = table("default", tableName = "validated")
+    client.validateTable(tbl)
+    Seq(
+      // Hive column names are case insensitive.
+      new StructType().add("key", "int").add("KEY", "int"),
+      // The Hive SerDe can't parse the type.
+      new StructType().add("key", "int").add("value", VariantType)
+    ).foreach { schema =>
+      val e = intercept[Exception](client.validateTable(tbl.copy(schema = 
schema)))
+      // Compare the class names because the Hive client has its own class 
loader.
+      assert(e.getClass.getName === classOf[HiveException].getName)
+    }
+    assert(!client.tableExists("default", "validated"))
+  }
+
   test("create/alter table with collations") {
     client.createTable(table("default", tableName = "collation_table",
       collation = Some("UNICODE")), ignoreIfExists = false)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to