This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new cd80bea5a669 [SPARK-60515][SQL] Keep the existing table when Hive
rejects the new table of `saveAsTable` overwrite
cd80bea5a669 is described below
commit cd80bea5a6699c6cc72f4fbf05f0299d0c79b7ec
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Oct 25 07:12:26 2026 -0700
[SPARK-60515][SQL] Keep the existing table when Hive rejects the new table
of `saveAsTable` overwrite
### What changes were proposed in this pull request?
This PR makes `CreateHiveTableAsSelectCommand` run the checks of Hive on
the new table before it drops the existing table for `saveAsTable` with
`SaveMode.Overwrite` and `format("hive")`.
- `HiveClient.validateTable` runs the checks that `Hive.createTable` runs
before calling Hive metastore, i.e., the SerDe initialization with the table
columns and `Table.checkValidity`, without creating the table. Like
`Hive.createTable`, it wraps the errors in `HiveException`.
- `HiveExternalCatalog.validateHiveSerdeTable` calls it after removing the
collations, like `HiveExternalCatalog.createTable`.
- `CreateHiveTableAsSelectCommand` calls it before dropping the existing
table.
### Why are the changes needed?
SPARK-60387 made `saveAsTable` with `SaveMode.Overwrite` drop the existing
table as late as it can. `CreateHiveTableAsSelectCommand` drops it after the
analysis, because it creates the new table before writing. When Hive rejects
the new table there, the existing table, and the data of a managed table, are
already gone.
```scala
sql("CREATE TABLE t (i INT) USING HIVE")
sql("INSERT INTO t VALUES (1)")
sql("SELECT 1 AS i, CAST(NULL AS VARIANT) AS
c").write.format("hive").mode("overwrite").saveAsTable("t")
// AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.IllegalArgumentException: Error: type expected at the position 4 of
'int:variant' but 'variant' is found.
spark.table("t")
// AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view `t`
cannot be found. Verify the spelling and correctness of the schema and catalog.
...
```
Hive rejects, for example:
- data column types that the Hive SerDe can't parse: `TIME`,
`TIMESTAMP_NTZ`, `TIMESTAMP_NTZ(9)`, `VARIANT`, `GEOMETRY`, `GEOGRAPHY`, and
nested ANSI intervals
- the void type in Parquet and ORC tables
- nested field names like `` `a-b` ``, and `` `a b` `` in Avro tables
- column names that differ only in case when `spark.sql.caseSensitive` is
`true`
This has been the behavior since Apache Spark 2.2.0 (SPARK-19151).
### Does this PR introduce _any_ user-facing change?
Yes. When Hive rejects the new table before calling Hive metastore, such an
overwrite keeps the existing table and its data. The error is the same as
before. Valid overwrites are unchanged. A rejection by Hive metastore itself,
e.g., for a `path` that points to a file, or a failure after creating the new
table, e.g., in the write, still loses the existing table.
### How was this patch tested?
Added tests to `MetastoreDataSourcesSuite` and `HiveClientSuite`. The test
of the rejected overwrites in `MetastoreDataSourcesSuite` fails without this
fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59757 from dongjoon-hyun/SPARK-60515.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/hive/HiveExternalCatalog.scala | 10 +++++
.../apache/spark/sql/hive/client/HiveClient.scala | 6 +++
.../spark/sql/hive/client/HiveClientImpl.scala | 15 +++++++
.../execution/CreateHiveTableAsSelectCommand.scala | 19 +++++----
.../spark/sql/hive/MetastoreDataSourcesSuite.scala | 47 ++++++++++++++++++++++
.../spark/sql/hive/client/HiveClientSuite.scala | 19 ++++++++-
6 files changed, 108 insertions(+), 8 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
index 02dd55f15e15..e7997c7e862f 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala
@@ -306,6 +306,16 @@ private[spark] class HiveExternalCatalog(conf: SparkConf,
hadoopConf: Configurat
}
}
+ /**
+ * Runs the checks that Hive runs on a new Hive serde table before creating
it in Hive
+ * metastore, e.g., the SerDe initialization with the table columns, without
creating it.
+ * A command that replaces an existing table calls this before dropping the
existing table.
+ */
+ private[hive] def validateHiveSerdeTable(tableDefinition: CatalogTable):
Unit = withClient {
+ // Like `createTable`, remove the collations, which Hive doesn't support.
+ client.validateTable(tableDefinition.copy(schema =
removeCollation(tableDefinition.schema)))
+ }
+
private def createDataSourceTable(table: CatalogTable, ignoreIfExists:
Boolean): Unit = {
// data source table always have a provider, it's guaranteed by
`DDLUtils.isDatasourceTable`.
val provider = table.provider.get
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
index ad7648096354..7503d7fdcd14 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
@@ -132,6 +132,12 @@ private[hive] trait HiveClient {
/** Creates a table with the given metadata. */
def createTable(table: CatalogTable, ignoreIfExists: Boolean): Unit
+ /**
+ * Runs the checks that [[createTable]] runs on the given table before
calling Hive metastore,
+ * e.g., the SerDe initialization with the table columns, without creating
the table.
+ */
+ def validateTable(table: CatalogTable): Unit
+
/** Drop the specified table. */
def dropTable(dbName: String, tableName: String, ignoreIfNotExists: Boolean,
purge: Boolean): Unit
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index ddae3785e304..0c94dbbad9a8 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -589,6 +589,21 @@ private[hive] class HiveClientImpl(
shim.createTable(client, toHiveTable(table, Some(userName)),
ignoreIfExists)
}
+ override def validateTable(table: CatalogTable): Unit = withHiveState {
+ val hiveTable = toHiveTable(table, Some(userName))
+ // These are the checks of `Hive.createTable` before it calls Hive
metastore. Like it, wrap
+ // their errors in `HiveException`.
+ try {
+ if (hiveTable.getCols.isEmpty || hiveTable.getSd.getColsSize == 0) {
+ // `Hive.createTable` gets the columns from the SerDe here, e.g., for
an Avro table.
+ hiveTable.getDeserializer
+ }
+ hiveTable.checkValidity(conf)
+ } catch {
+ case NonFatal(e) => throw new HiveException(e)
+ }
+ }
+
override def dropTable(
dbName: String,
tableName: String,
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
index c0f3bab3143b..280978b8543e 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateHiveTableAsSelectCommand.scala
@@ -25,6 +25,7 @@ import
org.apache.spark.sql.catalyst.plans.logical.{CTEInChildren, CTERelationDe
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.command.{DataWritingCommand,
LeafRunnableCommand}
+import org.apache.spark.sql.hive.HiveExternalCatalog
import org.apache.spark.sql.hive.execution.InsertIntoHiveTable.BY_CTAS
/**
@@ -65,20 +66,24 @@ case class CreateHiveTableAsSelectCommand(
tableDesc.storage.locationUri.foreach { p =>
DataWritingCommand.assertEmptyRootPath(p, mode,
sparkSession.sessionState.newHadoopConf())
}
+ val outputColumns = DataWritingCommand.logicalPlanOutputWithNames(query,
outputColumnNames)
+ val tableSchema = CharVarcharUtils.getRawSchema(
+ outputColumns.toStructType, sparkSession.sessionState.conf)
+ assert(tableDesc.schema.isEmpty)
+ val newTableDesc = tableDesc.copy(schema = tableSchema)
if (tableExists) {
// `SaveAsV1TableCommand` runs this command with `SaveMode.Overwrite`
to replace the
- // table. Drop it here, after the analysis, because the new table is
created before the
- // write.
+ // table. Drop it here, after the analysis and the checks of the new
table, because the
+ // new table is created before the write.
+
sparkSession.sharedState.externalCatalog.unwrapped.asInstanceOf[HiveExternalCatalog]
+ .validateHiveSerdeTable(
+ newTableDesc.copy(identifier =
catalog.qualifyIdentifier(tableIdentifier)))
catalog.dropTable(tableIdentifier, ignoreIfNotExists = true, purge =
false)
}
// TODO ideally, we should get the output data ready first and then
// add the relation into catalog, just in case of failure occurs while
data
// processing.
- val outputColumns = DataWritingCommand.logicalPlanOutputWithNames(query,
outputColumnNames)
- val tableSchema = CharVarcharUtils.getRawSchema(
- outputColumns.toStructType, sparkSession.sessionState.conf)
- assert(tableDesc.schema.isEmpty)
- catalog.createTable(tableDesc.copy(schema = tableSchema), ignoreIfExists
= false)
+ catalog.createTable(newTableDesc, ignoreIfExists = false)
try {
// Read back the metadata of the table which was created just now.
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
index 0cdaa2b41642..8730cb178440 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
@@ -1324,6 +1324,53 @@ class MetastoreDataSourcesSuite extends QueryTest
}
}
+ test("SPARK-60515: saveAsTable with mode Overwrite keeps the existing hive
table when " +
+ "Hive rejects the new table") {
+ def checkKeepsTable(fileFormat: String, columns: String*): Unit = {
+ withTable("t", "t2") {
+ spark.range(3).write.format("hive").saveAsTable("t")
+ def newWriter = spark.range(5).selectExpr(columns:
_*).write.format("hive")
+ .option("fileFormat", fileFormat)
+ val e = intercept[AnalysisException] {
+ newWriter.mode(SaveMode.Overwrite).saveAsTable("t")
+ }
+ assert(e.getCondition === "_LEGACY_ERROR_TEMP_3065")
+ // The same error as creating a new table.
+ val expected =
intercept[AnalysisException](newWriter.saveAsTable("t2"))
+ assert(e.getMessage === expected.getMessage)
+ checkAnswer(table("t"), spark.range(3).toDF())
+ }
+ }
+ // The Hive SerDe can't parse these types.
+ Seq("TIME(6)", "TIMESTAMP_NTZ", "TIMESTAMP_NTZ(9)", "VARIANT").foreach {
dataType =>
+ checkKeepsTable("textfile", "id", s"CAST(NULL AS $dataType) AS c")
+ }
+ // The Parquet Hive SerDe doesn't support the void type.
+ checkKeepsTable("parquet", "id", "CAST(NULL AS VOID) AS c")
+ // The Hive SerDe can't parse the nested field name.
+ checkKeepsTable("textfile", "id", "CAST(NULL AS STRUCT<`a-b`: INT>) AS c")
+ // The Avro Hive SerDe doesn't allow a space in a field name.
+ checkKeepsTable("avro", "id", "CAST(NULL AS STRUCT<`a b`: INT>) AS c")
+ // Hive column names are case insensitive.
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+ checkKeepsTable("textfile", "id", "id AS ID")
+ }
+ }
+
+ test("SPARK-60515: saveAsTable with mode Overwrite replaces the existing
hive table with " +
+ "a collated string column") {
+ withTable("t") {
+ spark.range(3).write.format("hive").saveAsTable("t")
+ // Like `createTable`, the checks of Hive on the new table remove the
collation, which Hive
+ // doesn't support.
+ val df = spark.range(5).selectExpr("id", "CAST(id AS STRING COLLATE
UNICODE_CI) AS c")
+ df.write.format("hive").option("fileFormat", "parquet")
+ .mode(SaveMode.Overwrite).saveAsTable("t")
+ checkAnswer(table("t"), df)
+ assert(table("t").schema("c").dataType === StringType("UNICODE_CI"))
+ }
+ }
+
test("insertInto - source and target are the same table") {
val tableName = "tab1"
withTable(tableName) {
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
index bb701d80bea0..59959366bdda 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
@@ -24,6 +24,7 @@ import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.hive.common.StatsSetupConst
import org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+import org.apache.hadoop.hive.ql.metadata.HiveException
import org.apache.hadoop.hive.serde2.`lazy`.LazySimpleSerDe
import org.apache.hadoop.mapred.TextInputFormat
import org.apache.hadoop.security.UserGroupInformation
@@ -36,7 +37,7 @@ import
org.apache.spark.sql.catalyst.expressions.{AttributeReference, EqualTo, L
import org.apache.spark.sql.connector.catalog.TableCatalog
import org.apache.spark.sql.hive.HiveExternalCatalog
import org.apache.spark.sql.hive.test.TestHiveVersion
-import org.apache.spark.sql.types.{IntegerType, StructType}
+import org.apache.spark.sql.types.{IntegerType, StructType, VariantType}
import org.apache.spark.tags.SlowHiveTest
import org.apache.spark.util.{MutableURLClassLoader, Utils}
@@ -227,6 +228,22 @@ class HiveClientSuite(version: String) extends
HiveVersionSuite(version) {
ignoreIfExists = false)
}
+ test("validateTable") {
+ val tbl = table("default", tableName = "validated")
+ client.validateTable(tbl)
+ Seq(
+ // Hive column names are case insensitive.
+ new StructType().add("key", "int").add("KEY", "int"),
+ // The Hive SerDe can't parse the type.
+ new StructType().add("key", "int").add("value", VariantType)
+ ).foreach { schema =>
+ val e = intercept[Exception](client.validateTable(tbl.copy(schema =
schema)))
+ // Compare the class names because the Hive client has its own class
loader.
+ assert(e.getClass.getName === classOf[HiveException].getName)
+ }
+ assert(!client.tableExists("default", "validated"))
+ }
+
test("create/alter table with collations") {
client.createTable(table("default", tableName = "collation_table",
collation = Some("UNICODE")), ignoreIfExists = false)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]