This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0f10db1231de [SPARK-60413][SQL] Warn when evolving a DSv2 table that
accepts any schema
0f10db1231de is described below
commit 0f10db1231dea58eaedb29a6a0cdaaaaf4ea748c
Author: Szehon Ho <[email protected]>
AuthorDate: Fri Oct 23 11:38:40 2026 -0700
[SPARK-60413][SQL] Warn when evolving a DSv2 table that accepts any schema
### What changes were proposed in this pull request?
JIRA: [SPARK-60413](https://issues.apache.org/jira/browse/SPARK-60413)
Log a warning before `ResolveSchemaEvolution` applies pending schema
changes to a DSv2 table reporting both `AUTOMATIC_SCHEMA_EVOLUTION` and
`ACCEPT_ANY_SCHEMA`.
The warning is emitted once for the batch of schema changes. Writes without
pending evolution and tables reporting only one capability remain quiet.
### Why are the changes needed?
`AUTOMATIC_SCHEMA_EVOLUTION` applies requested schema changes during write
analysis, while `ACCEPT_ANY_SCHEMA` skips native write schema resolution and
alignment. A write can therefore fail after updating the table schema. A
nontransactional catalog retains those changes even when the write fails.
The interaction also arose in [Iceberg PR
#17957](https://github.com/apache/iceberg/pull/17957). The warning identifies
the affected data source and table and suggests disabling `ACCEPT_ANY_SCHEMA`.
### Does this PR introduce _any_ user-facing change?
Yes. When applying schema evolution to a table reporting both capabilities,
Spark logs:
```text
Applying schema evolution to <data source> table <table>, which reports both
AUTOMATIC_SCHEMA_EVOLUTION and ACCEPT_ANY_SCHEMA capabilities. While
AUTOMATIC_SCHEMA_EVOLUTION natively resolves and evolves the schema on a
write
schema evolution request, ACCEPT_ANY_SCHEMA skips native write schema
resolution
and alignment. This may cause the write to fail after the table schema has
been
updated. Consider disabling ACCEPT_ANY_SCHEMA.
```
### How was this patch tested?
Added an end-to-end test in `MergeIntoDataFrameSuite` using a
nontransactional row-level catalog. It covers INSERT and MERGE, all four
capability combinations, and schema evolution enabled and disabled: 16
configurations and 64 public write calls.
The test checks exact warning messages and counts, including multiple
column additions in one write, later evolution requests, and repeated writes
needing no further changes. It also verifies persisted rows and schemas,
including schema changes retained after failed MERGE.
The focused test passed:
```sh
SPARK_LOCAL_IP=127.0.0.1 build/sbt 'sql/testOnly
org.apache.spark.sql.connector.MergeIntoDataFrameSuite -- -z "schema evolution
warns"'
```
Whitespace, ASCII, and line-length checks also passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59660 from szehon-ho/codex/warn-conflicting-schema-capabilities.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit 0aa4f9136bccc40856a5af688e3d16f8b38ebfc4)
Signed-off-by: Szehon Ho <[email protected]>
---
.../catalyst/analysis/ResolveSchemaEvolution.scala | 12 ++
.../sql/connector/MergeIntoDataFrameSuite.scala | 197 ++++++++++++++++++++-
2 files changed, 205 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
index 08237128358a..db746be2dffc 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.SparkThrowable
+import org.apache.spark.internal.LogKeys.{DATA_SOURCE, TABLE_NAME}
import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeMap}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
@@ -49,6 +50,17 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
case write: WriteWithSchemaEvolution if
write.pendingSchemaChanges.nonEmpty =>
write.table match {
case relation @ ExtractV2CatalogAndIdentifier(catalog, ident) =>
+ if (relation.skipSchemaResolution && relation.autoSchemaEvolution) {
+ val dataSource = relation.table.properties().getOrDefault(
+ TableCatalog.PROP_PROVIDER, relation.table.getClass.getName)
+ logWarning(log"Applying schema evolution to ${MDC(DATA_SOURCE,
dataSource)} table " +
+ log"${MDC(TABLE_NAME, relation.name)}, which reports both " +
+ log"AUTOMATIC_SCHEMA_EVOLUTION and ACCEPT_ANY_SCHEMA
capabilities. While " +
+ log"AUTOMATIC_SCHEMA_EVOLUTION natively resolves and evolves the
schema on a write " +
+ log"schema evolution request, ACCEPT_ANY_SCHEMA skips native
write schema " +
+ log"resolution and alignment. This may cause the write to fail
after the table " +
+ log"schema has been updated. Consider disabling
ACCEPT_ANY_SCHEMA.")
+ }
evolveSchema(catalog, ident, write.pendingSchemaChanges)
val newTable = CatalogV2Util.loadTableForV2Write(
catalog, ident, write.writePrivileges, relation.options)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala
index 9c471031eeb9..a627b33d2e45 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoDataFrameSuite.scala
@@ -17,21 +17,33 @@
package org.apache.spark.sql.connector
+import java.util
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.logging.log4j.Level
+
import org.apache.spark.SparkConf
-import org.apache.spark.sql.{sources, Column, Row}
+import org.apache.spark.sql.{sources, Column, DataFrame, Row}
import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.ResolveSchemaEvolution
import org.apache.spark.sql.classic.MergeIntoWriter
-import org.apache.spark.sql.connector.catalog.{Aborted, Committed,
InMemoryBaseTable, InMemoryRowLevelOperationTableCatalog}
+import org.apache.spark.sql.connector.catalog.{Aborted, Column => V2Column,
Committed, InMemoryBaseTable, InMemoryRowLevelOperationTable,
InMemoryRowLevelOperationTableCatalog, InMemoryTableCatalog, TableCapability}
import org.apache.spark.sql.connector.catalog.Identifier
+import
org.apache.spark.sql.connector.catalog.TableCapability.{ACCEPT_ANY_SCHEMA,
AUTOMATIC_SCHEMA_EVOLUTION}
import org.apache.spark.sql.connector.catalog.TableInfo
+import org.apache.spark.sql.connector.catalog.constraints.Constraint
+import org.apache.spark.sql.connector.distributions.Distribution
+import org.apache.spark.sql.connector.expressions.{SortOrder, Transform}
import org.apache.spark.sql.functions._
-import org.apache.spark.sql.types.IntegerType
-import org.apache.spark.sql.types.StringType
+import org.apache.spark.sql.internal.connector.SchemaAlignmentConfig
+import org.apache.spark.sql.types.{BooleanType, IntegerType, LongType,
StringType, StructType}
class MergeIntoDataFrameSuite extends RowLevelOperationSuiteBase {
override protected def sparkConf: SparkConf = super.sparkConf
.set(InMemoryBaseTable.ASSIGN_COLUMN_IDS, "true")
+ .set("spark.sql.catalog.non_txn",
classOf[NonTransactionalRowLevelOperationCatalog].getName)
import testImplicits._
@@ -1188,6 +1200,76 @@ class MergeIntoDataFrameSuite extends
RowLevelOperationSuiteBase {
}
}
+ test("schema evolution warns once per write only when both capabilities are
present") {
+ val tableName = "non_txn.ns1.test_table"
+ for {
+ command <- Seq("INSERT", "MERGE")
+ (description, capabilities) <- Seq(
+ "neither capability" -> Set.empty[TableCapability],
+ "only ACCEPT_ANY_SCHEMA" -> Set(ACCEPT_ANY_SCHEMA),
+ "only AUTOMATIC_SCHEMA_EVOLUTION" -> Set(AUTOMATIC_SCHEMA_EVOLUTION),
+ "both capabilities" -> Set(AUTOMATIC_SCHEMA_EVOLUTION,
ACCEPT_ANY_SCHEMA))
+ evolutionEnabled <- Seq(false, true)
+ } {
+ withClue(s"$command with $description, schema
evolution=$evolutionEnabled: ") {
+ withTable(tableName) {
+ val acceptAnySchema = capabilities.contains(ACCEPT_ANY_SCHEMA)
+ val autoSchemaEvolution =
capabilities.contains(AUTOMATIC_SCHEMA_EVOLUTION)
+ val warnOnEvolution = evolutionEnabled && acceptAnySchema &&
autoSchemaEvolution
+ sql(
+ s"""CREATE TABLE $tableName (pk INT) USING foo
+ |TBLPROPERTIES (
+ | 'accept-any-schema' = '$acceptAnySchema',
+ | 'auto-schema-evolution' = '$autoSchemaEvolution')
+ |""".stripMargin)
+ val initial = spark.createDataFrame(
+ Seq(Row(0)).asJava,
+ new StructType().add("pk", IntegerType))
+ val first = spark.createDataFrame(
+ Seq(Row(1, "a", true)).asJava,
+ initial.schema.add("data", StringType).add("active", BooleanType))
+ val next = spark.createDataFrame(
+ Seq(Row(2, "b", false, 2L)).asJava,
+ first.schema.add("score", LongType))
+
+ // A write matching the table schema does not warn.
+ checkSchemaEvolutionWriteWithWarning(
+ tableName, command, initial, evolutionEnabled, capabilities,
+ expectWarning = false)
+
+ // Adding multiple columns emits at most one warning for the entire
write.
+ checkSchemaEvolutionWriteWithWarning(
+ tableName, command, first, evolutionEnabled, capabilities,
+ expectWarning = warnOnEvolution)
+
+ // A later request adding another column gets its own warning when
applicable.
+ checkSchemaEvolutionWriteWithWarning(
+ tableName, command, next, evolutionEnabled, capabilities,
+ expectWarning = warnOnEvolution)
+
+ // Repeating the write does not warn.
+ checkSchemaEvolutionWriteWithWarning(
+ tableName, command, next, evolutionEnabled, capabilities,
expectWarning = false)
+
+ val acceptsWrite = command == "INSERT" && acceptAnySchema
+ val evolvesAndWrites = evolutionEnabled && autoSchemaEvolution &&
!acceptAnySchema
+ val expectedRows = if (acceptsWrite || evolvesAndWrites) {
+ Seq(
+ Row(0, null, null, null),
+ Row(1, "a", true, null),
+ Row(2, "b", false, 2L),
+ Row(2, "b", false, 2L))
+ } else if (command == "MERGE" && acceptAnySchema) {
+ Nil
+ } else {
+ Seq(Row(0))
+ }
+ checkAnswer(spark.table(tableName), expectedRows)
+ }
+ }
+ }
+ }
+
test("SPARK-54444: any schema changes after analysis are prohibited") {
import org.apache.spark.sql.connector.catalog.Column
val sourceTable = "cat.ns1.source_table"
@@ -1551,4 +1633,111 @@ class MergeIntoDataFrameSuite extends
RowLevelOperationSuiteBase {
}
}
}
+
+ // Test helpers.
+
+ private def checkSchemaEvolutionWriteWithWarning(
+ tableName: String,
+ command: String,
+ source: DataFrame,
+ evolutionEnabled: Boolean,
+ capabilities: Set[TableCapability],
+ expectWarning: Boolean): Unit = {
+ val originalSchema = spark.table(tableName).schema
+ val sourceSchema = source.schema.asNullable.toDDL
+ val schemaMatches = originalSchema.asNullable.toDDL == sourceSchema
+ val acceptAnySchema = capabilities.contains(ACCEPT_ANY_SCHEMA)
+ val evolvesSchema = evolutionEnabled &&
+ capabilities.contains(AUTOMATIC_SCHEMA_EVOLUTION) && !schemaMatches
+ val writeSucceeds = if (command == "MERGE" && acceptAnySchema) {
+ false
+ } else {
+ schemaMatches || evolvesSchema || acceptAnySchema
+ }
+ val expectedWarnings = if (expectWarning) {
+ Seq(schemaEvolutionWarning(tableName))
+ } else {
+ Nil
+ }
+ val appender = new LogAppender("schema evolution warning")
+ withLogAppender(
+ appender,
+ loggerNames =
Seq(ResolveSchemaEvolution.getClass.getName.stripSuffix("$")),
+ level = Some(Level.WARN)) {
+ if (writeSucceeds) {
+ writeWithSchemaEvolution(tableName, command, source, evolutionEnabled)
+ assert(spark.table(tableName).schema.asNullable.toDDL == sourceSchema)
+ } else {
+ val error = intercept[AnalysisException] {
+ writeWithSchemaEvolution(tableName, command, source,
evolutionEnabled)
+ }
+ val expectedError = if (command == "INSERT") {
+ "INSERT_COLUMN_ARITY_MISMATCH.TOO_MANY_DATA_COLUMNS"
+ } else {
+ "UNRESOLVED_COLUMN.WITH_SUGGESTION"
+ }
+ assert(error.getCondition == expectedError)
+ val expectedSchema = if (evolvesSchema) sourceSchema else
originalSchema.toDDL
+ assert(spark.table(tableName).schema.toDDL == expectedSchema)
+ }
+ }
+ val warnings = appender.loggingEvents.filter(_.getLevel == Level.WARN)
+ .map(_.getMessage.getFormattedMessage).toSeq
+ assert(warnings == expectedWarnings)
+ }
+
+ private def writeWithSchemaEvolution(
+ tableName: String,
+ command: String,
+ source: DataFrame,
+ evolutionEnabled: Boolean): Unit = {
+ if (command == "INSERT") {
+ val writer = source.write.mode("append")
+ if (evolutionEnabled) {
+ writer.withSchemaEvolution()
+ }
+ writer.insertInto(tableName)
+ } else {
+ // DataFrame columns supply resolved source values; Spark must still
resolve target keys.
+ val assignments = source.columns.map(name => name -> source(name)).toMap
+ val writer = source.mergeInto(tableName, lit(false))
+ .whenNotMatched().insert(assignments)
+ if (evolutionEnabled) {
+ writer.withSchemaEvolution()
+ }
+ writer.merge()
+ }
+ }
+
+ private def schemaEvolutionWarning(tableName: String): String = {
+ s"Applying schema evolution to foo table $tableName, which reports both " +
+ "AUTOMATIC_SCHEMA_EVOLUTION and ACCEPT_ANY_SCHEMA capabilities. While " +
+ "AUTOMATIC_SCHEMA_EVOLUTION natively resolves and evolves the schema on
a write " +
+ "schema evolution request, ACCEPT_ANY_SCHEMA skips native write schema "
+
+ "resolution and alignment. This may cause the write to fail after the
table " +
+ "schema has been updated. Consider disabling ACCEPT_ANY_SCHEMA."
+ }
+}
+
+class NonTransactionalRowLevelOperationCatalog extends InMemoryTableCatalog {
+
+ // scalastyle:off argcount
+ override protected def newInMemoryTable(
+ name: String,
+ columns: Array[V2Column],
+ partitioning: Array[Transform],
+ properties: util.Map[String, String],
+ constraints: Array[Constraint],
+ distribution: Distribution,
+ ordering: Array[SortOrder],
+ requiredNumPartitions: Option[Int],
+ advisoryPartitionSize: Option[Long],
+ distributionStrictlyRequired: Boolean,
+ numRowsPerSplit: Int,
+ id: String,
+ schemaAlignmentConfig: SchemaAlignmentConfig): InMemoryBaseTable = {
+ // scalastyle:on argcount
+ InMemoryRowLevelOperationTable.withColumns(
+ name, columns, partitioning, properties, constraints, id,
schemaAlignmentConfig)
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]