This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ec9b616b8f89 [SPARK-60709][SQL] Remap target metadata columns after
schema evolution in DSv2 MERGE
ec9b616b8f89 is described below
commit ec9b616b8f894e53d0b49feb412184b9bf3adafd
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 11:49:46 2026 -0700
[SPARK-60709][SQL] Remap target metadata columns after schema evolution in
DSv2 MERGE
### What changes were proposed in this pull request?
This PR makes `ResolveSchemaEvolution` remap references to the metadata
columns of the write target when it replaces the target after evolving the
table schema.
`ResolveSchemaEvolution.replaceWriteTarget` replaces the target with
`relation.copy(table = newTable, output = newOutput)` and rewrites references
to the old output with `rewriteAttrs`. However,
`DataSourceV2Relation.metadataOutput` is a lazy val that is derived from the
table, so the new relation exposes its metadata columns with new expression
IDs, while the MERGE condition, clause conditions and assignments that were
resolved before the evolution still refer to the metadata columns [...]
The new `rewriteMetadataAttrs` rewrites them to the metadata columns of the
new relation. Metadata columns are matched by their logical name
(`MetadataAttributeWithLogicalName`), which stays the same even if a data
column of the new table renames a conflicting metadata column. References to
metadata columns that the new relation does not expose are left unchanged.
This PR also adds
`DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`, which runs
the MERGE schema evolution SQL tests for delta-based operations that split
updates into deletes and inserts, as
`DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite` does for the MERGE tests.
### Why are the changes needed?
`MERGE WITH SCHEMA EVOLUTION` fails when the statement evolves the target
schema and references a metadata column of the target table. For instance, with
the in-memory row-level table of `RowLevelOperationSuiteBase` (`pk INT NOT
NULL, salary INT, dep STRING`, partitioned by `dep`) and a source with an extra
column `active`:
```sql
MERGE WITH SCHEMA EVOLUTION INTO cat.ns1.test_table t
USING source s
ON t.pk = s.pk
WHEN MATCHED AND t._partition = 'hr' THEN UPDATE SET *
```
```
[MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION] Resolved
attribute(s) "_partition" missing from "pk", "salary", "dep", "active",
"_partition", "index", ... in operator MergeRows[...]. Attribute(s) with the
same name appear in the operation: "_partition".
```
The same statement works without `WITH SCHEMA EVOLUTION`, and `MERGE WITH
SCHEMA EVOLUTION` works when no schema change is needed. The failure happens
with group-based and delta-based operations (including split updates and
operations that require no metadata columns), with the SQL and DataFrame APIs,
and wherever the metadata column is referenced: the ON condition, a WHEN
MATCHED or WHEN NOT MATCHED BY SOURCE condition, or an assignment value. It
also happens when the evolution only [...]
### Does this PR introduce _any_ user-facing change?
Yes. `MERGE WITH SCHEMA EVOLUTION` statements that reference metadata
columns of the target table now succeed instead of failing with
`MISSING_ATTRIBUTES`. By code reading, MERGE schema evolution has remapped only
the output of the replaced target since it was added in Apache Spark
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
### How was this patch tested?
Added four tests:
- Three tests to `MergeIntoSchemaEvolutionExtraSourceColumnTests`, where
the source has an extra column: an update condition with `UPDATE SET *`, the
same with `INSERT *`, and the ON and WHEN NOT MATCHED BY SOURCE conditions with
an explicit assignment to the new column.
- One test to `MergeIntoSchemaEvolutionTypeWideningTests`, where the
evolution only widens a column type: an update condition with `UPDATE SET *`.
In this case, the MERGE is resolved before the evolution, so the metadata
columns of the target are already part of the relation output when the target
is replaced.
They run with and without the evolution clause in
`GroupBasedMergeIntoSchemaEvolutionSQLSuite`,
`GroupBasedMergeIntoSchemaEvolutionScalaSuite`,
`DeltaBasedMergeIntoSchemaEvolutionSQLSuite`,
`DeltaBasedMergeIntoSchemaEvolutionScalaSuite` and the new
`DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`. Without
the fix, the tests with the evolution clause fail with `MISSING_ATTRIBUTES` in
all five suites.
```
build/sbt "sql/testOnly *MergeIntoSchemaEvolution*Suite -- -z SPARK-60709"
```
Also ran the DSv2 DELETE, UPDATE and MERGE suites (including the
NoMetadata, split update and NoCodegen variants), the MERGE and INSERT schema
evolution suites, `MetadataColumnSuite` and `PlanResolutionSuite` locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59919 from dongjoon-hyun/SPARK-60709.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 7d63628f95921c1a50f1b90204485cda017ef9d9)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../catalyst/analysis/ResolveSchemaEvolution.scala | 27 +++++++-
.../DeltaBasedMergeIntoSchemaEvolutionSuite.scala | 14 ++++
...IntoSchemaEvolutionExtraSourceColumnTests.scala | 77 ++++++++++++++++++++++
...MergeIntoSchemaEvolutionTypeWideningTests.scala | 17 +++++
4 files changed, 133 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
index db746be2dffc..2f7e9f81129b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
@@ -21,7 +21,7 @@ import scala.util.control.NonFatal
import org.apache.spark.SparkThrowable
import org.apache.spark.internal.LogKeys.{DATA_SOURCE, TABLE_NAME}
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeMap}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeMap,
MetadataAttributeWithLogicalName}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
@@ -100,7 +100,9 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
val newOutput = DataTypeUtils.toAttributes(newTable.columns)
val newRelation = relation.copy(table = newTable, output = newOutput)
val writeWithNewTargetTable = write.withNewTable(newRelation)
- rewriteAttrs(writeWithNewTargetTable, oldOutput, newOutput)
+ val writeWithNewMetadataAttrs =
+ rewriteMetadataAttrs(writeWithNewTargetTable, relation, newRelation)
+ rewriteAttrs(writeWithNewMetadataAttrs, oldOutput, newOutput)
}
private def rewriteAttrs[T <: LogicalPlan](
@@ -111,6 +113,27 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
plan.rewriteAttrs(attrMap).asInstanceOf[T]
}
+ /**
+ * Rewrites references to the metadata columns of the old relation (e.g. in
MERGE conditions)
+ * to the metadata columns of the new relation, which exposes them with new
expression IDs.
+ * Metadata columns are matched by their logical name, which stays the same
even if a data column
+ * of the new table renames a conflicting metadata column. References to
metadata columns that
+ * the new relation does not expose are left unchanged.
+ */
+ private def rewriteMetadataAttrs[T <: LogicalPlan](
+ plan: T,
+ oldRelation: DataSourceV2Relation,
+ newRelation: DataSourceV2Relation): T = {
+ val newMetadataAttrs: Map[String, Attribute] =
newRelation.metadataOutput.collect {
+ case MetadataAttributeWithLogicalName(attr, name) => name -> attr
+ }.toMap
+ val attrMap = AttributeMap(oldRelation.metadataOutput.collect {
+ case MetadataAttributeWithLogicalName(attr, name) if
newMetadataAttrs.contains(name) =>
+ attr -> newMetadataAttrs(name)
+ })
+ plan.rewriteAttrs(attrMap).asInstanceOf[T]
+ }
+
/**
* Computes the set of table changes needed to evolve `targetTable`'s schema
* to accommodate the `originalSource` schema. Only returns schema changes
that are supported by
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
index 27d8eeff6942..56c7bb6f00f5 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
@@ -31,6 +31,20 @@ class DeltaBasedMergeIntoSchemaEvolutionSQLSuite
}
}
+// SQL-based tests for delta-based row-level operations that split updates
into deletes and inserts
+class DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite
+ extends MergeIntoSchemaEvolutionSQLSuiteBase
+ with MergeIntoSchemaEvolutionTests
+ with MergeIntoSchemaEvolutionExtraSQLTests {
+
+ override protected lazy val extraTableProps: java.util.Map[String, String] =
{
+ val props = new java.util.HashMap[String, String]()
+ props.put("supports-deltas", "true")
+ props.put("split-updates", "true")
+ props
+ }
+}
+
// Scala/DataFrame API-based tests for delta-based row-level operations
class DeltaBasedMergeIntoSchemaEvolutionScalaSuite
extends MergeIntoSchemaEvolutionScalaSuiteBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
index 4312e4ffe71d..fb120d6e887a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
@@ -189,6 +189,83 @@ trait MergeIntoSchemaEvolutionExtraSourceColumnTests
extends MergeIntoSchemaEvol
(6, 350, "finance")).toDF("pk", "salary", "dep")
)
+ testEvolution("SPARK-60709: source has extra column with set all columns" +
+ " and target metadata column in update condition")(
+ targetData = Seq(
+ (1, 100, "hr"),
+ (2, 200, "software"),
+ (3, 300, "hr")
+ ).toDF("pk", "salary", "dep"),
+ sourceData = Seq(
+ (1, 150, "hr", true),
+ (2, 250, "software", true),
+ (4, 450, "hr", false)
+ ).toDF("pk", "salary", "dep", "active"),
+ clauses = Seq(
+ updateAll(condition = "t._partition = 'hr'")
+ ),
+ expected = Seq[(Int, Int, String, java.lang.Boolean)](
+ (1, 150, "hr", true),
+ (2, 200, "software", null),
+ (3, 300, "hr", null)).toDF("pk", "salary", "dep", "active"),
+ expectedWithoutEvolution = Seq(
+ (1, 150, "hr"),
+ (2, 200, "software"),
+ (3, 300, "hr")).toDF("pk", "salary", "dep")
+ )
+
+ testEvolution("SPARK-60709: source has extra column with set all columns,
insert all" +
+ " and target metadata column in update condition")(
+ targetData = Seq(
+ (1, 100, "hr"),
+ (2, 200, "software"),
+ (3, 300, "hr")
+ ).toDF("pk", "salary", "dep"),
+ sourceData = Seq(
+ (1, 150, "hr", true),
+ (2, 250, "software", true),
+ (4, 450, "hr", false)
+ ).toDF("pk", "salary", "dep", "active"),
+ clauses = Seq(
+ updateAll(condition = "t._partition = 'hr'"),
+ insertAll()
+ ),
+ expected = Seq[(Int, Int, String, java.lang.Boolean)](
+ (1, 150, "hr", true),
+ (2, 200, "software", null),
+ (3, 300, "hr", null),
+ (4, 450, "hr", false)).toDF("pk", "salary", "dep", "active"),
+ expectedWithoutEvolution = Seq(
+ (1, 150, "hr"),
+ (2, 200, "software"),
+ (3, 300, "hr"),
+ (4, 450, "hr")).toDF("pk", "salary", "dep")
+ )
+
+ testEvolution("SPARK-60709: source has extra column with set explicit
column" +
+ " and target metadata column in merge and not matched by source
conditions")(
+ targetData = Seq(
+ (1, 100, "hr"),
+ (2, 200, "software"),
+ (3, 300, "hr")
+ ).toDF("pk", "salary", "dep"),
+ sourceData = Seq(
+ (1, 150, "hr", true),
+ (2, 250, "software", true),
+ (4, 450, "hr", false)
+ ).toDF("pk", "salary", "dep", "active"),
+ cond = "t.pk = s.pk AND t._partition = 'hr'",
+ clauses = Seq(
+ update(set = "active = s.active"),
+ deleteNotMatched(condition = "t._partition = 'software'")
+ ),
+ expected = Seq[(Int, Int, String, java.lang.Boolean)](
+ (1, 100, "hr", true),
+ (3, 300, "hr", null)).toDF("pk", "salary", "dep", "active"),
+ expectErrorWithoutEvolutionContains = "A column, variable, or function
parameter with name " +
+ "`active` cannot be resolved"
+ )
+
testEvolution("source has extra column - empty table and insert")(
targetData = Seq.empty[(Int, Int, String)].toDF("pk", "salary", "dep"),
sourceData = Seq(
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
index 192dc94dd69d..35df9f6c4e9c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
@@ -132,6 +132,23 @@ trait MergeIntoSchemaEvolutionTypeWideningTests extends
MergeIntoSchemaEvolution
expectErrorWithoutEvolutionContains = "CAST_OVERFLOW_IN_TABLE_INSERT"
)
+ testEvolution("SPARK-60709: type widening top-level field" +
+ " and target metadata column in update condition")(
+ targetData = Seq((1, 100, "hr"), (2, 200, "software"), (3, 300, "hr"))
+ .toDF("pk", "salary", "dep"),
+ sourceData = Seq((1, 150L, "hr"), (2, 250L, "software"), (4, 450L, "hr"))
+ .toDF("pk", "salary", "dep"),
+ clauses = Seq(updateAll(condition = "t._partition = 'hr'")),
+ expected = Seq((1, 150L, "hr"), (2, 200L, "software"), (3, 300L, "hr"))
+ .toDF("pk", "salary", "dep"),
+ expectedSchema = StructType(Seq(
+ StructField("pk", IntegerType, nullable = false),
+ StructField("salary", LongType, nullable = false),
+ StructField("dep", StringType))),
+ expectedWithoutEvolution = Seq((1, 150, "hr"), (2, 200, "software"), (3,
300, "hr"))
+ .toDF("pk", "salary", "dep")
+ )
+
testEvolution("type widening - column not used in assignments")(
targetData = Seq((1, 100, "hr"), (2, 200, "software")).toDF("pk",
"salary", "dep"),
sourceData = Seq((2, Long.MaxValue, "software"), (3, Long.MaxValue,
"engineering"))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]