This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 7d63628f9592 [SPARK-60709][SQL] Remap target metadata columns after 
schema evolution in DSv2 MERGE
7d63628f9592 is described below

commit 7d63628f95921c1a50f1b90204485cda017ef9d9
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 11:49:46 2026 -0700

    [SPARK-60709][SQL] Remap target metadata columns after schema evolution in 
DSv2 MERGE
    
    ### What changes were proposed in this pull request?
    
    This PR makes `ResolveSchemaEvolution` remap references to the metadata 
columns of the write target when it replaces the target after evolving the 
table schema.
    
    `ResolveSchemaEvolution.replaceWriteTarget` replaces the target with 
`relation.copy(table = newTable, output = newOutput)` and rewrites references 
to the old output with `rewriteAttrs`. However, 
`DataSourceV2Relation.metadataOutput` is a lazy val that is derived from the 
table, so the new relation exposes its metadata columns with new expression 
IDs, while the MERGE condition, clause conditions and assignments that were 
resolved before the evolution still refer to the metadata columns [...]
    
    The new `rewriteMetadataAttrs` rewrites them to the metadata columns of the 
new relation. Metadata columns are matched by their logical name 
(`MetadataAttributeWithLogicalName`), which stays the same even if a data 
column of the new table renames a conflicting metadata column. References to 
metadata columns that the new relation does not expose are left unchanged.
    
    This PR also adds 
`DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`, which runs 
the MERGE schema evolution SQL tests for delta-based operations that split 
updates into deletes and inserts, as 
`DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite` does for the MERGE tests.
    
    ### Why are the changes needed?
    
    `MERGE WITH SCHEMA EVOLUTION` fails when the statement evolves the target 
schema and references a metadata column of the target table. For instance, with 
the in-memory row-level table of `RowLevelOperationSuiteBase` (`pk INT NOT 
NULL, salary INT, dep STRING`, partitioned by `dep`) and a source with an extra 
column `active`:
    
    ```sql
    MERGE WITH SCHEMA EVOLUTION INTO cat.ns1.test_table t
    USING source s
    ON t.pk = s.pk
    WHEN MATCHED AND t._partition = 'hr' THEN UPDATE SET *
    ```
    
    ```
    [MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_APPEAR_IN_OPERATION] Resolved 
attribute(s) "_partition" missing from "pk", "salary", "dep", "active", 
"_partition", "index", ... in operator MergeRows[...]. Attribute(s) with the 
same name appear in the operation: "_partition".
    ```
    
    The same statement works without `WITH SCHEMA EVOLUTION`, and `MERGE WITH 
SCHEMA EVOLUTION` works when no schema change is needed. The failure happens 
with group-based and delta-based operations (including split updates and 
operations that require no metadata columns), with the SQL and DataFrame APIs, 
and wherever the metadata column is referenced: the ON condition, a WHEN 
MATCHED or WHEN NOT MATCHED BY SOURCE condition, or an assignment value. It 
also happens when the evolution only  [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `MERGE WITH SCHEMA EVOLUTION` statements that reference metadata 
columns of the target table now succeed instead of failing with 
`MISSING_ATTRIBUTES`. By code reading, MERGE schema evolution has remapped only 
the output of the replaced target since it was added in Apache Spark 
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
    
    ### How was this patch tested?
    
    Added four tests:
    - Three tests to `MergeIntoSchemaEvolutionExtraSourceColumnTests`, where 
the source has an extra column: an update condition with `UPDATE SET *`, the 
same with `INSERT *`, and the ON and WHEN NOT MATCHED BY SOURCE conditions with 
an explicit assignment to the new column.
    - One test to `MergeIntoSchemaEvolutionTypeWideningTests`, where the 
evolution only widens a column type: an update condition with `UPDATE SET *`. 
In this case, the MERGE is resolved before the evolution, so the metadata 
columns of the target are already part of the relation output when the target 
is replaced.
    
    They run with and without the evolution clause in 
`GroupBasedMergeIntoSchemaEvolutionSQLSuite`, 
`GroupBasedMergeIntoSchemaEvolutionScalaSuite`, 
`DeltaBasedMergeIntoSchemaEvolutionSQLSuite`, 
`DeltaBasedMergeIntoSchemaEvolutionScalaSuite` and the new 
`DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite`. Without 
the fix, the tests with the evolution clause fail with `MISSING_ATTRIBUTES` in 
all five suites.
    
    ```
    build/sbt "sql/testOnly *MergeIntoSchemaEvolution*Suite -- -z SPARK-60709"
    ```
    
    Also ran the DSv2 DELETE, UPDATE and MERGE suites (including the 
NoMetadata, split update and NoCodegen variants), the MERGE and INSERT schema 
evolution suites, `MetadataColumnSuite` and `PlanResolutionSuite` locally.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59919 from dongjoon-hyun/SPARK-60709.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../catalyst/analysis/ResolveSchemaEvolution.scala | 27 +++++++-
 .../DeltaBasedMergeIntoSchemaEvolutionSuite.scala  | 14 ++++
 ...IntoSchemaEvolutionExtraSourceColumnTests.scala | 77 ++++++++++++++++++++++
 ...MergeIntoSchemaEvolutionTypeWideningTests.scala | 17 +++++
 4 files changed, 133 insertions(+), 2 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
index db746be2dffc..2f7e9f81129b 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSchemaEvolution.scala
@@ -21,7 +21,7 @@ import scala.util.control.NonFatal
 
 import org.apache.spark.SparkThrowable
 import org.apache.spark.internal.LogKeys.{DATA_SOURCE, TABLE_NAME}
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeMap}
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeMap, 
MetadataAttributeWithLogicalName}
 import org.apache.spark.sql.catalyst.plans.logical._
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.TreePattern.COMMAND
@@ -100,7 +100,9 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
     val newOutput = DataTypeUtils.toAttributes(newTable.columns)
     val newRelation = relation.copy(table = newTable, output = newOutput)
     val writeWithNewTargetTable = write.withNewTable(newRelation)
-    rewriteAttrs(writeWithNewTargetTable, oldOutput, newOutput)
+    val writeWithNewMetadataAttrs =
+      rewriteMetadataAttrs(writeWithNewTargetTable, relation, newRelation)
+    rewriteAttrs(writeWithNewMetadataAttrs, oldOutput, newOutput)
   }
 
   private def rewriteAttrs[T <: LogicalPlan](
@@ -111,6 +113,27 @@ object ResolveSchemaEvolution extends Rule[LogicalPlan] {
     plan.rewriteAttrs(attrMap).asInstanceOf[T]
   }
 
+  /**
+   * Rewrites references to the metadata columns of the old relation (e.g. in 
MERGE conditions)
+   * to the metadata columns of the new relation, which exposes them with new 
expression IDs.
+   * Metadata columns are matched by their logical name, which stays the same 
even if a data column
+   * of the new table renames a conflicting metadata column. References to 
metadata columns that
+   * the new relation does not expose are left unchanged.
+   */
+  private def rewriteMetadataAttrs[T <: LogicalPlan](
+      plan: T,
+      oldRelation: DataSourceV2Relation,
+      newRelation: DataSourceV2Relation): T = {
+    val newMetadataAttrs: Map[String, Attribute] = 
newRelation.metadataOutput.collect {
+      case MetadataAttributeWithLogicalName(attr, name) => name -> attr
+    }.toMap
+    val attrMap = AttributeMap(oldRelation.metadataOutput.collect {
+      case MetadataAttributeWithLogicalName(attr, name) if 
newMetadataAttrs.contains(name) =>
+        attr -> newMetadataAttrs(name)
+    })
+    plan.rewriteAttrs(attrMap).asInstanceOf[T]
+  }
+
   /**
    * Computes the set of table changes needed to evolve `targetTable`'s schema
    * to accommodate the `originalSource` schema. Only returns schema changes 
that are supported by
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
index 27d8eeff6942..56c7bb6f00f5 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoSchemaEvolutionSuite.scala
@@ -31,6 +31,20 @@ class DeltaBasedMergeIntoSchemaEvolutionSQLSuite
   }
 }
 
+// SQL-based tests for delta-based row-level operations that split updates 
into deletes and inserts
+class DeltaBasedMergeIntoSchemaEvolutionUpdateAsDeleteAndInsertSQLSuite
+  extends MergeIntoSchemaEvolutionSQLSuiteBase
+  with MergeIntoSchemaEvolutionTests
+  with MergeIntoSchemaEvolutionExtraSQLTests {
+
+  override protected lazy val extraTableProps: java.util.Map[String, String] = 
{
+    val props = new java.util.HashMap[String, String]()
+    props.put("supports-deltas", "true")
+    props.put("split-updates", "true")
+    props
+  }
+}
+
 // Scala/DataFrame API-based tests for delta-based row-level operations
 class DeltaBasedMergeIntoSchemaEvolutionScalaSuite
   extends MergeIntoSchemaEvolutionScalaSuiteBase
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
index 4312e4ffe71d..fb120d6e887a 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionExtraSourceColumnTests.scala
@@ -189,6 +189,83 @@ trait MergeIntoSchemaEvolutionExtraSourceColumnTests 
extends MergeIntoSchemaEvol
       (6, 350, "finance")).toDF("pk", "salary", "dep")
   )
 
+  testEvolution("SPARK-60709: source has extra column with set all columns" +
+    " and target metadata column in update condition")(
+    targetData = Seq(
+      (1, 100, "hr"),
+      (2, 200, "software"),
+      (3, 300, "hr")
+    ).toDF("pk", "salary", "dep"),
+    sourceData = Seq(
+      (1, 150, "hr", true),
+      (2, 250, "software", true),
+      (4, 450, "hr", false)
+    ).toDF("pk", "salary", "dep", "active"),
+    clauses = Seq(
+      updateAll(condition = "t._partition = 'hr'")
+    ),
+    expected = Seq[(Int, Int, String, java.lang.Boolean)](
+      (1, 150, "hr", true),
+      (2, 200, "software", null),
+      (3, 300, "hr", null)).toDF("pk", "salary", "dep", "active"),
+    expectedWithoutEvolution = Seq(
+      (1, 150, "hr"),
+      (2, 200, "software"),
+      (3, 300, "hr")).toDF("pk", "salary", "dep")
+  )
+
+  testEvolution("SPARK-60709: source has extra column with set all columns, 
insert all" +
+    " and target metadata column in update condition")(
+    targetData = Seq(
+      (1, 100, "hr"),
+      (2, 200, "software"),
+      (3, 300, "hr")
+    ).toDF("pk", "salary", "dep"),
+    sourceData = Seq(
+      (1, 150, "hr", true),
+      (2, 250, "software", true),
+      (4, 450, "hr", false)
+    ).toDF("pk", "salary", "dep", "active"),
+    clauses = Seq(
+      updateAll(condition = "t._partition = 'hr'"),
+      insertAll()
+    ),
+    expected = Seq[(Int, Int, String, java.lang.Boolean)](
+      (1, 150, "hr", true),
+      (2, 200, "software", null),
+      (3, 300, "hr", null),
+      (4, 450, "hr", false)).toDF("pk", "salary", "dep", "active"),
+    expectedWithoutEvolution = Seq(
+      (1, 150, "hr"),
+      (2, 200, "software"),
+      (3, 300, "hr"),
+      (4, 450, "hr")).toDF("pk", "salary", "dep")
+  )
+
+  testEvolution("SPARK-60709: source has extra column with set explicit 
column" +
+    " and target metadata column in merge and not matched by source 
conditions")(
+    targetData = Seq(
+      (1, 100, "hr"),
+      (2, 200, "software"),
+      (3, 300, "hr")
+    ).toDF("pk", "salary", "dep"),
+    sourceData = Seq(
+      (1, 150, "hr", true),
+      (2, 250, "software", true),
+      (4, 450, "hr", false)
+    ).toDF("pk", "salary", "dep", "active"),
+    cond = "t.pk = s.pk AND t._partition = 'hr'",
+    clauses = Seq(
+      update(set = "active = s.active"),
+      deleteNotMatched(condition = "t._partition = 'software'")
+    ),
+    expected = Seq[(Int, Int, String, java.lang.Boolean)](
+      (1, 100, "hr", true),
+      (3, 300, "hr", null)).toDF("pk", "salary", "dep", "active"),
+    expectErrorWithoutEvolutionContains = "A column, variable, or function 
parameter with name " +
+      "`active` cannot be resolved"
+  )
+
   testEvolution("source has extra column - empty table and insert")(
     targetData = Seq.empty[(Int, Int, String)].toDF("pk", "salary", "dep"),
     sourceData = Seq(
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
index 192dc94dd69d..35df9f6c4e9c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoSchemaEvolutionTypeWideningTests.scala
@@ -132,6 +132,23 @@ trait MergeIntoSchemaEvolutionTypeWideningTests extends 
MergeIntoSchemaEvolution
     expectErrorWithoutEvolutionContains = "CAST_OVERFLOW_IN_TABLE_INSERT"
   )
 
+  testEvolution("SPARK-60709: type widening top-level field" +
+    " and target metadata column in update condition")(
+    targetData = Seq((1, 100, "hr"), (2, 200, "software"), (3, 300, "hr"))
+      .toDF("pk", "salary", "dep"),
+    sourceData = Seq((1, 150L, "hr"), (2, 250L, "software"), (4, 450L, "hr"))
+      .toDF("pk", "salary", "dep"),
+    clauses = Seq(updateAll(condition = "t._partition = 'hr'")),
+    expected = Seq((1, 150L, "hr"), (2, 200L, "software"), (3, 300L, "hr"))
+      .toDF("pk", "salary", "dep"),
+    expectedSchema = StructType(Seq(
+      StructField("pk", IntegerType, nullable = false),
+      StructField("salary", LongType, nullable = false),
+      StructField("dep", StringType))),
+    expectedWithoutEvolution = Seq((1, 150, "hr"), (2, 200, "software"), (3, 
300, "hr"))
+      .toDF("pk", "salary", "dep")
+  )
+
   testEvolution("type widening - column not used in assignments")(
     targetData = Seq((1, 100, "hr"), (2, 200, "software")).toDF("pk", 
"salary", "dep"),
     sourceData = Seq((2, Long.MaxValue, "software"), (3, Long.MaxValue, 
"engineering"))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to