This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 7a14b77666c9 [SPARK-60667][SQL] Do not add metadata columns to the 
target table of DSv2 DELETE, UPDATE and MERGE
7a14b77666c9 is described below

commit 7a14b77666c9a5622a1984c340fd950d47ea3c3c
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 16:50:43 2026 -0700

    [SPARK-60667][SQL] Do not add metadata columns to the target table of DSv2 
DELETE, UPDATE and MERGE
    
    ### What changes were proposed in this pull request?
    
    This PR makes `AddMetadataColumns` skip the target table of `DELETE`, 
`UPDATE` and `MERGE` when Spark rewrites the command into a row-level 
operation, that is, when the table supports row-level operations 
(`SupportsRowLevelOperations`). `UPDATE` and `MERGE` on a table that accepts 
any schema keep the current behavior, because Spark leaves their assignments to 
the rules injected for that table. Spark still rewrites such a command once its 
assignments are aligned, for example a `MERGE`  [...]
    
    `RewriteDeleteFromTable`, `RewriteUpdateTable` and `RewriteMergeIntoTable` 
read on their own the metadata columns that the operation requires and, since 
SPARK-60673 (#59900), the ones that the command references 
(`RewriteRowLevelCommand.buildRelationWithAttrs`). They take the output of the 
target table as the data columns to write, and 
`ResolveRowLevelCommandAssignments` aligns assignments over the same output.
    
    That output has only data columns when the command is resolved in the first 
iteration of the `Resolution` batch, because `AddMetadataColumns` runs before 
`ResolveReferences`. When the command is resolved only in a later iteration, 
for example because its condition needs type coercion, `AddMetadataColumns` 
sees a resolved command that references a metadata column. It then calls 
`DataSourceV2Relation.withMetadataColumns()` on the target table, which appends 
every metadata column to its  [...]
    
    - `ResolveRowLevelCommandAssignments` adds identity assignments such as 
`index = index` for the metadata columns.
    - `RewriteUpdateTable` handles them as data columns. A metadata column with 
`PRESERVE_ON_UPDATE = false` keeps its old value instead of `null`.
    - Group-based `DELETE` and `UPDATE`, and delta-based `UPDATE`, pass the 
metadata columns in every row and report them in `LogicalWriteInfo.schema()`.
    - An `UPDATE` that is represented as a delete and an insert fails in 
`ExpandExec`, because the delete and reinsert projections have different widths.
    - A `MERGE` with a `NOT MATCHED` clause fails with `No assignment for 
'index'`, and a `MERGE` with only `MATCHED` clauses fails with an assertion 
error.
    
    For example, take an in-memory table `pk INT NOT NULL, id INT, dep STRING` 
partitioned by `dep`, with the metadata columns `index` (`PRESERVE_ON_UPDATE = 
false`) and `_partition`, and run:
    
    ```sql
    UPDATE t SET id = -1 WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
    ```
    
    **BEFORE**
    
    The target table gains `index` and `_partition`, and `index` is copied:
    
    ```
    ReplaceData RelationV2[pk#165, id#166, dep#167, index#168, _partition#169] 
cat.ns1.test_table
    +- Project [..., if (...) index#168 else index#168 AS index#174, if (...) 
_partition#169 else _partition#169 AS _partition#175]
       +- RelationV2[pk#165, id#166, dep#167, index#168, _partition#169] 
cat.ns1.test_table
    ```
    
    The connector receives the updated row with the metadata `(hr, 0)` and the 
write schema `struct<pk:int,id:int,dep:string,index:int,_partition:string>`.
    
    **AFTER**
    
    The plan is the same as for the statement without `AND id > 0.5`. The 
connector receives the metadata `(hr, null)` and the write schema 
`struct<pk:int,id:int,dep:string>`.
    
    ### Why are the changes needed?
    
    This is a correctness issue. `MetadataColumn.PRESERVE_ON_UPDATE` says that 
the metadata value is replaced with `null` if it is false. However, an `UPDATE` 
that is resolved in a later analyzer iteration passes the old value to the 
connector without any error. Row lineage in Iceberg and Delta Lake relies on 
this contract to reset the row version on update (SPARK-50820), so a connector 
that accepts such a write can record a stale row version.
    
    The same statements also pass metadata columns to connectors as data 
columns. An `UPDATE` represented as a delete and an insert, and `MERGE`, fail 
instead.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. This is a bug fix for DSv2 `DELETE`, `UPDATE` and `MERGE` statements 
that reference a metadata column and are resolved in a later analyzer iteration:
    
    - `UPDATE` replaces metadata columns with `PRESERVE_ON_UPDATE = false` with 
`null`, as documented, instead of passing the old value to the connector 
silently. This fixes the correctness issue above.
    - Connectors no longer receive metadata columns as data columns, in rows or 
in `LogicalWriteInfo.schema()`.
    - An `UPDATE` represented as a delete and an insert, and `MERGE`, no longer 
fail.
    
    These statements are now analyzed in the same way as statements that are 
resolved in the first iteration. For example, `DELETE ... WHERE _partition = 
'hr' AND id < 1.5` on a table whose operations request no metadata columns 
reads `_partition` without writing it, like `DELETE ... WHERE _partition = 'hr' 
AND id < 2`.
    
    This also holds for tables with `CHAR` or `VARCHAR` columns. Since 
SPARK-60708 (#59918), an `UPDATE`, or a `MERGE` with `UPDATE` or `INSERT` 
actions, on such a table can reference a metadata column when it is resolved in 
the first iteration. With this PR, it can also do so when it is resolved in a 
later iteration, without the problems above.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases. Each runs a statement that 
references a metadata column and needs type coercion, and checks the result, 
the write schema and the metadata that the connector receives.
    
    - `GroupBasedUpdateTableSuite`, `DeltaBasedUpdateTableSuite`, 
`DeltaBasedUpdateAsDeleteAndInsertTableSuite`: `SPARK-60667: update with type 
coercion handles metadata columns correctly`
    - `GroupBasedDeleteFromTableSuite`: `SPARK-60667: delete with type coercion 
preserves metadata columns for carried-over records`
    - `GroupBasedMergeIntoTableSuite`, `DeltaBasedMergeIntoTableSuite`, 
`DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite`: `SPARK-60667: merge 
with type coercion handles metadata columns correctly`
    - `MergeIntoTableSuiteBase`, for all group-based and delta-based `MERGE` 
suites: `SPARK-60667: insert-only merge with type coercion handles metadata 
columns correctly`. Its `MERGE` has only a `NOT MATCHED` action and references 
a metadata column in the `ON` condition. Spark rewrites it into an append, so 
the test checks the write schema of the append.
    - `GroupBasedNoMetadataDeleteFromTableSuite`, 
`DeltaBasedNoMetadataDeleteFromTableSuite`: `SPARK-60667: delete with type 
coercion reads non-required metadata columns without writing them`. They run 
`DELETE ... WHERE _partition = 'hr' AND id < 1.5` next to the `id < 2` tests of 
SPARK-60673, and check the scan schema too.
    
    All of them fail without this fix, except the one in 
`DeltaBasedNoMetadataDeleteFromTableSuite`. That `DELETE` already returned the 
correct result, and the test makes sure that it still does.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59895 from dongjoon-hyun/SPARK-60667.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 8a66f040f9b30fcdfc875eca804a328a07c42d62)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/analysis/Analyzer.scala     | 32 +++++++++++--
 .../connector/DeltaBasedMergeIntoTableSuite.scala  | 50 ++++++++++++++++++++
 ...ergeIntoTableUpdateAsDeleteAndInsertSuite.scala | 53 ++++++++++++++++++++++
 .../DeltaBasedNoMetadataDeleteFromTableSuite.scala | 24 ++++++++++
 ...ltaBasedUpdateAsDeleteAndInsertTableSuite.scala | 32 +++++++++++++
 .../sql/connector/DeltaBasedUpdateTableSuite.scala | 31 +++++++++++++
 .../connector/GroupBasedDeleteFromTableSuite.scala | 27 +++++++++++
 .../connector/GroupBasedMergeIntoTableSuite.scala  | 52 +++++++++++++++++++++
 .../GroupBasedNoMetadataDeleteFromTableSuite.scala | 24 ++++++++++
 .../sql/connector/GroupBasedUpdateTableSuite.scala | 28 ++++++++++++
 .../sql/connector/MergeIntoTableSuiteBase.scala    | 41 +++++++++++++++++
 11 files changed, 391 insertions(+), 3 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 4f295db0a547..a6bbb75d22af 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -1144,9 +1144,12 @@ class Analyzer(
           if (metaCols.isEmpty) {
             node
           } else {
-            val newNode = node.mapChildren(
-              addMetadataCol(_, metaCols.map(_.exprId).toSet, 
onlyUniqueAndNecessaryMetadataColumns)
-            )
+            val requiredAttrIds = metaCols.map(_.exprId).toSet
+            val newNode = node.mapChildren {
+              case child if isRowLevelCommandTarget(node, child) => child
+              case child =>
+                addMetadataCol(child, requiredAttrIds, 
onlyUniqueAndNecessaryMetadataColumns)
+            }
             // We should not change the output schema of the plan. We should 
project away the extra
             // metadata columns if necessary.
             if (newNode.sameOutput(node)) {
@@ -1181,6 +1184,29 @@ class Analyzer(
       })
     }
 
+    // DELETE, UPDATE and MERGE on a table that supports row-level operations 
are rewritten into
+    // plans that read the metadata columns they need (see 
RewriteRowLevelCommand). The rewrite
+    // takes the output of the target table as the data columns to write, so 
metadata columns must
+    // not be added to it, even if the command is resolved only in a later 
iteration. This does not
+    // apply to UPDATE and MERGE on a table that accepts any schema, whose 
assignments Spark leaves
+    // to the rules injected for that table (see 
ResolveRowLevelCommandAssignments). Spark still
+    // rewrites such a command once its assignments are aligned, for example a 
MERGE with only
+    // DELETE actions, which has nothing to align.
+    private def isRowLevelCommandTarget(node: LogicalPlan, child: 
LogicalPlan): Boolean = {
+      node match {
+        case d: DeleteFromTable =>
+          (d.table eq child) && (EliminateSubqueryAliases(child) match {
+            case ExtractV2Table(_: SupportsRowLevelOperations) => true
+            case _ => false
+          })
+        case u: UpdateTable =>
+          (u.table eq child) && u.rewritable && !u.skipSchemaResolution
+        case m: MergeIntoTable =>
+          (m.targetTable eq child) && m.rewritable && !m.skipSchemaResolution
+        case _ => false
+      }
+    }
+
     private def addMetadataCol(
         plan: LogicalPlan,
         requiredAttrIds: Set[ExprId],
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
index e193669ecf33..44c9f85a1934 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
@@ -328,4 +328,54 @@ class DeltaBasedMergeIntoTableSuite extends 
DeltaBasedMergeIntoTableSuiteBase {
         insertWriteLogEntry(data = Row(6, 0, "new")))
     }
   }
+
+  test("SPARK-60667: merge with type coercion handles metadata columns 
correctly") {
+    withTempView("source") {
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "software" }
+          |{ "pk": 3, "salary": 300, "dep": "hr" }
+          |{ "pk": 4, "salary": 400, "dep": "hr" }
+          |{ "pk": 5, "salary": 500, "dep": "hr" }
+          |""".stripMargin)
+
+      val sourceDF = Seq(3, 4, 5, 6).toDF("pk")
+      sourceDF.createOrReplaceTempView("source")
+
+      // the MATCHED condition references a metadata column and needs type 
coercion
+      // (`t.salary < 600.5`), so the MERGE is resolved only in a later 
iteration of the analyzer
+      sql(
+        s"""MERGE INTO $tableNameAsString t
+           |USING source s
+           |ON t.pk = s.pk
+           |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+           | UPDATE SET t.salary = t.salary + 1
+           |WHEN NOT MATCHED THEN
+           | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+           |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+           | DELETE
+           |""".stripMargin)
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(2, 200, "software"), // unchanged
+          Row(3, 301, "hr"), // update
+          Row(4, 401, "hr"), // update
+          Row(5, 501, "hr"), // update
+          Row(6, 0, "new"))) // insert
+
+      checkLastWriteInfo(
+        expectedRowSchema = table.schema,
+        expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+        expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+      checkLastWriteLog(
+        deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+        updateWriteLogEntry(id = 3, metadata = Row("hr", null), data = Row(3, 
301, "hr")),
+        updateWriteLogEntry(id = 4, metadata = Row("hr", null), data = Row(4, 
401, "hr")),
+        updateWriteLogEntry(id = 5, metadata = Row("hr", null), data = Row(5, 
501, "hr")),
+        insertWriteLogEntry(data = Row(6, 0, "new")))
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
index e93d4165be33..4d075c9d5bf1 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
@@ -82,4 +82,57 @@ class DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite
         insertWriteLogEntry(data = Row(6, 0, "new")))
     }
   }
+
+  test("SPARK-60667: merge with type coercion handles metadata columns 
correctly") {
+    withTempView("source") {
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "software" }
+          |{ "pk": 3, "salary": 300, "dep": "hr" }
+          |{ "pk": 4, "salary": 400, "dep": "hr" }
+          |{ "pk": 5, "salary": 500, "dep": "hr" }
+          |""".stripMargin)
+
+      val sourceDF = Seq(3, 4, 5, 6).toDF("pk")
+      sourceDF.createOrReplaceTempView("source")
+
+      // the MATCHED condition references a metadata column and needs type 
coercion
+      // (`t.salary < 600.5`), so the MERGE is resolved only in a later 
iteration of the analyzer
+      sql(
+        s"""MERGE INTO $tableNameAsString t
+           |USING source s
+           |ON t.pk = s.pk
+           |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+           | UPDATE SET t.salary = t.salary + 1
+           |WHEN NOT MATCHED THEN
+           | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+           |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+           | DELETE
+           |""".stripMargin)
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(2, 200, "software"), // unchanged
+          Row(3, 301, "hr"), // update
+          Row(4, 401, "hr"), // update
+          Row(5, 501, "hr"), // update
+          Row(6, 0, "new"))) // insert
+
+      checkLastWriteInfo(
+        expectedRowSchema = table.schema,
+        expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+        expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+      checkLastWriteLog(
+        deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+        deleteWriteLogEntry(id = 3, metadata = Row("hr", null)),
+        reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(3, 301, 
"hr")),
+        deleteWriteLogEntry(id = 4, metadata = Row("hr", null)),
+        reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(4, 401, 
"hr")),
+        deleteWriteLogEntry(id = 5, metadata = Row("hr", null)),
+        reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(5, 501, 
"hr")),
+        insertWriteLogEntry(data = Row(6, 0, "new")))
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
index 086a696aa8a4..8f57c0b369e5 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
@@ -53,4 +53,28 @@ class DeltaBasedNoMetadataDeleteFromTableSuite extends 
DeleteFromTableSuiteBase
     checkLastWriteInfo(expectedRowIdSchema = Some(StructType(Array(PK_FIELD))))
     checkLastWriteLog(deleteWriteLogEntry(id = 1, metadata = null))
   }
+
+  test("SPARK-60667: delete with type coercion reads non-required metadata 
columns " +
+    "without writing them") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "hr" }
+        |{ "pk": 3, "id": 1, "dep": "software" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
< 1.5`),
+    // so the DELETE is resolved only in a later iteration of the analyzer
+    executeAndCheckScan(
+      s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5",
+      // `pk` is used to encode deletes
+      // `id` and `_partition` are used in the condition
+      expectedScanSchema = "pk INT, id INT, _partition STRING")
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(2, 2, "hr") :: Row(3, 1, "software") :: Nil)
+
+    checkLastWriteInfo(expectedRowIdSchema = Some(StructType(Array(PK_FIELD))))
+    checkLastWriteLog(deleteWriteLogEntry(id = 1, metadata = null))
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
index 612a26e756ab..1b3398f012be 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
@@ -94,4 +94,36 @@ class DeltaBasedUpdateAsDeleteAndInsertTableSuite extends 
DeltaBasedUpdateTableS
         reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(1, -1, 
"hr")))
     }
   }
+
+  test("SPARK-60667: update with type coercion handles metadata columns 
correctly") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "software" }
+        |{ "pk": 3, "id": 3, "dep": "hr" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
> 0.5`),
+    // so the UPDATE is resolved only in a later iteration of the analyzer
+    sql(
+      s"""UPDATE $tableNameAsString
+         |SET id = -1
+         |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+         |""".stripMargin)
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+    checkLastWriteInfo(
+      expectedRowSchema = StructType(table.schema.map {
+        case attr if attr.name == "id" => attr.copy(nullable = false) // input 
is a constant
+        case attr => attr
+      }),
+      expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+      expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+    checkLastWriteLog(
+      deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+      reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(1, -1, 
"hr")))
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
index aa02ab118948..afd9b5ce4e17 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
@@ -94,6 +94,37 @@ class DeltaBasedUpdateTableSuite extends 
DeltaBasedUpdateTableSuiteBase {
     }
   }
 
+  test("SPARK-60667: update with type coercion handles metadata columns 
correctly") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "software" }
+        |{ "pk": 3, "id": 3, "dep": "hr" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
> 0.5`),
+    // so the UPDATE is resolved only in a later iteration of the analyzer
+    sql(
+      s"""UPDATE $tableNameAsString
+         |SET id = -1
+         |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+         |""".stripMargin)
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+    checkLastWriteInfo(
+      expectedRowSchema = StructType(table.schema.map {
+        case attr if attr.name == "id" => attr.copy(nullable = false) // input 
is a constant
+        case attr => attr
+      }),
+      expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+      expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+    checkLastWriteLog(
+      updateWriteLogEntry(id = 1, metadata = Row("hr", null), data = Row(1, 
-1, "hr")))
+  }
+
   test("update runtime group filtering (DPP enabled)") {
     withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
       checkUpdateRuntimeGroupFiltering()
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
index 3889b0d172ad..416cb4f37633 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
@@ -50,6 +50,33 @@ class GroupBasedDeleteFromTableSuite extends 
DeleteFromTableSuiteBase {
     checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 2)
   }
 
+  test("SPARK-60667: delete with type coercion preserves metadata columns for 
" +
+    "carried-over records") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "software" }
+        |{ "pk": 3, "id": 3, "dep": "hr" }
+        |{ "pk": 4, "id": 4, "dep": "hr" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
< 1.5`),
+    // so the DELETE is resolved only in a later iteration of the analyzer
+    sql(s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5")
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(2, 2, "software") :: Row(3, 3, "hr") :: Row(4, 4, "hr") :: Nil)
+
+    checkLastWriteInfo(
+      expectedRowSchema = table.schema,
+      expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD))))
+
+    checkLastWriteLog(
+      writeWithMetadataLogEntry(metadata = Row("hr", 1), data = Row(3, 3, 
"hr")),
+      writeWithMetadataLogEntry(metadata = Row("hr", 2), data = Row(4, 4, 
"hr")))
+    checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 2)
+  }
+
   test("delete with nondeterministic conditions") {
     createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
       """{ "pk": 1, "id": 1, "dep": "hr" }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
index a79c45fdbb16..6433f9f7214e 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
@@ -82,6 +82,58 @@ class GroupBasedMergeIntoTableSuite extends 
MergeIntoTableSuiteBase {
     }
   }
 
+  test("SPARK-60667: merge with type coercion handles metadata columns 
correctly") {
+    withTempView("source") {
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "software" }
+          |{ "pk": 3, "salary": 300, "dep": "hr" }
+          |{ "pk": 4, "salary": 400, "dep": "hr" }
+          |{ "pk": 5, "salary": 500, "dep": "hr" }
+          |{ "pk": 7, "salary": 700, "dep": "hr" }
+          |""".stripMargin)
+
+      val sourceDF = Seq(3, 4, 5, 6, 7).toDF("pk")
+      sourceDF.createOrReplaceTempView("source")
+
+      // the MATCHED condition references a metadata column and needs type 
coercion
+      // (`t.salary < 600.5`), so the MERGE is resolved only in a later 
iteration of the analyzer
+      sql(
+        s"""MERGE INTO $tableNameAsString t
+           |USING source s
+           |ON t.pk = s.pk
+           |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+           | UPDATE SET t.salary = t.salary + 1
+           |WHEN NOT MATCHED THEN
+           | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+           |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+           | DELETE
+           |""".stripMargin)
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(2, 200, "software"), // unchanged
+          Row(3, 301, "hr"), // update
+          Row(4, 401, "hr"), // update
+          Row(5, 501, "hr"), // update
+          Row(6, 0, "new"), // insert
+          Row(7, 700, "hr"))) // unchanged
+
+      checkLastWriteInfo(
+        expectedRowSchema = table.schema,
+        expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+      checkLastWriteLog(
+        writeWithMetadataLogEntry(metadata = Row("software", 0), data = Row(2, 
200, "software")),
+        writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(3, 
301, "hr")),
+        writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(4, 
401, "hr")),
+        writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(5, 
501, "hr")),
+        writeLogEntry(data = Row(6, 0, "new")),
+        writeWithMetadataLogEntry(metadata = Row("hr", 4), data = Row(7, 700, 
"hr")))
+    }
+  }
+
   test("merge runtime filtering is disabled with NOT MATCHED BY SOURCE 
clauses") {
     withTempView("source") {
       createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
index 219305294ce2..b96fae3b7c96 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
@@ -47,4 +47,28 @@ class GroupBasedNoMetadataDeleteFromTableSuite extends 
DeleteFromTableSuiteBase
     checkLastWriteInfo(expectedRowSchema = table.schema)
     checkLastWriteLog(writeLogEntry(data = Row(2, 2, "hr")))
   }
+
+  test("SPARK-60667: delete with type coercion reads non-required metadata 
columns " +
+    "without writing them") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "hr" }
+        |{ "pk": 3, "id": 1, "dep": "software" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
< 1.5`),
+    // so the DELETE is resolved only in a later iteration of the analyzer
+    executeAndCheckScans(
+      s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5",
+      primaryScanSchema = "pk INT, id INT, dep STRING, _partition STRING",
+      groupFilterScanSchema = Some("id INT, dep STRING, _partition STRING"))
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(2, 2, "hr") :: Row(3, 1, "software") :: Nil)
+
+    checkReplacedPartitions(Seq("hr"))
+    checkLastWriteInfo(expectedRowSchema = table.schema)
+    checkLastWriteLog(writeLogEntry(data = Row(2, 2, "hr")))
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
index 4ce568ed7f68..eaf6ab03d9e9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
@@ -84,6 +84,34 @@ class GroupBasedUpdateTableSuite extends 
UpdateTableSuiteBase {
     }
   }
 
+  test("SPARK-60667: update with type coercion handles metadata columns 
correctly") {
+    createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+      """{ "pk": 1, "id": 1, "dep": "hr" }
+        |{ "pk": 2, "id": 2, "dep": "software" }
+        |{ "pk": 3, "id": 3, "dep": "hr" }
+        |""".stripMargin)
+
+    // the condition references a metadata column and needs type coercion (`id 
> 0.5`),
+    // so the UPDATE is resolved only in a later iteration of the analyzer
+    sql(
+      s"""UPDATE $tableNameAsString
+         |SET id = -1
+         |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+         |""".stripMargin)
+
+    checkAnswer(
+      sql(s"SELECT * FROM $tableNameAsString"),
+      Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+    checkLastWriteInfo(
+      expectedRowSchema = table.schema,
+      expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD, 
INDEX_FIELD_NULLABLE))))
+
+    checkLastWriteLog(
+      writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(1, -1, 
"hr")),
+      writeWithMetadataLogEntry(metadata = Row("hr", 1), data = Row(3, 3, 
"hr")))
+  }
+
   test("update runtime group filtering") {
     Seq(true, false).foreach { ddpEnabled =>
       Seq(true, false).foreach { aqeEnabled =>
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
index ea5d01c77180..9dd664b9cb74 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
@@ -2094,6 +2094,47 @@ abstract class MergeIntoTableSuiteBase extends 
RowLevelOperationSuiteBase
     }
   }
 
+  test("SPARK-60667: insert-only merge with type coercion handles metadata 
columns correctly") {
+    withTempView("source") {
+      createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+        """{ "pk": 1, "salary": 100, "dep": "hr" }
+          |{ "pk": 2, "salary": 200, "dep": "software" }
+          |{ "pk": 3, "salary": 700, "dep": "hr" }
+          |""".stripMargin)
+
+      Seq(1, 2, 3, 4).toDF("pk").createOrReplaceTempView("source")
+
+      // the ON condition references a metadata column and needs type coercion
+      // (`t.salary < 600.5`), so the MERGE is resolved only in a later 
iteration of the analyzer
+      val executedPlan = executeAndKeepPlan {
+        sql(
+          s"""MERGE INTO $tableNameAsString t
+             |USING source s
+             |ON t.pk = s.pk AND t._partition = 'hr' AND t.salary < 600.5
+             |WHEN NOT MATCHED THEN
+             | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+             |""".stripMargin)
+      }
+
+      // the MERGE is rewritten into InsertOnlyMerge (an append) that writes 
only data columns
+      val write = executedPlan.collectFirst {
+        case e: InsertOnlyMergeExec => e.write
+      }.getOrElse(fail("expected an InsertOnlyMergeExec in the executed plan"))
+      val append = write.toBatch.asInstanceOf[InMemoryBaseTable#Append]
+      assert(append.info.schema.fieldNames === table.schema.fieldNames)
+
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Seq(
+          Row(1, 100, "hr"), // unchanged (matched)
+          Row(2, 200, "software"), // unchanged
+          Row(3, 700, "hr"), // unchanged
+          Row(2, 0, "new"), // insert (not matched in hr)
+          Row(3, 0, "new"), // insert (not matched by salary)
+          Row(4, 0, "new"))) // insert
+    }
+  }
+
   test("merge with NOT NULL checks") {
     withTempView("source") {
       createAndInitTable("pk INT NOT NULL, s STRUCT<n_i: INT NOT NULL, n_l: 
LONG>, dep STRING",


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to