This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8a66f040f9b3 [SPARK-60667][SQL] Do not add metadata columns to the
target table of DSv2 DELETE, UPDATE and MERGE
8a66f040f9b3 is described below
commit 8a66f040f9b30fcdfc875eca804a328a07c42d62
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 29 16:50:43 2026 -0700
[SPARK-60667][SQL] Do not add metadata columns to the target table of DSv2
DELETE, UPDATE and MERGE
### What changes were proposed in this pull request?
This PR makes `AddMetadataColumns` skip the target table of `DELETE`,
`UPDATE` and `MERGE` when Spark rewrites the command into a row-level
operation, that is, when the table supports row-level operations
(`SupportsRowLevelOperations`). `UPDATE` and `MERGE` on a table that accepts
any schema keep the current behavior, because Spark leaves their assignments to
the rules injected for that table. Spark still rewrites such a command once its
assignments are aligned, for example a `MERGE` [...]
`RewriteDeleteFromTable`, `RewriteUpdateTable` and `RewriteMergeIntoTable`
read on their own the metadata columns that the operation requires and, since
SPARK-60673 (#59900), the ones that the command references
(`RewriteRowLevelCommand.buildRelationWithAttrs`). They take the output of the
target table as the data columns to write, and
`ResolveRowLevelCommandAssignments` aligns assignments over the same output.
That output has only data columns when the command is resolved in the first
iteration of the `Resolution` batch, because `AddMetadataColumns` runs before
`ResolveReferences`. When the command is resolved only in a later iteration,
for example because its condition needs type coercion, `AddMetadataColumns`
sees a resolved command that references a metadata column. It then calls
`DataSourceV2Relation.withMetadataColumns()` on the target table, which appends
every metadata column to its [...]
- `ResolveRowLevelCommandAssignments` adds identity assignments such as
`index = index` for the metadata columns.
- `RewriteUpdateTable` handles them as data columns. A metadata column with
`PRESERVE_ON_UPDATE = false` keeps its old value instead of `null`.
- Group-based `DELETE` and `UPDATE`, and delta-based `UPDATE`, pass the
metadata columns in every row and report them in `LogicalWriteInfo.schema()`.
- An `UPDATE` that is represented as a delete and an insert fails in
`ExpandExec`, because the delete and reinsert projections have different widths.
- A `MERGE` with a `NOT MATCHED` clause fails with `No assignment for
'index'`, and a `MERGE` with only `MATCHED` clauses fails with an assertion
error.
For example, take an in-memory table `pk INT NOT NULL, id INT, dep STRING`
partitioned by `dep`, with the metadata columns `index` (`PRESERVE_ON_UPDATE =
false`) and `_partition`, and run:
```sql
UPDATE t SET id = -1 WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
```
**BEFORE**
The target table gains `index` and `_partition`, and `index` is copied:
```
ReplaceData RelationV2[pk#165, id#166, dep#167, index#168, _partition#169]
cat.ns1.test_table
+- Project [..., if (...) index#168 else index#168 AS index#174, if (...)
_partition#169 else _partition#169 AS _partition#175]
+- RelationV2[pk#165, id#166, dep#167, index#168, _partition#169]
cat.ns1.test_table
```
The connector receives the updated row with the metadata `(hr, 0)` and the
write schema `struct<pk:int,id:int,dep:string,index:int,_partition:string>`.
**AFTER**
The plan is the same as for the statement without `AND id > 0.5`. The
connector receives the metadata `(hr, null)` and the write schema
`struct<pk:int,id:int,dep:string>`.
### Why are the changes needed?
This is a correctness issue. `MetadataColumn.PRESERVE_ON_UPDATE` says that
the metadata value is replaced with `null` if it is false. However, an `UPDATE`
that is resolved in a later analyzer iteration passes the old value to the
connector without any error. Row lineage in Iceberg and Delta Lake relies on
this contract to reset the row version on update (SPARK-50820), so a connector
that accepts such a write can record a stale row version.
The same statements also pass metadata columns to connectors as data
columns. An `UPDATE` represented as a delete and an insert, and `MERGE`, fail
instead.
### Does this PR introduce _any_ user-facing change?
Yes. This is a bug fix for DSv2 `DELETE`, `UPDATE` and `MERGE` statements
that reference a metadata column and are resolved in a later analyzer iteration:
- `UPDATE` replaces metadata columns with `PRESERVE_ON_UPDATE = false` with
`null`, as documented, instead of passing the old value to the connector
silently. This fixes the correctness issue above.
- Connectors no longer receive metadata columns as data columns, in rows or
in `LogicalWriteInfo.schema()`.
- An `UPDATE` represented as a delete and an insert, and `MERGE`, no longer
fail.
These statements are now analyzed in the same way as statements that are
resolved in the first iteration. For example, `DELETE ... WHERE _partition =
'hr' AND id < 1.5` on a table whose operations request no metadata columns
reads `_partition` without writing it, like `DELETE ... WHERE _partition = 'hr'
AND id < 2`.
This also holds for tables with `CHAR` or `VARCHAR` columns. Since
SPARK-60708 (#59918), an `UPDATE`, or a `MERGE` with `UPDATE` or `INSERT`
actions, on such a table can reference a metadata column when it is resolved in
the first iteration. With this PR, it can also do so when it is resolved in a
later iteration, without the problems above.
### How was this patch tested?
Pass the CIs with the newly added test cases. Each runs a statement that
references a metadata column and needs type coercion, and checks the result,
the write schema and the metadata that the connector receives.
- `GroupBasedUpdateTableSuite`, `DeltaBasedUpdateTableSuite`,
`DeltaBasedUpdateAsDeleteAndInsertTableSuite`: `SPARK-60667: update with type
coercion handles metadata columns correctly`
- `GroupBasedDeleteFromTableSuite`: `SPARK-60667: delete with type coercion
preserves metadata columns for carried-over records`
- `GroupBasedMergeIntoTableSuite`, `DeltaBasedMergeIntoTableSuite`,
`DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite`: `SPARK-60667: merge
with type coercion handles metadata columns correctly`
- `MergeIntoTableSuiteBase`, for all group-based and delta-based `MERGE`
suites: `SPARK-60667: insert-only merge with type coercion handles metadata
columns correctly`. Its `MERGE` has only a `NOT MATCHED` action and references
a metadata column in the `ON` condition. Spark rewrites it into an append, so
the test checks the write schema of the append.
- `GroupBasedNoMetadataDeleteFromTableSuite`,
`DeltaBasedNoMetadataDeleteFromTableSuite`: `SPARK-60667: delete with type
coercion reads non-required metadata columns without writing them`. They run
`DELETE ... WHERE _partition = 'hr' AND id < 1.5` next to the `id < 2` tests of
SPARK-60673, and check the scan schema too.
All of them fail without this fix, except the one in
`DeltaBasedNoMetadataDeleteFromTableSuite`. That `DELETE` already returned the
correct result, and the test makes sure that it still does.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59895 from dongjoon-hyun/SPARK-60667.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 32 +++++++++++--
.../connector/DeltaBasedMergeIntoTableSuite.scala | 50 ++++++++++++++++++++
...ergeIntoTableUpdateAsDeleteAndInsertSuite.scala | 53 ++++++++++++++++++++++
.../DeltaBasedNoMetadataDeleteFromTableSuite.scala | 24 ++++++++++
...ltaBasedUpdateAsDeleteAndInsertTableSuite.scala | 32 +++++++++++++
.../sql/connector/DeltaBasedUpdateTableSuite.scala | 31 +++++++++++++
.../connector/GroupBasedDeleteFromTableSuite.scala | 27 +++++++++++
.../connector/GroupBasedMergeIntoTableSuite.scala | 52 +++++++++++++++++++++
.../GroupBasedNoMetadataDeleteFromTableSuite.scala | 24 ++++++++++
.../sql/connector/GroupBasedUpdateTableSuite.scala | 28 ++++++++++++
.../sql/connector/MergeIntoTableSuiteBase.scala | 41 +++++++++++++++++
11 files changed, 391 insertions(+), 3 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 472c0a544e38..d5464f075416 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -1139,9 +1139,12 @@ class Analyzer(
if (metaCols.isEmpty) {
node
} else {
- val newNode = node.mapChildren(
- addMetadataCol(_, metaCols.map(_.exprId).toSet,
onlyUniqueAndNecessaryMetadataColumns)
- )
+ val requiredAttrIds = metaCols.map(_.exprId).toSet
+ val newNode = node.mapChildren {
+ case child if isRowLevelCommandTarget(node, child) => child
+ case child =>
+ addMetadataCol(child, requiredAttrIds,
onlyUniqueAndNecessaryMetadataColumns)
+ }
// We should not change the output schema of the plan. We should
project away the extra
// metadata columns if necessary.
if (newNode.sameOutput(node)) {
@@ -1176,6 +1179,29 @@ class Analyzer(
})
}
+ // DELETE, UPDATE and MERGE on a table that supports row-level operations
are rewritten into
+ // plans that read the metadata columns they need (see
RewriteRowLevelCommand). The rewrite
+ // takes the output of the target table as the data columns to write, so
metadata columns must
+ // not be added to it, even if the command is resolved only in a later
iteration. This does not
+ // apply to UPDATE and MERGE on a table that accepts any schema, whose
assignments Spark leaves
+ // to the rules injected for that table (see
ResolveRowLevelCommandAssignments). Spark still
+ // rewrites such a command once its assignments are aligned, for example a
MERGE with only
+ // DELETE actions, which has nothing to align.
+ private def isRowLevelCommandTarget(node: LogicalPlan, child:
LogicalPlan): Boolean = {
+ node match {
+ case d: DeleteFromTable =>
+ (d.table eq child) && (EliminateSubqueryAliases(child) match {
+ case ExtractV2Table(_: SupportsRowLevelOperations) => true
+ case _ => false
+ })
+ case u: UpdateTable =>
+ (u.table eq child) && u.rewritable && !u.skipSchemaResolution
+ case m: MergeIntoTable =>
+ (m.targetTable eq child) && m.rewritable && !m.skipSchemaResolution
+ case _ => false
+ }
+ }
+
private def addMetadataCol(
plan: LogicalPlan,
requiredAttrIds: Set[ExprId],
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
index e193669ecf33..44c9f85a1934 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableSuite.scala
@@ -328,4 +328,54 @@ class DeltaBasedMergeIntoTableSuite extends
DeltaBasedMergeIntoTableSuiteBase {
insertWriteLogEntry(data = Row(6, 0, "new")))
}
}
+
+ test("SPARK-60667: merge with type coercion handles metadata columns
correctly") {
+ withTempView("source") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 300, "dep": "hr" }
+ |{ "pk": 4, "salary": 400, "dep": "hr" }
+ |{ "pk": 5, "salary": 500, "dep": "hr" }
+ |""".stripMargin)
+
+ val sourceDF = Seq(3, 4, 5, 6).toDF("pk")
+ sourceDF.createOrReplaceTempView("source")
+
+ // the MATCHED condition references a metadata column and needs type
coercion
+ // (`t.salary < 600.5`), so the MERGE is resolved only in a later
iteration of the analyzer
+ sql(
+ s"""MERGE INTO $tableNameAsString t
+ |USING source s
+ |ON t.pk = s.pk
+ |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+ | UPDATE SET t.salary = t.salary + 1
+ |WHEN NOT MATCHED THEN
+ | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+ |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+ | DELETE
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(2, 200, "software"), // unchanged
+ Row(3, 301, "hr"), // update
+ Row(4, 401, "hr"), // update
+ Row(5, 501, "hr"), // update
+ Row(6, 0, "new"))) // insert
+
+ checkLastWriteInfo(
+ expectedRowSchema = table.schema,
+ expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+ updateWriteLogEntry(id = 3, metadata = Row("hr", null), data = Row(3,
301, "hr")),
+ updateWriteLogEntry(id = 4, metadata = Row("hr", null), data = Row(4,
401, "hr")),
+ updateWriteLogEntry(id = 5, metadata = Row("hr", null), data = Row(5,
501, "hr")),
+ insertWriteLogEntry(data = Row(6, 0, "new")))
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
index e93d4165be33..4d075c9d5bf1 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite.scala
@@ -82,4 +82,57 @@ class DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite
insertWriteLogEntry(data = Row(6, 0, "new")))
}
}
+
+ test("SPARK-60667: merge with type coercion handles metadata columns
correctly") {
+ withTempView("source") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 300, "dep": "hr" }
+ |{ "pk": 4, "salary": 400, "dep": "hr" }
+ |{ "pk": 5, "salary": 500, "dep": "hr" }
+ |""".stripMargin)
+
+ val sourceDF = Seq(3, 4, 5, 6).toDF("pk")
+ sourceDF.createOrReplaceTempView("source")
+
+ // the MATCHED condition references a metadata column and needs type
coercion
+ // (`t.salary < 600.5`), so the MERGE is resolved only in a later
iteration of the analyzer
+ sql(
+ s"""MERGE INTO $tableNameAsString t
+ |USING source s
+ |ON t.pk = s.pk
+ |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+ | UPDATE SET t.salary = t.salary + 1
+ |WHEN NOT MATCHED THEN
+ | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+ |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+ | DELETE
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(2, 200, "software"), // unchanged
+ Row(3, 301, "hr"), // update
+ Row(4, 401, "hr"), // update
+ Row(5, 501, "hr"), // update
+ Row(6, 0, "new"))) // insert
+
+ checkLastWriteInfo(
+ expectedRowSchema = table.schema,
+ expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+ deleteWriteLogEntry(id = 3, metadata = Row("hr", null)),
+ reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(3, 301,
"hr")),
+ deleteWriteLogEntry(id = 4, metadata = Row("hr", null)),
+ reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(4, 401,
"hr")),
+ deleteWriteLogEntry(id = 5, metadata = Row("hr", null)),
+ reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(5, 501,
"hr")),
+ insertWriteLogEntry(data = Row(6, 0, "new")))
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
index 086a696aa8a4..8f57c0b369e5 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedNoMetadataDeleteFromTableSuite.scala
@@ -53,4 +53,28 @@ class DeltaBasedNoMetadataDeleteFromTableSuite extends
DeleteFromTableSuiteBase
checkLastWriteInfo(expectedRowIdSchema = Some(StructType(Array(PK_FIELD))))
checkLastWriteLog(deleteWriteLogEntry(id = 1, metadata = null))
}
+
+ test("SPARK-60667: delete with type coercion reads non-required metadata
columns " +
+ "without writing them") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "hr" }
+ |{ "pk": 3, "id": 1, "dep": "software" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
< 1.5`),
+ // so the DELETE is resolved only in a later iteration of the analyzer
+ executeAndCheckScan(
+ s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5",
+ // `pk` is used to encode deletes
+ // `id` and `_partition` are used in the condition
+ expectedScanSchema = "pk INT, id INT, _partition STRING")
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(2, 2, "hr") :: Row(3, 1, "software") :: Nil)
+
+ checkLastWriteInfo(expectedRowIdSchema = Some(StructType(Array(PK_FIELD))))
+ checkLastWriteLog(deleteWriteLogEntry(id = 1, metadata = null))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
index 612a26e756ab..1b3398f012be 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala
@@ -94,4 +94,36 @@ class DeltaBasedUpdateAsDeleteAndInsertTableSuite extends
DeltaBasedUpdateTableS
reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(1, -1,
"hr")))
}
}
+
+ test("SPARK-60667: update with type coercion handles metadata columns
correctly") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "software" }
+ |{ "pk": 3, "id": 3, "dep": "hr" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
> 0.5`),
+ // so the UPDATE is resolved only in a later iteration of the analyzer
+ sql(
+ s"""UPDATE $tableNameAsString
+ |SET id = -1
+ |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+ checkLastWriteInfo(
+ expectedRowSchema = StructType(table.schema.map {
+ case attr if attr.name == "id" => attr.copy(nullable = false) // input
is a constant
+ case attr => attr
+ }),
+ expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ deleteWriteLogEntry(id = 1, metadata = Row("hr", null)),
+ reinsertWriteLogEntry(metadata = Row("hr", null), data = Row(1, -1,
"hr")))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
index aa02ab118948..afd9b5ce4e17 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DeltaBasedUpdateTableSuite.scala
@@ -94,6 +94,37 @@ class DeltaBasedUpdateTableSuite extends
DeltaBasedUpdateTableSuiteBase {
}
}
+ test("SPARK-60667: update with type coercion handles metadata columns
correctly") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "software" }
+ |{ "pk": 3, "id": 3, "dep": "hr" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
> 0.5`),
+ // so the UPDATE is resolved only in a later iteration of the analyzer
+ sql(
+ s"""UPDATE $tableNameAsString
+ |SET id = -1
+ |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+ checkLastWriteInfo(
+ expectedRowSchema = StructType(table.schema.map {
+ case attr if attr.name == "id" => attr.copy(nullable = false) // input
is a constant
+ case attr => attr
+ }),
+ expectedRowIdSchema = Some(StructType(Array(PK_FIELD))),
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ updateWriteLogEntry(id = 1, metadata = Row("hr", null), data = Row(1,
-1, "hr")))
+ }
+
test("update runtime group filtering (DPP enabled)") {
withSQLConf(SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true") {
checkUpdateRuntimeGroupFiltering()
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
index 3889b0d172ad..416cb4f37633 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedDeleteFromTableSuite.scala
@@ -50,6 +50,33 @@ class GroupBasedDeleteFromTableSuite extends
DeleteFromTableSuiteBase {
checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 2)
}
+ test("SPARK-60667: delete with type coercion preserves metadata columns for
" +
+ "carried-over records") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "software" }
+ |{ "pk": 3, "id": 3, "dep": "hr" }
+ |{ "pk": 4, "id": 4, "dep": "hr" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
< 1.5`),
+ // so the DELETE is resolved only in a later iteration of the analyzer
+ sql(s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5")
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(2, 2, "software") :: Row(3, 3, "hr") :: Row(4, 4, "hr") :: Nil)
+
+ checkLastWriteInfo(
+ expectedRowSchema = table.schema,
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD))))
+
+ checkLastWriteLog(
+ writeWithMetadataLogEntry(metadata = Row("hr", 1), data = Row(3, 3,
"hr")),
+ writeWithMetadataLogEntry(metadata = Row("hr", 2), data = Row(4, 4,
"hr")))
+ checkDeleteMetrics(numDeletedRows = 1, numCopiedRows = 2)
+ }
+
test("delete with nondeterministic conditions") {
createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
"""{ "pk": 1, "id": 1, "dep": "hr" }
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
index a79c45fdbb16..6433f9f7214e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedMergeIntoTableSuite.scala
@@ -82,6 +82,58 @@ class GroupBasedMergeIntoTableSuite extends
MergeIntoTableSuiteBase {
}
}
+ test("SPARK-60667: merge with type coercion handles metadata columns
correctly") {
+ withTempView("source") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 300, "dep": "hr" }
+ |{ "pk": 4, "salary": 400, "dep": "hr" }
+ |{ "pk": 5, "salary": 500, "dep": "hr" }
+ |{ "pk": 7, "salary": 700, "dep": "hr" }
+ |""".stripMargin)
+
+ val sourceDF = Seq(3, 4, 5, 6, 7).toDF("pk")
+ sourceDF.createOrReplaceTempView("source")
+
+ // the MATCHED condition references a metadata column and needs type
coercion
+ // (`t.salary < 600.5`), so the MERGE is resolved only in a later
iteration of the analyzer
+ sql(
+ s"""MERGE INTO $tableNameAsString t
+ |USING source s
+ |ON t.pk = s.pk
+ |WHEN MATCHED AND t._partition = 'hr' AND t.salary < 600.5 THEN
+ | UPDATE SET t.salary = t.salary + 1
+ |WHEN NOT MATCHED THEN
+ | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+ |WHEN NOT MATCHED BY SOURCE AND pk = 1 THEN
+ | DELETE
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(2, 200, "software"), // unchanged
+ Row(3, 301, "hr"), // update
+ Row(4, 401, "hr"), // update
+ Row(5, 501, "hr"), // update
+ Row(6, 0, "new"), // insert
+ Row(7, 700, "hr"))) // unchanged
+
+ checkLastWriteInfo(
+ expectedRowSchema = table.schema,
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ writeWithMetadataLogEntry(metadata = Row("software", 0), data = Row(2,
200, "software")),
+ writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(3,
301, "hr")),
+ writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(4,
401, "hr")),
+ writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(5,
501, "hr")),
+ writeLogEntry(data = Row(6, 0, "new")),
+ writeWithMetadataLogEntry(metadata = Row("hr", 4), data = Row(7, 700,
"hr")))
+ }
+ }
+
test("merge runtime filtering is disabled with NOT MATCHED BY SOURCE
clauses") {
withTempView("source") {
createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
index 219305294ce2..b96fae3b7c96 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedNoMetadataDeleteFromTableSuite.scala
@@ -47,4 +47,28 @@ class GroupBasedNoMetadataDeleteFromTableSuite extends
DeleteFromTableSuiteBase
checkLastWriteInfo(expectedRowSchema = table.schema)
checkLastWriteLog(writeLogEntry(data = Row(2, 2, "hr")))
}
+
+ test("SPARK-60667: delete with type coercion reads non-required metadata
columns " +
+ "without writing them") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "hr" }
+ |{ "pk": 3, "id": 1, "dep": "software" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
< 1.5`),
+ // so the DELETE is resolved only in a later iteration of the analyzer
+ executeAndCheckScans(
+ s"DELETE FROM $tableNameAsString WHERE _partition = 'hr' AND id < 1.5",
+ primaryScanSchema = "pk INT, id INT, dep STRING, _partition STRING",
+ groupFilterScanSchema = Some("id INT, dep STRING, _partition STRING"))
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(2, 2, "hr") :: Row(3, 1, "software") :: Nil)
+
+ checkReplacedPartitions(Seq("hr"))
+ checkLastWriteInfo(expectedRowSchema = table.schema)
+ checkLastWriteLog(writeLogEntry(data = Row(2, 2, "hr")))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
index 4ce568ed7f68..eaf6ab03d9e9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/GroupBasedUpdateTableSuite.scala
@@ -84,6 +84,34 @@ class GroupBasedUpdateTableSuite extends
UpdateTableSuiteBase {
}
}
+ test("SPARK-60667: update with type coercion handles metadata columns
correctly") {
+ createAndInitTable("pk INT NOT NULL, id INT, dep STRING",
+ """{ "pk": 1, "id": 1, "dep": "hr" }
+ |{ "pk": 2, "id": 2, "dep": "software" }
+ |{ "pk": 3, "id": 3, "dep": "hr" }
+ |""".stripMargin)
+
+ // the condition references a metadata column and needs type coercion (`id
> 0.5`),
+ // so the UPDATE is resolved only in a later iteration of the analyzer
+ sql(
+ s"""UPDATE $tableNameAsString
+ |SET id = -1
+ |WHERE id IN (1, 100) AND _partition = 'hr' AND id > 0.5
+ |""".stripMargin)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, -1, "hr") :: Row(2, 2, "software") :: Row(3, 3, "hr") :: Nil)
+
+ checkLastWriteInfo(
+ expectedRowSchema = table.schema,
+ expectedMetadataSchema = Some(StructType(Array(PARTITION_FIELD,
INDEX_FIELD_NULLABLE))))
+
+ checkLastWriteLog(
+ writeWithMetadataLogEntry(metadata = Row("hr", null), data = Row(1, -1,
"hr")),
+ writeWithMetadataLogEntry(metadata = Row("hr", 1), data = Row(3, 3,
"hr")))
+ }
+
test("update runtime group filtering") {
Seq(true, false).foreach { ddpEnabled =>
Seq(true, false).foreach { aqeEnabled =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
index ea5d01c77180..9dd664b9cb74 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
@@ -2094,6 +2094,47 @@ abstract class MergeIntoTableSuiteBase extends
RowLevelOperationSuiteBase
}
}
+ test("SPARK-60667: insert-only merge with type coercion handles metadata
columns correctly") {
+ withTempView("source") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 700, "dep": "hr" }
+ |""".stripMargin)
+
+ Seq(1, 2, 3, 4).toDF("pk").createOrReplaceTempView("source")
+
+ // the ON condition references a metadata column and needs type coercion
+ // (`t.salary < 600.5`), so the MERGE is resolved only in a later
iteration of the analyzer
+ val executedPlan = executeAndKeepPlan {
+ sql(
+ s"""MERGE INTO $tableNameAsString t
+ |USING source s
+ |ON t.pk = s.pk AND t._partition = 'hr' AND t.salary < 600.5
+ |WHEN NOT MATCHED THEN
+ | INSERT (pk, salary, dep) VALUES (s.pk, 0, 'new')
+ |""".stripMargin)
+ }
+
+ // the MERGE is rewritten into InsertOnlyMerge (an append) that writes
only data columns
+ val write = executedPlan.collectFirst {
+ case e: InsertOnlyMergeExec => e.write
+ }.getOrElse(fail("expected an InsertOnlyMergeExec in the executed plan"))
+ val append = write.toBatch.asInstanceOf[InMemoryBaseTable#Append]
+ assert(append.info.schema.fieldNames === table.schema.fieldNames)
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Seq(
+ Row(1, 100, "hr"), // unchanged (matched)
+ Row(2, 200, "software"), // unchanged
+ Row(3, 700, "hr"), // unchanged
+ Row(2, 0, "new"), // insert (not matched in hr)
+ Row(3, 0, "new"), // insert (not matched by salary)
+ Row(4, 0, "new"))) // insert
+ }
+ }
+
test("merge with NOT NULL checks") {
withTempView("source") {
createAndInitTable("pk INT NOT NULL, s STRUCT<n_i: INT NOT NULL, n_l:
LONG>, dep STRING",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]