dongjoon-hyun opened a new pull request, #59930:
URL: https://github.com/apache/spark/pull/59930
### What changes were proposed in this pull request?
This PR makes DSv2 UPDATE and MERGE fail analysis when an assignment targets
a metadata column of the target table, instead of silently ignoring the
assignment.
The assignment keys of UPDATE and MERGE are resolved against the target
table including its metadata columns, so a key such as `_partition` can resolve
to a metadata column. `ResolveRowLevelCommandAssignments` then aligns the
assignments with the table attributes using
`AssignmentUtils.alignUpdateAssignments` and
`AssignmentUtils.alignInsertAssignments`, which produce one value per table
attribute, so an assignment to a metadata column is dropped.
Both alignment methods now report assignments whose key refers to a metadata
column with the existing
`DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS` error, which the
alignment already uses for other invalid assignments such as multiple
assignments to the same column. A key refers to a metadata column if its
attribute carries the metadata column marker (`MetadataAttribute.isValid`), so
the check does not depend on expression IDs and also applies when
`AddMetadataColumns` has already added the metadata columns to the target
output.
### Why are the changes needed?
The values of metadata columns are produced by the data source. DSv2
row-level operations pass the metadata values they read (or nulls, see
`MetadataColumn.PRESERVE_ON_UPDATE`) back to the writer and have no way to
write a value that a statement assigns to a metadata column. However, DSv2
UPDATE and MERGE statements that assign to a metadata column succeed, apply the
other assignments and silently drop the assigned value. For instance, with the
in-memory row-level table of `RowLevelOperationSuiteBase` (`pk INT NOT NULL, id
INT, dep STRING`, partitioned by `dep`, with the metadata columns `_partition`
and `index`):
```sql
UPDATE cat.ns1.test_table SET id = -1, index = 5 WHERE pk = 1
```
sets `id` to `-1` and ignores `index = 5`. The same happens with:
- `WHEN MATCHED` and `WHEN NOT MATCHED BY SOURCE` `UPDATE SET` clauses and
`WHEN NOT MATCHED` `INSERT` clauses of MERGE, including insert-only MERGE.
- The DataFrame merge API, e.g. `whenNotMatched().insert(...)` with
`"_partition" -> lit("x")`.
- `MERGE WITH SCHEMA EVOLUTION` with `UPDATE SET *` or `INSERT *` when the
source has a column named like a metadata column of the target, e.g. `index`.
The expanded assignment resolves to the metadata column, so the schema is not
evolved and the values of the source column are dropped.
When the command is resolved in a later analyzer iteration, e.g. because of
an implicit cast in a condition such as `WHERE id < 1.5`, `AddMetadataColumns`
has already added the metadata columns to the target output (see SPARK-60667).
Then the assigned value reaches the writer as an extra data column and, if the
operation requires metadata columns, as the metadata value of the row, or the
query fails at runtime with an internal error (`ArrayIndexOutOfBoundsException`
for an UPDATE that splits updates into deletes and inserts, `AssertionError`
for MERGE).
All of these happen with group-based and delta-based operations, including
operations that split updates and operations that require no metadata columns.
For comparison, `INSERT INTO cat.ns1.test_table (pk, id, dep, index) VALUES
(...)` already fails with `INSERT_COLUMN_ARITY_MISMATCH.TOO_MANY_DATA_COLUMNS`.
### Does this PR introduce _any_ user-facing change?
Yes. DSv2 UPDATE and MERGE statements that assign to a metadata column of
the target table now fail instead of ignoring the assignment. For the example
above:
```
[DATATYPE_MISMATCH.INVALID_ROW_LEVEL_OPERATION_ASSIGNMENTS] Cannot resolve
"id = -1", "index = 5" due to data type mismatch:
- Cannot assign to metadata column 'index' SQLSTATE: 42K09
```
`MERGE WITH SCHEMA EVOLUTION` with `UPDATE SET *` or `INSERT *` and a source
column named like a metadata column of the target fails the same way, so the
source column has to be renamed or dropped. Without the evolution clause,
`UPDATE SET *` and `INSERT *` still ignore such a source column as before.
By code reading, the assignments have been dropped since the alignment of
UPDATE and MERGE assignments was added in Apache Spark
[3.5.0](https://github.com/apache/spark/releases/tag/v3.5.0) (2023-09-09), and
the `UPDATE SET *` and `INSERT *` case since MERGE schema evolution was added
in Apache Spark [4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0)
(2025-12-11).
### How was this patch tested?
Added tests:
- `UpdateTableSuiteBase`: an unqualified and a qualified assignment to a
metadata column, and an UPDATE resolved in a later analyzer iteration.
- `MergeIntoTableSuiteBase`: `WHEN MATCHED` and `WHEN NOT MATCHED BY SOURCE`
updates, a `WHEN NOT MATCHED` insert in a MERGE with an update clause and in an
insert-only MERGE, and a MERGE resolved in a later analyzer iteration.
- `MergeIntoDataFrameSuite`: the same through the DataFrame API `update` and
`insert`.
- `MergeIntoSchemaEvolutionExtraSourceColumnTests`: `UPDATE SET *` with
`INSERT *`, and insert-only `INSERT *`, with a source column named `index` or
`_partition`. Without the evolution clause, the source column is ignored as
before.
They run in the group-based and delta-based suites, including the split
update, NoMetadata and NoCodegen variants and the SQL and DataFrame schema
evolution suites. Without the fix, all 21 tests that expect the error fail
because the statements succeed; the 8 schema evolution runs without the
evolution clause pass with and without the fix.
```
build/sbt "sql/testOnly *UpdateTableSuite *UpdateAsDeleteAndInsertTableSuite
*MergeIntoTableSuite *MergeIntoTableNoCodegenSuite
*MergeIntoTableUpdateAsDeleteAndInsertSuite *MergeIntoDataFrameSuite
*MergeIntoSchemaEvolution*Suite -- -z SPARK-60721"
```
Also ran the DSv2 DELETE, UPDATE and MERGE suites (including the NoMetadata,
split update and NoCodegen variants), the MERGE schema evolution suites,
`MergeIntoDataFrameSuite`, the row-level runtime filter suites,
`MetadataColumnSuite`, `PlanResolutionSuite`, `AlignUpdateAssignmentsSuite` and
`AlignMergeAssignmentsSuite` locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
