dongjoon-hyun opened a new pull request, #60155:
URL: https://github.com/apache/spark/pull/60155
### What changes were proposed in this pull request?
This PR aims to fix a correctness issue of DSv2 row-level DELETE, UPDATE and
MERGE. `RewriteRowLevelCommand.findColOrdinal` now looks for the column with
exactly the given name first, and falls back to `conf.resolver` only when there
is no such column.
The fallback is still needed because a connector may refer to a column in a
different case, e.g. the row ID `pk` of a column `PK`.
`V2ExpressionUtils.resolveRef` gives the resolved attribute the name of the
reference.
### Why are the changes needed?
This is a correctness issue. When a table has columns whose names differ
only by case, such as `salary` and `SALARY`, an UPDATE, a MERGE or a
group-based DELETE in a case-insensitive session, which is the default, writes
the value of one column into the other without any error. The statement does
not need to refer to these columns, so nothing is ambiguous.
`RewriteRowLevelCommand` builds the row projections of `ReplaceData` and
`WriteDelta` by looking up the ordinal of each table column in the query output
by name. `findColOrdinal` returned the first column that `conf.resolver`
matches, so `SALARY` got the ordinal of `salary`. For example, the row
projection of the UPDATE below uses the ordinals `[1, 2, 2, 4]` instead of `[1,
2, 3, 4]`.
```scala
//
spark.sql.catalog.cat=org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog
// Create the table in a case-sensitive session.
spark.conf.set("spark.sql.caseSensitive", "true")
sql("CREATE TABLE cat.ns1.t (pk INT NOT NULL, salary INT, SALARY INT, dep
STRING) PARTITIONED BY (dep)")
sql("INSERT INTO cat.ns1.t VALUES (1, 100, 200, 'hr'), (2, 300, 400, 'hr')")
spark.conf.set("spark.sql.caseSensitive", "false")
// Issue 1: It should keep `SALARY`, but the table ends up with (3, 100,
100, hr) and (2, 300, 300, hr).
sql("UPDATE cat.ns1.t SET pk = 3 WHERE pk = 1")
// Issue 2: On the same initial table, it should keep (2, 300, 400, hr), but
the table ends up with
// (2, 300, 300, hr).
sql("DELETE FROM cat.ns1.t WHERE pk <= 1")
// Issue 3: On the same initial table, it should keep `SALARY`, but the
table ends up with
// (1, 100, 100, x) and (2, 300, 300, hr).
Seq(1).toDF("pk").createOrReplaceTempView("src")
sql("MERGE INTO cat.ns1.t t USING src s ON t.pk = s.pk WHEN MATCHED THEN
UPDATE SET dep = 'x'")
```
The results above are read with `spark.sql.caseSensitive=true`.
- Group-based operations write wrong values into both the updated rows and
the rows that they copy over. Delta-based operations write wrong values into
the updated rows.
- Delta-based DELETE writes no data columns, and an insert-only MERGE does
not use these projections, so they are not affected.
- If `SALARY` is `NOT NULL` and `salary` is nullable, the UPDATE fails with
a misleading `INSERT_COLUMN_ARITY_MISMATCH.TOO_MANY_DATA_COLUMNS` error instead.
- The lookup was added by
[SPARK-43324](https://issues.apache.org/jira/browse/SPARK-43324)
([v3.5.0](https://github.com/apache/spark/releases/tag/v3.5.0), 2023-09-09) for
delta-based operations.
[SPARK-50820](https://issues.apache.org/jira/browse/SPARK-50820)
([v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0), 2025-05-19)
started to use it for group-based operations.
### Does this PR introduce _any_ user-facing change?
Yes. An UPDATE, a MERGE or a group-based DELETE on a table whose column
names differ only by case now writes each column's own value, and the `NOT
NULL` case above succeeds. This is a long-standing issue in the released
versions, including
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
### How was this patch tested?
Pass the CIs with newly added test cases.
- `SPARK-60978: update keeps columns whose names differ only by case` fails
without this PR in the five UPDATE suites.
- `SPARK-60978: update with a row ID column whose name differs in case`
checks the fallback to `conf.resolver` in the delta-based UPDATE suites. It
fails if `findColOrdinal` matches exact names only.
```
$ build/sbt 'sql/testOnly *BasedUpdateTableSuite
*BasedNoMetadataUpdateTableSuite *DeltaBasedUpdateAsDeleteAndInsertTableSuite
*BasedDeleteFromTableSuite *BasedNoMetadataDeleteFromTableSuite
*BasedMergeIntoTableSuite *BasedNoMetadataMergeIntoTableSuite
*DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite *MergeIntoDataFrameSuite
*BasedRowLevelOperationCatalystRuntimeFilterSuite'
$ build/sbt 'sql/testOnly *BasedMergeIntoSchemaEvolution*Suite
*AlignUpdateAssignmentsSuite *AlignMergeAssignmentsSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
