Raviston6296 opened a new pull request, #60207:
URL: https://github.com/apache/spark/pull/60207
### What changes were proposed in this pull request?
This is a backport of #60155 to `branch-4.3`. The commit cherry-picks
cleanly.
`RewriteRowLevelCommand.findColOrdinal` now looks for a column with exactly
the given name first. It falls back to `conf.resolver` only when there is no
such column.
### Why are the changes needed?
This is a correctness issue in released versions, including 4.3.0. If a
table has columns whose names differ only by case, such as `salary` and
`SALARY`, a DSv2 UPDATE, a MERGE or a group-based DELETE in a case-insensitive
session (the default) silently writes one column's value into the other.
`branch-4.3` has the same `findColOrdinal` as master had before #60155, so the
same issue exists here.
```scala
//
spark.sql.catalog.cat=org.apache.spark.sql.connector.catalog.InMemoryRowLevelOperationTableCatalog
spark.conf.set("spark.sql.caseSensitive", "true")
sql("CREATE TABLE cat.ns1.t (pk INT NOT NULL, salary INT, SALARY INT, dep
STRING) PARTITIONED BY (dep)")
sql("INSERT INTO cat.ns1.t VALUES (1, 100, 200, 'hr'), (2, 300, 400, 'hr')")
spark.conf.set("spark.sql.caseSensitive", "false")
// It should keep `SALARY`, but the table ends up with (3, 100, 100, hr) and
(2, 300, 300, hr).
sql("UPDATE cat.ns1.t SET pk = 3 WHERE pk = 1")
```
See #60155 for the full description, including DELETE and MERGE.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix. An UPDATE, a MERGE or a group-based DELETE on a table whose
column names differ only by case now writes each column's own value.
### How was this patch tested?
The test cases from #60155 were run on `branch-4.3` with JDK 17:
```
build/sbt 'sql/testOnly *GroupBasedDeleteFromTableSuite
*DeltaBasedDeleteFromTableSuite *GroupBasedUpdateTableSuite
*DeltaBasedUpdateTableSuite *DeltaBasedUpdateAsDeleteAndInsertTableSuite
*GroupBasedMergeIntoTableSuite *DeltaBasedMergeIntoTableSuite
*DeltaBasedMergeIntoTableUpdateAsDeleteAndInsertSuite -- -z SPARK-60978'
```
- With this PR, all 10 tests pass.
- Without the `RewriteRowLevelCommand` change, 7 fail: the UPDATE and MERGE
tests, and the group-based DELETE test. The delta-based DELETE test passes
either way, because delta-based DELETE writes no data columns. The row-ID test
passes either way too, because it checks the `conf.resolver` fallback. Both
match #60155.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
