Raviston6296 opened a new pull request, #60208: URL: https://github.com/apache/spark/pull/60208
### What changes were proposed in this pull request? This is a backport of #60156 to `branch-4.3`. It is on top of #60207, the `branch-4.3` backport of SPARK-60978 (#60155), so this PR shows both commits until #60207 is merged. With SPARK-60978 applied, the SPARK-60979 commit cherry-picks cleanly. Without it, the only conflict is in `UpdateTableSuiteBase`, where both PRs add a test at the same place. `SchemaPruning.pruneSchema` now matches requested fields by exact name first. It falls back to the resolver only when no field of the schema has the exact name of the requested field. `sortLeftFieldsByRight` does the same instead of using only the lowercase-keyed map. Serializer pruning in `ObjectSerializerPruning` follows the same rule. ### Why are the changes needed? This is a correctness issue in released versions, including 4.3.0. `branch-4.3` has the same code as master had before #60156. If a schema has fields whose names differ only by case, for example `salary INT NOT NULL, SALARY STRING`, the pruned schema in a case-insensitive session (the default) can take the type and nullability of the wrong field: ``` before: pk INT NOT NULL, salary STRING, SALARY STRING after: pk INT NOT NULL, salary INT NOT NULL, SALARY STRING ``` A DSv2 scan is then asked for the wrong type and nullability. This can store wrong values in DSv2 row-level operations, or fail them with `PLAN_VALIDATION_FAILED_RULE_IN_BATCH` or a Janino `CompileException`. See #60156 for details. ### Does this PR introduce _any_ user-facing change? Yes, a bug fix, only when a schema has columns whose names differ only by case and the session is case-insensitive. The data source is now asked for the correct type and nullability of each column. Nothing changes when no two names collide under case folding. ### How was this patch tested? The test cases from #60156 were run on `branch-4.3` with JDK 17: ``` build/sbt 'catalyst/testOnly *expressions.SchemaPruningSuite *ObjectSerializerPruningSuite' build/sbt 'sql/testOnly *DatasetOptimizationSuite *GroupBasedUpdateTableSuite *DeltaBasedUpdateTableSuite *DeltaBasedUpdateAsDeleteAndInsertTableSuite -- -z SPARK-60979' ``` - With this PR, `SchemaPruningSuite` and `ObjectSerializerPruningSuite` pass (17 tests), and the 5 new sql tests pass: 2 in `DatasetOptimizationSuite` and 1 in each of the three UPDATE suites. - Without the main code change, all 7 new tests fail: 2 catalyst and 5 sql. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
