charliec05 opened a new pull request, #59079: URL: https://github.com/apache/spark/pull/59079
### What changes were proposed in this pull request? Make `FileTable.mergedOptions` remove table options overridden by operation options using case-insensitive key comparison. Preserve the original spelling of the remaining table keys and operation keys, including keys that may be used as case-sensitive Hadoop configuration properties. Extend the read/write option-merging regression across ORC, Parquet, CSV, JSON, and text, covering identical casing and both directions of mixed casing. The tests also check the case-sensitive representation and verify that neither input map is mutated. Add a SQL migration note. This continues [SPARK-57472](https://issues.apache.org/jira/browse/SPARK-57472) after #56520 was closed as stale. Thanks to the earlier contributor for identifying the problem and proposing the case-insensitive filtering approach. There is no active PR for that ticket. ### Why are the changes needed? Table options and operation options currently merge as case-sensitive maps. For example, table `k2=table-value` and operation `K2=operation-value` both survive that merge. Constructing the resulting `CaseInsensitiveStringMap` then chooses a value based on map iteration order, rather than the intended operation-over-table precedence. ### Does this PR introduce _any_ user-facing change? Yes. Operation options deterministically take precedence even when their key casing differs from the table options. Same-casing precedence, unrelated options, and original key spelling remain unchanged. ### How was this patch tested? - On unmodified production code, the expanded regression ran 15 cases: the 5 same-casing cases passed and all 10 mixed-casing cases failed. - With the fix, `build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.v2.FileTableSuite"`: all 18 tests passed on Java 21. - `dev/lint-scala`: ScalaStyle and ScalaFmt checks passed. - [Full GitHub Actions CI](https://github.com/charliec05/spark/actions/runs/36294918211) is running on `0483c47c0f6`; its result is pending. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) This contribution is submitted under the project's Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
