charliec05 opened a new pull request, #59077: URL: https://github.com/apache/spark/pull/59077
### What changes were proposed in this pull request? When `spark.sql.mapKeyDedupPolicy=LAST_WIN`, have `SimplifyExtractValueOps` visit `CreateMap` entries in reverse order when rewriting a `GetMapValue` lookup into `CaseKeyWhen`. Reverse the key/value pairs together, not the individual expressions. The default `EXCEPTION` path is unchanged. Add an optimizer regression, SQL regressions with whole-stage codegen both enabled and disabled, and a SQL migration note. ### Why are the changes needed? `CASE` chooses the first matching branch, while `LAST_WIN` requires the last matching map entry. The existing rewrite visits entries in insertion order, so enabling the optimizer changes results. For example, with `spark.sql.mapKeyDedupPolicy=LAST_WIN`: ```sql SELECT id, map(id, id, id, id + 10)[id] AS value FROM range(3); ``` The old optimized query returns values `0, 1, 2`; the constructed map requires `10, 11, 12`. A final duplicate entry with a null value is also incorrectly ignored by the old rewrite. JIRA: [SPARK-59604](https://issues.apache.org/jira/browse/SPARK-59604). ### Does this PR introduce _any_ user-facing change? Yes. Optimized map lookups honor the configured `LAST_WIN` semantics, including a last matching value of null. Missing keys and null lookup keys still return null. Queries using the default deduplication policy are not changed by this patch. ### How was this patch tested? - Before the production fix, the new optimizer regression failed, and both SQL regressions (codegen on/off) failed with incorrect query results. - The SQL regression covers repeated dynamic keys, dynamic/literal key collisions, null values, distinct keys, missing keys, and null lookups. - `build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.ComplexTypesSuite"`: all 32 tests passed. - `build/sbt "sql/testOnly org.apache.spark.sql.DataFrameFunctionsSuite -- -z map"`: all 17 selected tests passed, including the new SQL regression in both codegen modes. - `dev/lint-scala`: ScalaStyle and ScalaFmt checks passed. Tests ran on Java 21. - [Full GitHub Actions CI](https://github.com/charliec05/spark/actions/runs/36294917126) is running on `26658c68feb`; its result is pending. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) This contribution is submitted under the project's Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
