AnhTtis opened a new pull request, #58376:
URL: https://github.com/apache/spark/pull/58376
### What changes were proposed in this pull request?
In Catalyst Optimizer, the rule `SimplifyCaseConversionExpressions`
previously simplified nested mixed case conversions:
- `Upper(Lower(child))` -> `Upper(child)`
- `Lower(Upper(child))` -> `Lower(child)`
However, in the Unicode standard and Java's case mapping semantics
(`UTF8String`), mixed case conversion is not idempotent and not symmetric for
several Unicode characters:
- For `ı` (U+0131 LATIN SMALL LETTER DOTLESS I): `upper('ı')` = `'I'`,
`lower('I')` = `'i'`. Therefore `lower(upper('ı'))` = `'i'`, but `lower('ı')` =
`'ı'`.
- For `µ` (U+00B5 MICRO SIGN): `upper('µ')` = `'Μ'` (U+039C), `lower('Μ')` =
`'μ'` (U+03BC). Therefore `lower(upper('µ'))` = `'μ'`, but `lower('µ')` = `'µ'`.
- For `ß` (U+00DF LATIN SMALL LETTER SHARP S): `upper('ß')` = `'SS'`,
`lower('SS')` = `'ss'`. Therefore `lower(upper('ß'))` = `'ss'`, but
`lower('ß')` = `'ß'`.
- For `K` (U+212A KELVIN SIGN): `lower('K')` = `'k'`, `upper('k')` = `'K'`.
Therefore `upper(lower('K'))` = `'K'`, but `upper('K')` = `'K'`.
Consequently, simplifying `Lower(Upper(child))` to `Lower(child)` or
`Upper(Lower(child))` to `Upper(child)` leads to silent data correctness bugs
and causes queries to return different results depending on whether
`SimplifyCaseConversionExpressions` is enabled.
This PR fixes the issue by removing the mixed case simplification rules from
`SimplifyCaseConversionExpressions`, retaining only same-case idempotent
transformations (`Upper(Upper(child))` -> `Upper(child)` and
`Lower(Lower(child))` -> `Lower(child)`), which are 100% idempotent across all
Unicode code points.
Fixes [SPARK-59043](https://issues.apache.org/jira/browse/SPARK-59043).
### Why are the changes needed?
To prevent incorrect query results and maintain Unicode case-conversion
semantic correctness under SQL expression optimization.
### Does this PR introduce _any_ user-facing change?
Yes. Queries with nested mixed case conversions (e.g. `lower(upper(str))`)
on Unicode characters now correctly preserve Unicode semantics and return
consistent results regardless of optimizer configuration.
### How was this patch tested?
- Updated optimizer unit tests in `SimplifyStringCaseConversionSuite.scala`
verifying that mixed case expressions are preserved and same-case expressions
are simplified.
- Added end-to-end SQL regression tests in `StringFunctionsSuite.scala`
covering Unicode edge cases (`'ı'`, `'µ'`, `'ß'`, `'K'`) under both default
optimizer configuration and with rule excluded.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
