[
https://issues.apache.org/jira/browse/SPARK-59043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun resolved SPARK-59043.
------------------------------
Fix Version/s: 4.3.0
5.0.0
Resolution: Fixed
Issue resolved by pull request 58376
[https://github.com/apache/spark/pull/58376]
> SimplifyCaseConversionExpressions changes Unicode case-conversion semantics
> ---------------------------------------------------------------------------
>
> Key: SPARK-59043
> URL: https://issues.apache.org/jira/browse/SPARK-59043
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark 5.0.0-SNAPSHOT, commit
> 9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17
> Reporter: Yibo Dong
> Assignee: AnhTris
> Priority: Major
> Labels: optimizer, pull-request-available, sql
> Fix For: 4.3.0, 5.0.0
>
>
> A legal nested Unicode case-conversion expression returns different results
> depending on whether SimplifyCaseConversionExpressions is enabled.
> h3. Reproduction
> {code:sql}
> SELECT lower(upper(s2)) AS result
> FROM (VALUES ('ı')) AS t(s)
> LATERAL VIEW explode(array(s)) e AS s2;
> {code}
> With the default optimizer configuration, the query returns:
> {code}
> ı
> {code}
> If only SimplifyCaseConversionExpressions is excluded:
> {code}
> --conf
> spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions
> {code}
> the same query returns:
> {code}
> i
> {code}
> h3. Expected behavior
> Both configurations should preserve the semantics of the original expression
> and
> return:
> {code}
> i
> {code}
> For the Unicode character U+0131 LATIN SMALL LETTER DOTLESS I:
> {code}
> upper('ı') = 'I'
> lower('I') = 'i'
> {code}
> Therefore:
> {code}
> lower(upper('ı')) = 'i'
> {code}
> h3. Actual behavior
> With SimplifyCaseConversionExpressions enabled, Spark returns `ı`.
> The optimizer rule simplifies:
> {code}
> lower(upper(child))
> {code}
> to:
> {code}
> lower(child)
> {code}
> However, this transformation is not semantics-preserving for all Unicode
> characters. For this input:
> {code}
> lower(upper('ı')) = 'i'
> lower('ı') = 'ı'
> {code}
> h3. Additional information
> The only configuration change needed to avoid the incorrect result is
> excluding
> SimplifyCaseConversionExpressions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
