[
https://issues.apache.org/jira/browse/SPARK-59633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59633:
-----------------------------------
Labels: pull-request-available (was: )
> Make unary trim collation-aware for case-insensitive ICU collations
> -------------------------------------------------------------------
>
> Key: SPARK-59633
> URL: https://issues.apache.org/jira/browse/SPARK-59633
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Menelaos Karavelas
> Priority: Major
> Labels: pull-request-available
>
> Unary trim, ltrim, and rtrim remove only the literal ASCII space character
> regardless of the input collation. The explicit two-argument forms instead use
> ICU matching for ICU-collated operands. Under case-insensitive ICU collations
> these two forms disagree, because ICU treats Unicode space separators such as
> NBSP (U+00A0) as equivalent to ASCII space (U+0020) at the applicable
> collation
> strength.
>
> Example (Spark master):
>
> {code:java}
> WITH input AS (SELECT concat(chr(160), 'abc', chr(160)) AS value)
> SELECT trim(value COLLATE UNICODE_CI) AS unary,
> trim(value COLLATE UNICODE_CI, ' ' COLLATE UNICODE_CI) AS explicit
> FROM input;{code}
>
> Result:
> {code:java}
> unary | explicit
> --------+--------
> \u00A0abc\u00A0 | abc{code}
>
> The unary form keeps the surrounding NBSP characters while the explicit form
> removes them. Both forms should apply the same default (ASCII space) trim
> string
> with the same collation semantics.
>
> Fix: route unary trim through the ICU implementation with an ASCII-space trim
> string for case-insensitive ICU collations (UNICODE_CI, UNICODE_CI_AI, and
> their
> _RTRIM variants). Binary, lowercase, and case-sensitive ICU collations are
> unchanged. After the fix, both columns above return "abc".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
