[
https://issues.apache.org/jira/browse/SPARK-59633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Menelaos Karavelas updated SPARK-59633:
---------------------------------------
Description:
Unary trim, ltrim, and rtrim remove only the literal ASCII space character
regardless of the input collation. The explicit two-argument forms instead use
ICU matching for ICU-collated operands. Under case-insensitive ICU collations
these two forms disagree, because ICU treats Unicode space separators such as
NBSP (U+00A0) as equivalent to ASCII space (U+0020) at the applicable collation
strength.
Example (Spark master):
```
WITH input AS (SELECT concat(chr(160), 'abc', chr(160)) AS value)
SELECT trim(value COLLATE UNICODE_CI) AS unary,
trim(value COLLATE UNICODE_CI, ' ' COLLATE UNICODE_CI) AS explicit
FROM input;
```
```
unary | explicit
---------+---------
\u00A0abc\u00A0 | abc
```
The unary form keeps the surrounding NBSP characters while the explicit form
removes them. Both forms should apply the same default (ASCII space) trim string
with the same collation semantics.
Fix: route unary trim through the ICU implementation with an ASCII-space trim
string for case-insensitive ICU collations (UNICODE_CI, UNICODE_CI_AI, and their
_RTRIM variants). Binary, lowercase, and case-sensitive ICU collations are
unchanged. After the fix, both columns above return "abc".
> Make unary trim collation-aware for case-insensitive ICU collations
> -------------------------------------------------------------------
>
> Key: SPARK-59633
> URL: https://issues.apache.org/jira/browse/SPARK-59633
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Menelaos Karavelas
> Priority: Major
>
> Unary trim, ltrim, and rtrim remove only the literal ASCII space character
> regardless of the input collation. The explicit two-argument forms instead use
> ICU matching for ICU-collated operands. Under case-insensitive ICU collations
> these two forms disagree, because ICU treats Unicode space separators such as
> NBSP (U+00A0) as equivalent to ASCII space (U+0020) at the applicable
> collation
> strength.
>
> Example (Spark master):
>
> ```
> WITH input AS (SELECT concat(chr(160), 'abc', chr(160)) AS value)
> SELECT trim(value COLLATE UNICODE_CI) AS unary,
> trim(value COLLATE UNICODE_CI, ' ' COLLATE UNICODE_CI) AS explicit
> FROM input;
> ```
>
> ```
> unary | explicit
> ---------+---------
> \u00A0abc\u00A0 | abc
> ```
>
> The unary form keeps the surrounding NBSP characters while the explicit form
> removes them. Both forms should apply the same default (ASCII space) trim
> string
> with the same collation semantics.
>
> Fix: route unary trim through the ICU implementation with an ASCII-space trim
> string for case-insensitive ICU collations (UNICODE_CI, UNICODE_CI_AI, and
> their
> _RTRIM variants). Binary, lowercase, and case-sensitive ICU collations are
> unchanged. After the fix, both columns above return "abc".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
