[ 
https://issues.apache.org/jira/browse/SPARK-61622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61622:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> Joins and IN on UNICODE_AI strings match a sharp s with "ss", but = does not
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-61622
>                 URL: https://issues.apache.org/jira/browse/SPARK-61622
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.3
>            Reporter: sepuri sai krishna
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> Under the case-sensitive, accent-insensitive ICU collations ({{UNICODE_AI}}, 
> {{en_AI}}, {{de_AI}}, ..., and their {{_RTRIM}} variants), an equi-join 
> returns rows for which its own join condition is false, and equality is not 
> transitive. Default configuration, reproduced on 4.0.3 (ICU 76.1) and on 
> master (ICU 78.3).
> {code:sql}
> CREATE TEMPORARY VIEW t AS SELECT id, CAST(s AS STRING COLLATE de_AI) AS s
>   FROM VALUES (1, 'Straße'), (2, 'Strasse') AS v(id, s);
> SELECT a.id, b.id, a.s = b.s FROM t a CROSS JOIN t b;
> -- (1,1,true) (1,2,false) (2,1,false) (2,2,true)
> SELECT a.id, b.id FROM t a JOIN t b ON a.s = b.s;
> -- (1,1) (1,2) (2,1) (2,2): the pairs (1,2) and (2,1) do not satisfy a.s = b.s
> SELECT count(DISTINCT s) FROM t;                    -- 2
> SELECT id FROM t WHERE s IN ('Strasse', 'x');       -- 2
> SELECT id FROM t WHERE s IN ('Strasse', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6', 
> 'x7', 'x8', 'x9', 'x10');
> -- 1 and 2 on master (InSet); 2 on 4.0.3
> -- equality is not transitive: U+FF53 is the fullwidth s
> SELECT 'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI,   -- false
>   'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI,       -- true
>   'ss' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI;      -- true
> {code}
> *Cause*
> Spark compares the strings of an ICU collation in two ways: with 
> {{Collator.compare}} (the comparator of the collation: {{=}}, {{<}}, sorting, 
> sort-based aggregation, {{In}}), and by their collation keys 
> ({{Collator.getCollationKey}}, the sort key of the collation: hash joins, 
> hash aggregation, shuffles, {{InSet}}). The two have to agree.
> The CS_AI collations are built with primary strength and the case level 
> ({{ks=level1}}, {{kc=true}} in 
> {{CollationFactory.CollationSpecICU.buildCollation}}). For this setting the 
> two do not agree in ICU4J when one of the strings contains U+00DF (sharp s). 
> Plain icu4j 78.3, without Spark:
> {noformat}
> ks=level1 kc=true
>   "ß"  vs "ss"    compare = 1    collation keys equal
>   "ß"  vs "SS"    compare = 1    collation key of "ß" is less
>   "ßあ" vs "ssあ"  compare = 0    collation keys equal
>   "ßあ" vs "SSあ"  compare = -1   collation key of "ßあ" is less
> {noformat}
> This is a bug of the fast path that {{Collator.compare}} takes for Latin text 
> ({{com.ibm.icu.impl.coll.CollationFastLatin}}). A character that maps to two 
> collation elements is held there as two 16-bit "mini CEs" in one {{int}}, and 
> {{getCases}} (also {{getSecondaries}}, {{getTertiaries}} and 
> {{getQuaternaries}}) tells one element from two with {{if (pair <= 0xffff)}}. 
> The code is a port of ICU4C, where {{pair}} is a {{uint32_t}}. In Java the 
> {{int}} is negative when the second mini CE has its top bit set, which is the 
> case for a sharp s (s + s) in the fast Latin table of every locale, so the 
> two elements are taken for one and their case weights are compared unmasked. 
> ICU4C returns equal for the same strings ({{ucol_strcoll}}, checked with 
> 74.2), and so does ICU4J when a character outside the fast path is appended 
> to both strings (the last two lines above). So the collation keys are right: 
> at the primary level a sharp s is "ss", and it is lowercase. The primary 
> level of the fast path masks the value first and is not affected.
> Among all pairs of one- and two-character strings over the letters up to 
> U+024F (36.0 million pairs), {{compare}} and the collation keys disagree on 
> the equality of two strings only for this setting, and only when one of them 
> contains U+00DF. With the other strengths (UNICODE, UNICODE_CI, 
> UNICODE_CI_AI) equality always agrees.
> SPARK-49667 already disallowed the CS_AI collations in the functions that use 
> the ICU {{StringSearch}}. Equality, ordering, grouping, joins and IN still 
> accept them.
> *Fix*
> For the CS_AI collations, order the strings with a collator of primary 
> strength (the one the CI_AI collations use, whose fast path is right), and 
> compare the strings that are equal at the primary level by their collation 
> keys. Then the comparator and the sort key agree by construction, and only 
> the strings that differ by case at most pay for the collation keys. The other 
> collations keep {{Collator.compare}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to