sepuri sai krishna created SPARK-61622:
------------------------------------------
Summary: Joins and IN on UNICODE_AI strings match a sharp s with
"ss", but = does not
Key: SPARK-61622
URL: https://issues.apache.org/jira/browse/SPARK-61622
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3
Reporter: sepuri sai krishna
Under the case-sensitive, accent-insensitive ICU collations ({{UNICODE_AI}},
{{en_AI}}, {{de_AI}}, ..., and their {{_RTRIM}} variants), an equi-join returns
rows for which its own join condition is false, and equality is not transitive.
Default configuration, reproduced on 4.0.3 (ICU 76.1) and on master (ICU 78.3).
{code:sql}
CREATE TEMPORARY VIEW t AS SELECT id, CAST(s AS STRING COLLATE de_AI) AS s
FROM VALUES (1, 'Straße'), (2, 'Strasse') AS v(id, s);
SELECT a.id, b.id, a.s = b.s FROM t a CROSS JOIN t b;
-- (1,1,true) (1,2,false) (2,1,false) (2,2,true)
SELECT a.id, b.id FROM t a JOIN t b ON a.s = b.s;
-- (1,1) (1,2) (2,1) (2,2): the pairs (1,2) and (2,1) do not satisfy a.s = b.s
SELECT count(DISTINCT s) FROM t; -- 2
SELECT id FROM t WHERE s IN ('Strasse', 'x'); -- 2
SELECT id FROM t WHERE s IN ('Strasse', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6',
'x7', 'x8', 'x9', 'x10');
-- 1 and 2 on master (InSet); 2 on 4.0.3
-- equality is not transitive: U+FF53 is the fullwidth s
SELECT 'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI, -- false
'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI, -- true
'ss' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI; -- true
{code}
*Cause*
Spark compares the strings of an ICU collation in two ways: with
{{Collator.compare}} (the comparator of the collation: {{=}}, {{<}}, sorting,
sort-based aggregation, {{In}}), and by their collation keys
({{Collator.getCollationKey}}, the sort key of the collation: hash joins, hash
aggregation, shuffles, {{InSet}}). The two have to agree.
The CS_AI collations are built with primary strength and the case level
({{ks=level1}}, {{kc=true}} in
{{CollationFactory.CollationSpecICU.buildCollation}}). For this setting the two
do not agree in ICU4J when one of the strings contains U+00DF (sharp s). Plain
icu4j 78.3, without Spark:
{noformat}
ks=level1 kc=true
"ß" vs "ss" compare = 1 collation keys equal
"ß" vs "SS" compare = 1 collation key of "ß" is less
"ßあ" vs "ssあ" compare = 0 collation keys equal
"ßあ" vs "SSあ" compare = -1 collation key of "ßあ" is less
{noformat}
This is a bug of the fast path that {{Collator.compare}} takes for Latin text
({{com.ibm.icu.impl.coll.CollationFastLatin}}). A character that maps to two
collation elements is held there as two 16-bit "mini CEs" in one {{int}}, and
{{getCases}} (also {{getSecondaries}}, {{getTertiaries}} and
{{getQuaternaries}}) tells one element from two with {{if (pair <= 0xffff)}}.
The code is a port of ICU4C, where {{pair}} is a {{uint32_t}}. In Java the
{{int}} is negative when the second mini CE has its top bit set, which is the
case for a sharp s (s + s) in the fast Latin table of every locale, so the two
elements are taken for one and their case weights are compared unmasked. ICU4C
returns equal for the same strings ({{ucol_strcoll}}, checked with 74.2), and
so does ICU4J when a character outside the fast path is appended to both
strings (the last two lines above). So the collation keys are right: at the
primary level a sharp s is "ss", and it is lowercase. The primary level of the
fast path masks the value first and is not affected.
Among all pairs of one- and two-character strings over the letters up to U+024F
(36.0 million pairs), {{compare}} and the collation keys disagree on the
equality of two strings only for this setting, and only when one of them
contains U+00DF. With the other strengths (UNICODE, UNICODE_CI, UNICODE_CI_AI)
equality always agrees.
SPARK-49667 already disallowed the CS_AI collations in the functions that use
the ICU {{StringSearch}}. Equality, ordering, grouping, joins and IN still
accept them.
*Fix*
For the CS_AI collations, order the strings with a collator of primary strength
(the one the CI_AI collations use, whose fast path is right), and compare the
strings that are equal at the primary level by their collation keys. Then the
comparator and the sort key agree by construction, and only the strings that
differ by case at most pay for the collation keys. The other collations keep
{{Collator.compare}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]