Eric Yang created SPARK-58096:
---------------------------------
Summary: approx_top_k returns wrong counts for ICU-collated string
columns (distinct non-ASCII values merged)
Key: SPARK-58096
URL: https://issues.apache.org/jira/browse/SPARK-58096
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Eric Yang
Over an ICU collation (UNICODE, UNICODE_CI, or a locale collation),
approx_top_k counts DISTINCT non-ASCII string values as a single item. It
returns fewer items than actually exist, with inflated (summed) counts -- a
silent wrong result for the frequency estimates.
*Repro* (three distinct characters, so the correct top-k has 3 items with
counts 3, 2, 1):
{code:sql}
SELECT approx_top_k(c, 10)
FROM (SELECT CAST(col AS STRING COLLATE UNICODE_CI) AS c
FROM VALUES ('且'),('且'),('且'),('丕'),('丕'),('世') AS t(col));
-- returns a SINGLE item with count 6, instead of 3 items with counts 3, 2, 1
{code}
Impact: wrong frequency counts and missing distinct items for any non-ASCII
data under
UNICODE / UNICODE_CI / locale collations. UTF8_BINARY and UTF8_LCASE are not
affected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
