[
https://issues.apache.org/jira/browse/SPARK-48701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-48701:
-----------------------------------
Labels: pull-request-available (was: )
> Make PandasMode collation-aware
> -------------------------------
>
> Key: SPARK-48701
> URL: https://issues.apache.org/jira/browse/SPARK-48701
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Uroš Bojanić (INACTIVE) -> @uros
> Priority: Major
> Labels: pull-request-available
>
> pandas_mode(the internal expression behind the Pandas API on Spark
> Series.mode() / DataFrame.mode()) silently returns wrong results on
> non-binary collated strings.
> It does not throw -- values equal under a collation
> (e.g. 'b' and 'B' under UTF8_LCASE) are counted as separate keys, so the
> reported mode(s) can be incorrect.
>
> {code:java}
> import pyspark.pandas as ps
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.master("local[2]").getOrCreate()
> # UTF8_LCASE-collated column: values a, a, b, B, B
> sdf = spark.sql("""
> SELECT CAST(c AS STRING COLLATE UTF8_L
> FROM VALUES ('a'), ('a'), ('b'), ('B'), ('B') AS t(c)
> """)
> psser = sdf.pandas_api()["s"]
> print(sorted(psser.mode().to_numpy().tolist()))
> {code}
> Under UTF8_LCASE, b and B are a -> 2 and b/B -> 3; the mode should be the
> single b/B group.
> *Actual* (current): ['B', 'a'] (2 modes) -- wrong; a=2 ties B=2 and b is
> counted separately.
> {*}Expected{*}: a single collation-equal mode (e.g. ['b']) with count 3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
