[
https://issues.apache.org/jira/browse/SPARK-48701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-48701:
-----------------------------
Description:
pandas_mode(the internal expression behind the Pandas API on Spark
Series.mode() / DataFrame.mode()) silently returns wrong results on non-binary
collated strings.
It does not throw -- values equal under a collation
(e.g. 'b' and 'B' under UTF8_LCASE) are counted as separate keys, so the
reported mode(s) can be incorrect.
{code:java}
import pyspark.pandas as ps
from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[2]").getOrCreate()
# UTF8_LCASE-collated column: values a, a, b, B, B
sdf = spark.sql("""
SELECT CAST(c AS STRING COLLATE UTF8_L
FROM VALUES ('a'), ('a'), ('b'), ('B'), ('B') AS t(c)
""")
psser = sdf.pandas_api()["s"]
print(sorted(psser.mode().to_numpy().tolist()))
{code}
Under UTF8_LCASE, b and B are a -> 2 and b/B -> 3; the mode should be the
single b/B group.
*Actual* (current): ['B', 'a'] (2 modes) -- wrong; a=2 ties B=2 and b is
counted separately.
{*}Expected{*}: a single collation-equal mode (e.g. ['b']) with count 3.
> PandasMode (all collations)
> ---------------------------
>
> Key: SPARK-48701
> URL: https://issues.apache.org/jira/browse/SPARK-48701
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Uroš Bojanić (INACTIVE) -> @uros
> Priority: Major
>
> pandas_mode(the internal expression behind the Pandas API on Spark
> Series.mode() / DataFrame.mode()) silently returns wrong results on
> non-binary collated strings.
> It does not throw -- values equal under a collation
> (e.g. 'b' and 'B' under UTF8_LCASE) are counted as separate keys, so the
> reported mode(s) can be incorrect.
>
> {code:java}
> import pyspark.pandas as ps
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.master("local[2]").getOrCreate()
> # UTF8_LCASE-collated column: values a, a, b, B, B
> sdf = spark.sql("""
> SELECT CAST(c AS STRING COLLATE UTF8_L
> FROM VALUES ('a'), ('a'), ('b'), ('B'), ('B') AS t(c)
> """)
> psser = sdf.pandas_api()["s"]
> print(sorted(psser.mode().to_numpy().tolist()))
> {code}
> Under UTF8_LCASE, b and B are a -> 2 and b/B -> 3; the mode should be the
> single b/B group.
> *Actual* (current): ['B', 'a'] (2 modes) -- wrong; a=2 ties B=2 and b is
> counted separately.
> {*}Expected{*}: a single collation-equal mode (e.g. ['b']) with count 3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
