Yibo Dong created SPARK-61527:
---------------------------------
Summary: findings/spark/partition_recomposition
Key: SPARK-61527
URL: https://issues.apache.org/jira/browse/SPARK-61527
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0
Environment: {code}
Spark: 4.3.0
Commit: 98bc0212c94705434471627068f41e374ccab143
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Master: local[2]
{code}
Reporter: Yibo Dong
{{approx_top_k}} treats positive zero ({{0.0}}) and negative zero ({{-0.0}}) as
separate items, although Spark SQL considers them equal.
For two input rows containing {{0.0}} and {{-0.0}}, {{approx_top_k(v, 2, 10)}}
returns two items with count 1 each, while {{GROUP BY}} returns one group with
count 2.
h2. Steps to Reproduce
Save the following SQL as {{tmp.sql}}:
{code}
WITH input AS (
SELECT /*+ REPARTITION(3) */ v
FROM VALUES (0D), (-0D) AS t(v)
)
SELECT approx_top_k(v, 2, 10) AS frequent_items
FROM input;
SELECT v, count(*) AS frequency
FROM VALUES (0D), (-0D) AS t(v)
GROUP BY v;
{code}
Execute:
{code}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false -f tmp.sql
{code}
h2. Expected Result
The first query should return a single zero-valued item with count 2:
{code}
[{"item":0.0,"count":2}]
{code}
The sign of the zero-valued item is not important.
Spark SQL treats positive and negative zero as equal in grouping, as documented
in the [SQL migration
guide|https://spark.apache.org/docs/4.3.0/sql-migration-guide.html].
With only two input rows and a tracking capacity of ten items, the discrepancy
cannot be attributed to insufficient sketch capacity.
h2. Actual Result
{code}
[{"item":0.0,"count":1},{"item":-0.0,"count":1}]
{code}
The two zero representations are counted separately. The independent {{GROUP
BY}} query correctly returns one zero-valued group with frequency 2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]