Yibo Dong created SPARK-61527:
---------------------------------

             Summary: findings/spark/partition_recomposition
                 Key: SPARK-61527
                 URL: https://issues.apache.org/jira/browse/SPARK-61527
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0
         Environment: {code}
Spark: 4.3.0 
Commit: 98bc0212c94705434471627068f41e374ccab143
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Master: local[2]
{code}


            Reporter: Yibo Dong


{{approx_top_k}} treats positive zero ({{0.0}}) and negative zero ({{-0.0}}) as 
separate items, although Spark SQL considers them equal.

For two input rows containing {{0.0}} and {{-0.0}}, {{approx_top_k(v, 2, 10)}} 
returns two items with count 1 each, while {{GROUP BY}} returns one group with 
count 2.

h2. Steps to Reproduce

Save the following SQL as {{tmp.sql}}:

{code}
WITH input AS (
SELECT /*+ REPARTITION(3) */ v
FROM VALUES (0D), (-0D) AS t(v)
)
SELECT approx_top_k(v, 2, 10) AS frequent_items
FROM input;

SELECT v, count(*) AS frequency
FROM VALUES (0D), (-0D) AS t(v)
GROUP BY v;
{code}

Execute:

{code}
spark-sql --master 'local[2]' 
--conf spark.ui.enabled=false -f tmp.sql
{code}

h2. Expected Result

The first query should return a single zero-valued item with count 2:

{code}
[{"item":0.0,"count":2}]
{code}

The sign of the zero-valued item is not important.

Spark SQL treats positive and negative zero as equal in grouping, as documented 
in the [SQL migration 
guide|https://spark.apache.org/docs/4.3.0/sql-migration-guide.html].

With only two input rows and a tracking capacity of ten items, the discrepancy 
cannot be attributed to insufficient sketch capacity.

h2. Actual Result

{code}
[{"item":0.0,"count":1},{"item":-0.0,"count":1}]
{code}

The two zero representations are counted separately. The independent {{GROUP 
BY}} query correctly returns one zero-valued group with frequency 2.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to