Yibo Dong created SPARK-61528:
---------------------------------

             Summary: theta_sketch_agg conflates arrays containing NULL and zero
                 Key: SPARK-61528
                 URL: https://issues.apache.org/jira/browse/SPARK-61528
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0
         Environment: {code}
Spark: 4.3.0 
Commit: 98bc0212c94705434471627068f41e374ccab143
Java: OpenJDK 17.0.17
OS: macOS 15.7.4
Master: local[2]
{code}


            Reporter: Yibo Dong


{\{theta_sketch_agg}} incorrectly treats the arrays \{{[NULL]}} and \{{[0]}} as 
the same value.

For these two non-NULL arrays, the Theta sketch estimates one distinct value, 
while \{{COUNT(DISTINCT)}} correctly returns two.

 

h2. Steps to Reproduce

Save the following SQL as \{{tmp.sql}}:

{code}
WITH input AS (
SELECT /*+ REPARTITION(3) */
CASE id
WHEN 0 THEN array(CAST(NULL AS INT))
ELSE array(0)
END AS a
FROM range(0, 2)
)
SELECT
theta_sketch_estimate(theta_sketch_agg(a, 12)) AS theta_estimate,
count(DISTINCT a) AS exact_count
FROM input;
{code}

Execute:

{code}
spark-sql --master 'local[2]' 
--conf spark.ui.enabled=false -f tmp.sql
{code}

h2. Expected Result

{code}
theta_estimate | exact_count
---------------+------------
2 | 2
{code}

The two arrays are distinct non-NULL values. A NULL element within an array 
should not be treated as an integer zero when estimating distinct array values.

h2. Actual Result

{code}
theta_estimate | exact_count
---------------+------------
1 | 2
{code}

The Theta sketch reports one distinct value, while the exact aggregate 
correctly identifies two distinct arrays.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to