[ 
https://issues.apache.org/jira/browse/SPARK-61525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145184#comment-18145184
 ] 

Ramesh Thamizhselvan commented on SPARK-61525:
----------------------------------------------

ere is the PR
https://github.com/apache/spark/pull/60527

> histogram_numeric produces duplicate NaN bins and loses a finite input value
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-61525
>                 URL: https://issues.apache.org/jira/browse/SPARK-61525
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.3.0
>         Environment: {code}
> Spark: 4.3.0 
> Commit: 98bc0212c94705434471627068f41e374ccab143
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Master: local[2]
> {code}
>            Reporter: Yibo Dong
>            Priority: Major
>              Labels: pull-request-available
>
> For the input values \{{NaN}}, \{{NaN}}, and \{{1.0}}, 
> \{{histogram_numeric(v, 2)}} returns two NaN-centered bins, with heights 2 
> and 1. The finite value \{{1.0}} is not represented by either bin.
> The input contains only two distinct values under Spark SQL equality 
> semantics, and two bins are requested. However, both bins are centered at 
> NaN, with a combined height of 3, although only two input rows contain NaN.
> h2. Steps to Reproduce
> Save the following SQL as {{tmp.sql}}:
> {code}
> WITH input AS (
> SELECT /*+ REPARTITION(1) */ v
> FROM VALUES
> (CAST('NaN' AS DOUBLE)),
> (CAST('NaN' AS DOUBLE)),
> (1D) AS t(v)
> )
> SELECT histogram_numeric(v, 2) AS h FROM input;
> {code}
> Execute using stock Spark 4.3.0:
> {code}
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> -f tmp.sql
> {code}
> h2. Expected Result
> The two bins should represent the two distinct input values, with heights 
> matching their frequencies:
> {code}
> [{"x":1.0,"y":1.0},{"x":"NaN","y":2.0}]
> {code}
> Spark SQL treats NaN values as equal and groups them together in aggregations 
> ([NaN 
> semantics|https://spark.apache.org/docs/4.3.0/sql-ref-datatypes.html#nan-semantics]).
> Although {{histogram_numeric}} is approximate, the input contains only two 
> distinct values and requests two bins. It should not produce duplicate 
> NaN-centered bins while losing the finite value.
> h2. Actual Result
> {code}
> [{"x":"NaN","y":2.0},{"x":"NaN","y":1.0}]
> {code}
> Both bins are centered at NaN, and the finite value {{1.0}} is missing from 
> the bin centers. The same result occurs when the input is repartitioned into 
> two partitions.
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to