[
https://issues.apache.org/jira/browse/SPARK-61525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145184#comment-18145184
]
Ramesh Thamizhselvan edited comment on SPARK-61525 at 11/15/26, 2:00 PM:
-------------------------------------------------------------------------
here is the PR
[https://github.com/apache/spark/pull/60527]
was (Author: ramesh_004):
ere is the PR
https://github.com/apache/spark/pull/60527
> histogram_numeric produces duplicate NaN bins and loses a finite input value
> ----------------------------------------------------------------------------
>
> Key: SPARK-61525
> URL: https://issues.apache.org/jira/browse/SPARK-61525
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Environment: {code}
> Spark: 4.3.0
> Commit: 98bc0212c94705434471627068f41e374ccab143
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Master: local[2]
> {code}
> Reporter: Yibo Dong
> Priority: Major
> Labels: pull-request-available
>
> For the input values \{{NaN}}, \{{NaN}}, and \{{1.0}},
> \{{histogram_numeric(v, 2)}} returns two NaN-centered bins, with heights 2
> and 1. The finite value \{{1.0}} is not represented by either bin.
> The input contains only two distinct values under Spark SQL equality
> semantics, and two bins are requested. However, both bins are centered at
> NaN, with a combined height of 3, although only two input rows contain NaN.
> h2. Steps to Reproduce
> Save the following SQL as {{tmp.sql}}:
> {code}
> WITH input AS (
> SELECT /*+ REPARTITION(1) */ v
> FROM VALUES
> (CAST('NaN' AS DOUBLE)),
> (CAST('NaN' AS DOUBLE)),
> (1D) AS t(v)
> )
> SELECT histogram_numeric(v, 2) AS h FROM input;
> {code}
> Execute using stock Spark 4.3.0:
> {code}
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> -f tmp.sql
> {code}
> h2. Expected Result
> The two bins should represent the two distinct input values, with heights
> matching their frequencies:
> {code}
> [{"x":1.0,"y":1.0},{"x":"NaN","y":2.0}]
> {code}
> Spark SQL treats NaN values as equal and groups them together in aggregations
> ([NaN
> semantics|https://spark.apache.org/docs/4.3.0/sql-ref-datatypes.html#nan-semantics]).
> Although {{histogram_numeric}} is approximate, the input contains only two
> distinct values and requests two bins. It should not produce duplicate
> NaN-centered bins while losing the finite value.
> h2. Actual Result
> {code}
> [{"x":"NaN","y":2.0},{"x":"NaN","y":1.0}]
> {code}
> Both bins are centered at NaN, and the finite value {{1.0}} is missing from
> the bin centers. The same result occurs when the input is repartitioned into
> two partitions.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]