[
https://issues.apache.org/jira/browse/SPARK-60093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaoxuan Wu updated SPARK-60093:
-------------------------------
Labels: correctness (was: )
> RemoveRedundantAggregates drops a DISTINCT that normalized -0.0 to 0.0
> ----------------------------------------------------------------------
>
> Key: SPARK-60093
> URL: https://issues.apache.org/jira/browse/SPARK-60093
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1, 5.0.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness
>
> A DISTINCT (an Aggregate) normalizes floating-point grouping keys: -0.0 and
> 0.0 become one row with value 0.0. When an outer aggregate only uses
> duplicate-insensitive functions, {{RemoveRedundantAggregates}} removes the
> inner DISTINCT as if it only removed duplicates. The outer query then sees
> the original -0.0 again.
> {code:java}
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.getOrCreate()print(spark.sql("SELECT DISTINCT x
> FROM VALUES (-0.0D), (0.0D) t(x)").collect())
> # [Row(x=0.0)]: DISTINCT keeps one row, normalized to 0.0q = """
> SELECT cast(x AS STRING) AS k, max(x) AS m
> FROM (SELECT DISTINCT x FROM VALUES (-0.0D), (0.0D) t(x))
> GROUP BY 1
> """
> print(sorted(spark.sql(q).collect()))
> # expected: [Row(k='0.0', m=0.0)]
> # Spark 4.2.0: [Row(k='-0.0', m=-0.0), Row(k='0.0',
> m=0.0)]spark.conf.set("spark.sql.optimizer.excludedRules",
>
> "org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAggregates")
> print(sorted(spark.sql(q).collect()))
> # [Row(k='0.0', m=0.0)]: correct once the rule is excluded {code}
> Same result on 3.5.8, 4.0.0, 4.1.1, 4.2.0 and master. NaN values with
> different bit patterns are affected the same way.
> *Cause:* the rule treats the lower Aggregate as a pure de-duplication, but
> {{NormalizeFloatingNumbers}} makes it also change values (-0.0 to 0.0, NaN
> canonicalization). The rule should not fire when the lower aggregate groups
> by float/double keys (or by structs/arrays containing them) whose values the
> upper aggregate reads.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
