Yaoxuan Wu created SPARK-60093:
----------------------------------
Summary: RemoveRedundantAggregates drops a DISTINCT that
normalized -0.0 to 0.0
Key: SPARK-60093
URL: https://issues.apache.org/jira/browse/SPARK-60093
Project: Spark
Issue Type: Bug
Components: Optimizer
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0, 5.0.0
Reporter: Yaoxuan Wu
A DISTINCT (an Aggregate) normalizes floating-point grouping keys: -0.0 and 0.0
become one row with value 0.0. When an outer aggregate only uses
duplicate-insensitive functions, {{RemoveRedundantAggregates}} removes the
inner DISTINCT as if it only removed duplicates. The outer query then sees the
original -0.0 again.
{code:java}
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()print(spark.sql("SELECT DISTINCT x
FROM VALUES (-0.0D), (0.0D) t(x)").collect())
# [Row(x=0.0)]: DISTINCT keeps one row, normalized to 0.0q = """
SELECT cast(x AS STRING) AS k, max(x) AS m
FROM (SELECT DISTINCT x FROM VALUES (-0.0D), (0.0D) t(x))
GROUP BY 1
"""
print(sorted(spark.sql(q).collect()))
# expected: [Row(k='0.0', m=0.0)]
# Spark 4.2.0: [Row(k='-0.0', m=-0.0), Row(k='0.0',
m=0.0)]spark.conf.set("spark.sql.optimizer.excludedRules",
"org.apache.spark.sql.catalyst.optimizer.RemoveRedundantAggregates")
print(sorted(spark.sql(q).collect()))
# [Row(k='0.0', m=0.0)]: correct once the rule is excluded {code}
Same result on 3.5.8, 4.0.0, 4.1.1, 4.2.0 and master. NaN values with different
bit patterns are affected the same way.
*Cause:* the rule treats the lower Aggregate as a pure de-duplication, but
{{NormalizeFloatingNumbers}} makes it also change values (-0.0 to 0.0, NaN
canonicalization). The rule should not fire when the lower aggregate groups by
float/double keys (or by structs/arrays containing them) whose values the upper
aggregate reads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
