GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/21094
[SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might
generate a wrong result by codegen.
## What changes were proposed in this pull request?
`EqualNullSafe` for `FloatType` and `DoubleType` might generate a wrong
result by codegen.
```scala
scala> val df = Seq((Some(-1.0d), None), (None, Some(-1.0d))).toDF()
df: org.apache.spark.sql.DataFrame = [_1: double, _2: double]
scala> df.show()
+----+----+
| _1| _2|
+----+----+
|-1.0|null|
|null|-1.0|
+----+----+
scala> df.filter("_1 <=> _2").show()
+----+----+
| _1| _2|
+----+----+
|-1.0|null|
|null|-1.0|
+----+----+
```
The result should be empty but the result remains two rows.
## How was this patch tested?
Added a test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark
issues/SPARK-24007/equalnullsafe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21094.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21094
----
commit ad74513002f610a9c82a18eeff8b4218a1c81116
Author: Takuya UESHIN <ueshin@...>
Date: 2018-04-18T05:12:11Z
Fix EqualNullSafe for FloatType and DoubleType.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
