GitHub user dilipbiswal opened a pull request:
https://github.com/apache/spark/pull/22408
[SPARK-25417][SQL] ArrayContains function may return incorrect result when
right expression is implicitly down casted
## What changes were proposed in this pull request?
In ArrayContains, we currently cast the right hand side expression to match
the element type of the left hand side Array. This may result in down casting
and may return wrong result or questionable result.
Example :
```SQL
spark-sql> select array_contains(array(1), 1.34);
true
```
```SQL
spark-sql> select array_contains(array(1), 'foo');
null
```
We should safely coerce both left and right hand side expressions.
## How was this patch tested?
Added tests in DataFrameFunctionsSuite
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dilipbiswal/spark SPARK-25417
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22408.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22408
----
commit 2c6e8dc16c3f7418374dcabea92f5dcf966c5469
Author: Dilip Biswal <dbiswal@...>
Date: 2018-09-12T19:06:14Z
[SPARK-25417] ArrayContains function may return incorrect result when right
expression is implicitly down casted
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
