GitHub user EntilZha opened a pull request:
https://github.com/apache/spark/pull/7580
[SPARK-8231][SQL] Add array_contains
PR for work on https://issues.apache.org/jira/browse/SPARK-8231
Currently, I have an initial implementation for contains. Based on
discussion on JIRA, it should behave same as Hive:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArrayContains.java#L102-L128
Main points are:
1. If the array is empty, null, or the value is null, return false
2. If there is a type mismatch, throw error
3. If comparison is not supported, throw error
Thus far, I have 1 implemented, however the codegen is failing. A nudge in
the right direction would be helpful. I am also uncertain if I should be
directly outputting the code or using a helper function. I am directly
outputting the code because my understanding is if the return value should not
be null if any of the arguments are null, then it needs to be custom.
I have an idea of how to do 2 based on other code I have seen, and google
probably knows how to find out 3.
Additionally, still need to:
1. Add docs
2. Add python API
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/EntilZha/spark SPARK-8231
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7580.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7580
----
commit c28508a196c50cce0aab6078e719d8e8fa5433e5
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-21T22:09:18Z
initial attempt at implementation
commit a4ef4e21cfc7f1567f9c03848d6babe650e41b38
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-21T23:27:28Z
added tests and codegen
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
