GitHub user EntilZha opened a pull request:
https://github.com/apache/spark/pull/7462
[SPARK-8230][SQL] Add array/map size method
Pull Request for: https://issues.apache.org/jira/browse/SPARK-8230
Primary issue resolved is to implement array/map size for Spark SQL. Code
is ready for review by a committer. Chen Hao is on the JIRA ticket, but I don't
know his username on github, @rxin is also on JIRA ticket.
Things to review:
1. Where to put added functions namespace wise, they seem to be part of a
few operations on collections which includes `sort_array` and `array_contains`.
Hence the name given `collectionOperations.scala` and `_collection_functions`
in python.
2. In Python code, should it be in a `1.5.0` function array or in a
collections array?
3. Are there any missing methods on the `Size` case class? Looks like many
of these functions have generated Java code, is that also needed in this case?
4. Something else?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/EntilZha/spark SPARK-8230
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7462.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7462
----
commit 22ea0144affde24565fad5fe8e2d82b86458046e
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-17T01:16:58Z
First attempt at implementing size for maps and arrays
commit 5dee77c6534e26931f834ffca2749ffe9dd3dfa9
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-17T07:07:01Z
Finished working version of size function and added it to python
commit deaeb649a469e10b4b8b615de661481032583b8d
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-17T07:25:03Z
added unit test for size
commit 55903d01d1337b449e396c789278332c76dbfa19
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-17T07:28:29Z
added documentation
commit 6f4aa7d9768dd606022ce90b33f0477422611fbd
Author: Pedro Rodriguez <[email protected]>
Date: 2015-07-17T07:32:53Z
added unit tests for map size
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
