GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/11004
[SPARK-10605][SQL] Add structs to collect_list/collect_set
We currently use the Hive implementations for the
```collect_list```/```collect_set``` aggregate functions. This has a few major
drawbacks, the use of HiveUDAF (which has quite a bit of overhead) and the lack
of support for struct datatypes. This PR adds native implementation of these
functions to Spark.
The size of the collected list/set vary, this means we cannot use the fast,
Tungsten, aggregation path to perform the aggregation, and that we fallback to
the slower sort based path. Another big problem with these operators is that
when the size of the collected list/set grows too large, we can start
experiencing large GC pauzes and OOMEs.
This PR implements these UDAFs as ```ImperativeAggregate```s and uses a
custom mutable ```ArrayData``` structure in order to efficiently collect the
items. The current PR allows for partial aggregates, but I am not sure yet if
we should support this.
Comments and feedback are appreciated.
cc @yhuai
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark implode
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11004.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11004
----
commit 9d8aeed7b18d58258097526f85f07c37a10c28d8
Author: Herman van Hovell <[email protected]>
Date: 2016-02-01T15:15:16Z
Add native collect_set/collect_list.
commit 8247d8eb6416b7b5d4eb61fa585c595d87930d63
Author: Herman van Hovell <[email protected]>
Date: 2016-02-01T15:43:24Z
Add test for struct types.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
