GitHub user hvanhovell opened a pull request:
https://github.com/apache/spark/pull/6278
[SPARK-7712] [SQL] Move Window Functions from Hive UDAFS to Spark Native
backend [WIP]
This PR aims to improve the current window functionality in Spark SQL in
the following ways:
- Moving away from Hive backed UDAF's to a more Spark SQL Native
implementation. The main advantages are that some overhead can be avoided, that
it it easier to extend, and that it opens up the opportunity to do more
aggressive code generation (AggregateEvaluation style).
- Improve the processing for the 'between unbounded and current' amd the
'between current and unbounded' cases. This was very expensive, now it performs
in linear time.
- Process different frames for the same group by/order by definition in the
same window clause. This should save some memory and reduce some of the
overhead.
This is currently a work in progress. Quite a bit of testing needs to be
done. Any feedback will be greatly appreciated. The JIRA ticket can be found
here: https://issues.apache.org/jira/browse/SPARK-7712
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hvanhovell/spark SPARK-7712
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6278.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6278
----
commit e4b81d5276076ce1f8130b56c557ff9a4e19dce8
Author: Herman van Hovell <[email protected]>
Date: 2015-05-20T07:30:38Z
Moved Code from separate project into Spark code base. Added HiveQL parsing.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
