GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/11298
[SPARK-13428] [SQL] Pushing Down Aggregate Expressions in Sort into
Aggregate
## What changes were proposed in this pull request?
When there exists the other operators between Sort and Aggregate, we are
unable to push down the aggregate expressions in `Sort` into `Aggregate`. For
example, in the following query,
```SQL
select area, sum(product) over () as c from windowData
where product > 3 group by area, product
having avg(month) > 0 order by area, avg(month), product
```
The parsed logical plan is like
```
'Sort ['area ASC,'avg('month) ASC,'product ASC], true
+- 'Filter cast(('avg('month) > 0) as boolean)
+- 'Aggregate ['area,'product],
[unresolvedalias('area,None),unresolvedalias('sum('product)
windowspecdefinition(UnspecifiedFrame) AS c#3,None)]
+- 'Filter ('product > 3)
+- 'UnresolvedRelation `windowData`, None
```
## How was the this patch tested?
Turn on a test case that `test("window function: Pushing aggregate
Expressions in Sort to Aggregate")` exposes this issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark aggExprInSortBy
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11298.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11298
----
commit 50071e2cf9c76fd1919c437500b82bfd287b5f95
Author: gatorsmile <[email protected]>
Date: 2016-02-22T03:57:45Z
pushing down aggregate expressions in Sort into Aggregate Operator
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
