GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/10896
[SPARK-12978][SQL] Skip unnecessary final group-by when input data already
clustered with group-by keys
This ticket targets the optimization to skip an unnecessary group-by
operation below;
Without opt.:
```
== Physical Plan ==
TungstenAggregate(key=[col0#159],
functions=[(sum(col1#160),mode=Final,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)],
output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenAggregate(key=[col0#159],
functions=[(sum(col1#160),mode=Partial,isDistinct=false),(avg(col2#161),mode=Partial,isDistinct=false)],
output=[col0#159,sum#200,sum#201,count#202L])
+- TungstenExchange hashpartitioning(col0#159,200), None
+- InMemoryColumnarTableScan [col0#159,col1#160,col2#161],
InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true,
true, false, true, 1), ConvertToUnsafe, None
```
With opt.:
```
== Physical Plan ==
TungstenAggregate(key=[col0#159],
functions=[(sum(col1#160),mode=Complete,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)],
output=[col0#159,sum(col1)#177,avg(col2)#178])
+- TungstenExchange hashpartitioning(col0#159,200), None
+- InMemoryColumnarTableScan [col0#159,col1#160,col2#161],
InMemoryRelation [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true,
true, false, true, 1), ConvertToUnsafe, None
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SkipGroupbySpike
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10896.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10896
----
commit 5ab19c1a97593b75512ba7b910849bef87f9810e
Author: Takeshi YAMAMURO <[email protected]>
Date: 2016-01-25T05:58:45Z
Skip unnecessary final group-by when input data already clustered
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
