GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/295
[SQL] SPARK-1371 Hash Aggregation Improvements
Given:
```scala
case class Data(a: Int, b: Int)
val rdd =
sparkContext
.parallelize(1 to 200)
.flatMap(_ => (1 to 50000).map(i => Data(i % 100, i)))
rdd.registerAsTable("data")
cacheTable("data")
```
Before:
```
SELECT COUNT(*) FROM data:[10000000]
16795.567ms
SELECT a, SUM(b) FROM data GROUP BY a
7536.436ms
SELECT SUM(b) FROM data
10954.1ms
```
After:
```
SELECT COUNT(*) FROM data:[10000000]
1372.175ms
SELECT a, SUM(b) FROM data GROUP BY a
2070.446ms
SELECT SUM(b) FROM data
958.969ms
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark hashAgg
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/295.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #295
----
commit 4fb01ad99000159d30ad0783f83f8ff34262dc1f
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T03:09:51Z
Increase default number of partitions.
commit f3b4bd662c6ed2c8f17ac360dfbccd3dadb590f4
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T03:10:09Z
Always include serialization debug info when running tests.
commit f1971c3fd4ad7a955cc570c8271e8b046d50e24f
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T03:10:28Z
Add option to avoid binding operators expressions automatically.
commit b1b05d97fba231e26807e8cb34c4b7f419bbf5d0
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T03:10:54Z
Make HiveUDAF fields transient since object inspectors are not serializable.
commit 9680f236492cff78e3f537287b7c16cdb344520a
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T03:12:41Z
Rewrite Aggregate operator to stream input and use projections. Remove
unused local RDD functions implicits.
commit e872850d80f9df3dcf508fa4290f85a71ad5bd8f
Author: Michael Armbrust <mich...@databricks.com>
Date: 2014-04-02T04:55:58Z
Capture child output statically to avoid issues with generators and
serialization.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
