GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/10786
[SPARK-10735] [SQL] Generate aggregation w/o grouping keys [WIP]
PR is based on #10735
The benchmark show that generated aggregation could be 6X time faster than
non-generated (with generated filter/range).
It also showed that declarative function is much faster than imperative one
(5X faster for stddev), we may need to switch to a declarative version. (the
difference need to be measured when there are grouping keys).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark gen_agg2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10786.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10786
----
commit 2da493f06d8b9b9c0e5cf641a84fb35d5a3a7ad8
Author: Davies Liu <[email protected]>
Date: 2016-01-11T23:01:05Z
whole stage codegen
commit 998b6a150bdc2e76cea973eddedaf6ab98161637
Author: Davies Liu <[email protected]>
Date: 2016-01-12T21:30:01Z
support range
commit 218412fbacf37a9c9621c277c082f45a25a9fc5e
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:11:20Z
add benchmark
commit 158cb36bd74ba11ec7dd87adaa5f1c40e8931da8
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:15:47Z
remove println and test
commit 88c51a6aa09cb4b7abc0ec6d7609d9c823a07460
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:37:20Z
fix bug
commit b76c00e2b75a869e7ea3d6a7fda28b4bdfa85be1
Author: Davies Liu <[email protected]>
Date: 2016-01-12T23:38:38Z
fix tests
commit 4e2ee8889c18de75f29243ced2b8b52b1f0a5b1b
Author: Davies Liu <[email protected]>
Date: 2016-01-13T00:44:59Z
clean the interface
commit f05524c2f9aba8c422f571ba6e6a7d8b02a661d8
Author: Davies Liu <[email protected]>
Date: 2016-01-13T01:07:08Z
update test
commit 43139a84e637598953f0c757ba2ea62f4ad4bfe8
Author: Davies Liu <[email protected]>
Date: 2016-01-13T23:13:30Z
fix style, improve comments
commit 73fe07499faed9fdb33192337b0996a299d8e6bf
Author: Davies Liu <[email protected]>
Date: 2016-01-13T23:32:48Z
fix style
commit 32309e14e39af9f418faa3d837cebe64c3c87b84
Author: Davies Liu <[email protected]>
Date: 2016-01-14T06:45:09Z
fix test
commit 38029bc5eff03cd4a7334bbe7c16b543f7247030
Author: Davies Liu <[email protected]>
Date: 2016-01-15T07:29:00Z
WholeStageCodegen/FakeInput as adapter
commit 1f9ddb8538ed5c2017e6310815eef32f065ca0cb
Author: Davies Liu <[email protected]>
Date: 2016-01-15T07:54:26Z
address comments
commit 2a3f0e32a81898da1399ad1d0b98ec913dbc5668
Author: Davies Liu <[email protected]>
Date: 2016-01-15T08:00:08Z
Merge branch 'master' of github.com:apache/spark into whole2
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
sql/core/src/main/scala/org/apache/spark/sql/execution/basicOperators.scala
commit c1dd60a21e1a016b5479cf89cdef035ac53167f0
Author: Davies Liu <[email protected]>
Date: 2016-01-15T08:05:27Z
fix style
commit 16ce50c4458b7ab4bd4c28e88d6f7f600bbeebf7
Author: Davies Liu <[email protected]>
Date: 2016-01-15T16:28:44Z
Merge branch 'master' of github.com:apache/spark into whole2
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
commit d4530812c29ea15c92a07cc9aae3db05a9d8e06a
Author: Davies Liu <[email protected]>
Date: 2016-01-15T16:39:20Z
fix bug
commit 34a0a6f15c48935bc79afd7b08e5553552690adc
Author: Davies Liu <[email protected]>
Date: 2016-01-15T17:39:51Z
fix test
commit 908c8cb22596e0ae662d388014624eb289c753d5
Author: Davies Liu <[email protected]>
Date: 2016-01-15T18:11:15Z
address comments
commit 1feab20decb487ed66e712323565bb4d58f7637f
Author: Davies Liu <[email protected]>
Date: 2016-01-15T19:24:58Z
use sparkPlan for checking
commit c9741ea856ecccdcb2dc349ee576080b32b50e19
Author: Davies Liu <[email protected]>
Date: 2016-01-15T21:52:41Z
create a rule for whole stage codegen
commit 7c0570335da47847c3462306a930abc91ee1f932
Author: Davies Liu <[email protected]>
Date: 2016-01-15T22:02:07Z
renaming
commit 0b401067347f57d0f7b92c999bb2bcf7e3e95d3d
Author: Davies Liu <[email protected]>
Date: 2016-01-15T22:39:05Z
fix style
commit d12a8da88e37344bc429fb4f67c0258a28b209c4
Author: Davies Liu <[email protected]>
Date: 2016-01-16T03:21:00Z
Merge branch 'master' of github.com:apache/spark into whole2
Conflicts:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodegenFallback.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateMutableProjection.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratePredicate.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateSafeProjection.scala
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala
commit 3df7e5dfad96d48eb47d6f81d85457ae6d3a4156
Author: Davies Liu <[email protected]>
Date: 2016-01-16T06:19:25Z
generate aggregation
commit b43f26257885a912a45850cd5fad1936a2dbfe1b
Author: Davies Liu <[email protected]>
Date: 2016-01-16T06:37:56Z
support impremitive function
commit 640a4b57a687e16ad02c22c4fe787a540ff890a8
Author: Davies Liu <[email protected]>
Date: 2016-01-16T09:13:29Z
support imperative functions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
