GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/10735
[SPARK-12796] [SQL] Whole stage codegen
This is the initial work for whole stage codegen, it support
Projection/Filter/Range, we will continue work on this to support more physical
operators.
A micro benchmark show that a query with range, filter and projection could
be 3X faster then before.
It's turned on by default, automatically fallback to old behavior if the
compile stage failed to compile (could because the generated Java source code
is too large, over 64k Loc).
Here is the call graph of three SparkPlan, A and B support codegen, but C
does not.
```
*
* SparkPlan A SparkPlan B SparkPlan C
* ===============================================================
* -> doExecute()
* |
* doCodeGen()
* |
* produce()
* |
* doProduce() --------> produce()
* |
* doProduce() -------> produce()
* |
* doProduce() (fetch row
from upstream)
* |
* consume()
* doConsume() ------------|
* |
* doConsume() <----- consume()
* |
* consume() (omit the rows)
*
* SparkPlan A and B should override doProduce() and doConsume().
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark whole2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10735.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10735
----
commit 2da493f06d8b9b9c0e5cf641a84fb35d5a3a7ad8
Author: Davies Liu <[email protected]>
Date: 2016-01-11T23:01:05Z
whole stage codegen
commit 998b6a150bdc2e76cea973eddedaf6ab98161637
Author: Davies Liu <[email protected]>
Date: 2016-01-12T21:30:01Z
support range
commit 218412fbacf37a9c9621c277c082f45a25a9fc5e
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:11:20Z
add benchmark
commit 158cb36bd74ba11ec7dd87adaa5f1c40e8931da8
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:15:47Z
remove println and test
commit 88c51a6aa09cb4b7abc0ec6d7609d9c823a07460
Author: Davies Liu <[email protected]>
Date: 2016-01-12T22:37:20Z
fix bug
commit b76c00e2b75a869e7ea3d6a7fda28b4bdfa85be1
Author: Davies Liu <[email protected]>
Date: 2016-01-12T23:38:38Z
fix tests
commit 4e2ee8889c18de75f29243ced2b8b52b1f0a5b1b
Author: Davies Liu <[email protected]>
Date: 2016-01-13T00:44:59Z
clean the interface
commit f05524c2f9aba8c422f571ba6e6a7d8b02a661d8
Author: Davies Liu <[email protected]>
Date: 2016-01-13T01:07:08Z
update test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
