GitHub user HeartSaVioR opened a pull request:
https://github.com/apache/storm/pull/1736
STORM-1446 Compile the Calcite logical plan to Storm Trident logical plan
First of all I should thank to @milinda for sharing great implementation,
because this patch is merely just a port of SamzaSQL (though it was from
@milinda repo, not Samza repo).
This is for converting Calcite logical plan to Storm's Trident logical plan
(as my understanding of logical plan / physical plan is right). Trident itself
does a physical plan.
After introducing this, much more things can be followed up, like
- setting up parallelism hint based on data source's hint (metadata)
- push down filter / project to data source
- go back generating source code of Trident topology, but avoiding raw
String concatenation
- and whatever...
This is rather huge patch and it requires understanding of core concept of
Calcite. If you haven't learn it, this slide can help you, as I did.
http://www.slideshare.net/JordanHalterman/introduction-to-apache-calcite
It doesn't touch standalone mode (but I will try later), and also doesn't
touch feature itself so it doesn't require documentation, but adding javadoc
comment might be needed.
If reviewers want to see it please let me know so that I can work on.
Thanks in advance!
Commit log is below:
* Port SamzaSQL implementation to Storm
* https://github.com/milinda/samza-sql
* Apply some rules to optimize
* optimize Calc
* merge filter and projection scripts into one
* also applying short circuit
* Modify Trident unit tests to use new query planner
* arrange some files
* Move some files which are only used from standalone
* Remove some files which are no longer used
* guard the possibility of stack overflow error on explaining
* just leave error logs, and print out empty plan and continue
* reported this behavior to Calcite community
* leave some comments to clarify what it means
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HeartSaVioR/storm STORM-1446
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1736
----
commit 692af990f051e1c37390c798d2b1eddcef60d01b
Author: Jungtaek Lim <kabh...@gmail.com>
Date: 2016-10-13T10:00:10Z
STORM-1446 Compile the Calcite logical plan to Storm Trident logical plan
* Port SamzaSQL implementation to Storm
* https://github.com/milinda/samza-sql
* Apply some rules to optimize
* optimize Calc
* merge filter and projection scripts into one
* also applying short circuit
* Modify Trident unit tests to use new query planner
* arrange some files
* Move some files which are only used from standalone
* Remove some files which are no longer used
* guard the possibility of stack overflow error on explaining
* just leave error logs, and print out empty plan and continue
* reported this behavior to Calcite community
* leave some comments to clarify what it means
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
