[
https://issues.apache.org/jira/browse/CALCITE-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17395112#comment-17395112
]
Botong Huang commented on CALCITE-4568:
---------------------------------------
(copied from dev email)
Hi all,
Please find our rebased Tempura code on top of a fairly recent version of
Calcite at:
https://github.com/hbtoo/calcite/tree/botong
There are six new commits in total:
739119282 Hack around the type issue to make tvr unit tests work, see
CALCITE-4713
c84da7216 4. Full tempura integration: all TVR rules and complete optimization
procedure. Newly added files only.
d3217f432 3. Full tempura integration: all TVR rules and complete optimization
procedure. Modifications to existing files only.
59342a654 2. Tempura core memo structure, rule engine, and interfaces. Newly
added files only.
0d310841d 1. Tempura core memo structure, rule engine, and interfaces.
Modifications to existing files only.
c1240ca7b 0. Add volcano visualizer for debugging.
The first three (0, 1, 2) is a compilable version with extended core system
support:
1. Memo extension with TvrMetaSet
2. Rule engine upgrade, capable of matching TvrMetaSet and RelNodes, as well as
links in between the nodes.
The next two (3 and 4) is a full version with:
All changes in this feature will consist of four parts:
3. A provided set of TvrRules, written using the upgraded rule engine API.
4. TvrVolcanoPlanner that puts everything together end to end.
5. Multi-query optimization, used to find the best incremental plan involving
multiple time points.
With up to 4, all existing CALCITE unit tests pass.
To demonstrate how Tempura works, we have added the following two example unit
tests that can be run directly (with the last commit to hack around
CALCITE-4713):
TvrOptimizationTest.java runs the Tempura optimizer. This program produces a
progressive physical plan by the Tempura optimizer that runs across several
time points. The physical plan is printed out to the console in DOT format,
which can be viewed using an online graphviz tool.
TvrExecutionTest.java uses the Tempura optimizer in an end-to-end query. This
program generates a progressive physical plan and then uses Calcite's built-in
executor to run the plan. The output at each time point is printed to the
console.
Everyone is welcome and encouraged to take a look and play with it. Let's take
some time and figure out a plan on how to incorporate Tempura into Calcite that
best suits everyone.
> Tempura: extending Calcite into an incremental query optimizer
> --------------------------------------------------------------
>
> Key: CALCITE-4568
> URL: https://issues.apache.org/jira/browse/CALCITE-4568
> Project: Calcite
> Issue Type: New Feature
> Reporter: Botong Huang
> Priority: Major
>
> As discussed in the email thread, this is an attempt to extend the Calcite
> optimizer into a general incremental query optimizer, based on our research
> paper published in VLDB 2021:
> Tempura: a general cost-based optimizer framework for incremental data
> processing
> To our best knowledge, this is the first general cost-based incremental
> optimizer that can find the best plan across multiple families of incremental
> computing methods, including IVM, Streaming, DBToaster, etc. Experiments (in
> the paper) shows that the generated best plan is consistently much better
> than the plans from each individual method alone.
> In general, incremental query planning is central to database view
> maintenance and stream processing systems, and are being adopted in active
> databases, resumable query execution, approximate query processing, etc. We
> are hoping that this feature can help widening the spectrum of Calcite,
> solicit more use cases and adoption of Calcite.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
