Thanks Julian for the sharing! Overall sounds reasonable! >Our side of the bargain, as committers, is that we should review in a timely manner, and not move the goal posts Huge +1. I will be very happy to help during this process.
On Thu, May 13, 2021 at 12:47 PM Julian Hyde <jhyde.apa...@gmail.com> wrote: > During the meeting we agreed to start progressing this contribution in the > usual Apache Way, with conversations on the dev list and in the > https://issues.apache.org/jira/browse/CALCITE-4568 < > https://issues.apache.org/jira/browse/CALCITE-4568> JIRA case. So, it > should be easy for you to participate. > > Botong said he would share the slides. (He might be unwilling to make them > public, because they are his presentation for a conference that has not > happened yet. Reach out to him one-to-one.) > > Next step is for someone on the Alibaba side to create a PR that is > rebased on the latest Calcite master, and add a comment to the JIRA case. > Then we can discuss what needs to be done for that PR. Code quality, adding > comments, breaking up into smaller commits, additional tests, renaming > packages/classes, restructuring into plugins are all possibilities. > Our side of the bargain, as committers, is that we should review in a > timely manner, and not move the goal posts — if the contributors make the > changes we request then we will land this code in master in a reasonable > amount of time. > > We also discussed incremental view maintenance (IVM). Tempura solves a > more general problem (finding the optimal K steps to maintain a > materialized view as data arrives in K points in time) but if we set K=2, > we can generate a plan for how to update a materialized view given a delta > table. The plan will be different based on cost - e.g. whether the delta > table is small or large. This is a problem that many of our users would > like to solve. It will exercise much of Tempura’s code base, and encourage > contributions. > > In my opinion, we should do IVM at launch. It should be the main example > we use in conference talks, blog posts, etc. When people understand that > case, we can explain how we generalize from K=2 to arbitrary K. > > Julian > > > > On May 13, 2021, at 9:51 AM, Rui Wang <amaliu...@apache.org> wrote: > > > > I apologize that I had a wrong impression on the meeting time (I thought > it > > should be on Thursday but it is Wednesday). I can follow up your meeting > > records if you have any. > > > > > > -Rui > > > > On Tue, May 11, 2021 at 8:17 PM Botong Huang <pku...@gmail.com> wrote: > > > >> Hi all, > >> > >> This is a reminder that we are going to have our second discussion > meeting > >> tomorrow at 10-11pm PST. Please find the link below, everyone is > welcome to > >> join! > >> > >> Join Zoom Meeting > >> https://uci.zoom.us/j/91986206610 > >> < > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fj%2F91986206610&sa=D&source=calendar&usd=2&usg=AOvVaw24sxPtI6hbukCSo3nlQsbn > >>> > >> > >> Meeting ID: 919 8620 6610 > >> One tap mobile > >> +16699006833 <(669)%20900-6833> <(669)%20900-6833>,,91986206610# US > (San Jose) > >> +12532158782 <(253)%20215-8782> <(253)%20215-8782>,,91986206610# US > (Tacoma) > >> > >> Dial by your location > >> +1 669 900 6833 <(669)%20900-6833> <(669)%20900-6833> US (San > Jose) > >> +1 253 215 8782 <(253)%20215-8782> <(253)%20215-8782> US > (Tacoma) > >> +1 346 248 7799 <(346)%20248-7799> <(346)%20248-7799> US > (Houston) > >> +1 301 715 8592 <(301)%20715-8592> <(301)%20715-8592> US > (Washington DC) > >> +1 312 626 6799 <(312)%20626-6799> <(312)%20626-6799> US > (Chicago) > >> +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> US (New > York) > >> Meeting ID: 919 8620 6610 > >> Find your local number: https://uci.zoom.us/u/acyXcc43Cd > >> < > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fu%2FacyXcc43Cd&sa=D&source=calendar&usd=2&usg=AOvVaw2W08kj_8hEx44dryeZlXb6 > >>> > >> > >> Join by Skype for Business > >> https://uci.zoom.us/skype/91986206610 > >> < > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fskype%2F91986206610&sa=D&source=calendar&usd=2&usg=AOvVaw3w0M0YYbcjPyBXzNpyyk0Z > >>> > >> > >> Thanks, > >> Botong > >> > >> On Wed, May 5, 2021 at 9:55 AM Botong Huang <pku...@gmail.com> wrote: > >> > >>> Hi Stamatis and all, > >>> > >>> Thanks for the interest! Let's tentatively schedule the next meeting > next > >>> Wednesday at May 12, 10pm-11pm PST then. Please let us know if there's > >> new > >>> needs showing up. > >>> > >>> Best, > >>> Botong > >>> > >>> On Sun, May 2, 2021 at 2:59 PM Stamatis Zampetakis <zabe...@gmail.com> > >>> wrote: > >>> > >>>> Hello, > >>>> > >>>> I really regret missing the first meeting, sorry about that. I added > my > >>>> preferences in the document. > >>>> I will make sure to attend the next one and help as much as I can. > >>>> > >>>> I didn't have the chance yet to go over the paper but will try to do > it > >>>> before the next meeting. > >>>> > >>>> For me the following dates are more convenient than others so it would > >> be > >>>> nice if we could arrange it then. > >>>> > >>>> Thu, May 6, 10pm PST > >>>> Tue, May 12, 10pm PST > >>>> > >>>> Best, > >>>> Stamatis > >>>> > >>>> On Sat, May 1, 2021 at 9:42 PM Julian Hyde <jh...@apache.org> wrote: > >>>> > >>>>> I have added my time preferences to the doc [1]. I am generally > >>>>> available any evening Mon - Thu. How about we meet Monday 10th May? > >>>>> > >>>>> Stamatis, Jesus, Given the complexity of this work, I would very much > >>>>> appreciate your insight, as experts in optimizer theory. Could one of > >>>>> you join the next meeting? Of course we should choose a time that > >>>>> works for everyone's schedule. > >>>>> > >>>>> Julian > >>>>> > >>>>> [1] > >>>>> > >>>> > >> > https://docs.google.com/document/d/1wyNjB94uSGwHtVvGYDwaLlCghUJE-7aDLnCdKKXJN1o/edit?usp=sharing > >>>>> > >>>>> On Wed, Apr 28, 2021 at 9:32 AM Botong Huang <pku...@gmail.com> > >> wrote: > >>>>>> > >>>>>> We didn't record it, we will try to record the following meetings. > >>>> Please > >>>>>> add your time preference in the docs, so that we can find a meeting > >>>> time > >>>>>> that works for more people. > >>>>>> > >>>>>> Thanks, > >>>>>> Botong > >>>>>> > >>>>>> On Wed, Apr 28, 2021 at 12:23 AM Viliam Durina < > >> vil...@hazelcast.com> > >>>>> wrote: > >>>>>> > >>>>>>> Is there a recording available? > >>>>>>> Viliam > >>>>>>> > >>>>>>> On Wed, 28 Apr 2021 at 00:15, Botong Huang <pku...@gmail.com> > >>>> wrote: > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> The meeting yesterday was fun and productive. As discussed, this > >>>> is > >>>>> the > >>>>>>>> call to schedule our second meeting. > >>>>>>>> > >>>>>>>> We encourage everyone to add their time preferences during > >> 05/01 - > >>>>> 05/15 > >>>>>>>> here: > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1wyNjB94uSGwHtVvGYDwaLlCghUJE-7aDLnCdKKXJN1o/edit?usp=sharing > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Botong > >>>>>>>> > >>>>>>>> On Wed, Apr 21, 2021 at 5:19 PM Botong Huang <pku...@gmail.com> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi all, > >>>>>>>>> We've created a zoom meeting below for our meeting next Monday > >>>>>>>>> (9pm-10:30pm PST on 04/26). > >>>>>>>>> Talk to you all soon! > >>>>>>>>> > >>>>>>>>> Join Zoom Meeting > >>>>>>>>> https://uci.zoom.us/j/91279732686 > >>>>>>>>> < > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fj%2F91279732686&sa=D&source=calendar&usd=2&usg=AOvVaw2C5LoOmCaSLWSi-YvMmsOE > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Meeting ID: 912 7973 2686 > >>>>>>>>> One tap mobile > >>>>>>>>> +16699006833 <(669)%20900-6833> > <(669)%20900-6833>,,91279732686# US (San Jose) > >>>>>>>>> +12532158782 <(253)%20215-8782> > <(253)%20215-8782>,,91279732686# US (Tacoma) > >>>>>>>>> > >>>>>>>>> Dial by your location > >>>>>>>>> +1 669 900 6833 <(669)%20900-6833> <(669)%20900-6833> US (San > Jose) > >>>>>>>>> +1 253 215 8782 <(253)%20215-8782> <(253)%20215-8782> US > (Tacoma) > >>>>>>>>> +1 346 248 7799 <(346)%20248-7799> <(346)%20248-7799> US > (Houston) > >>>>>>>>> +1 301 715 8592 <(301)%20715-8592> <(301)%20715-8592> US > (Washington DC) > >>>>>>>>> +1 312 626 6799 <(312)%20626-6799> <(312)%20626-6799> US > (Chicago) > >>>>>>>>> +1 646 558 8656 <(646)%20558-8656> <(646)%20558-8656> US (New > York) > >>>>>>>>> Meeting ID: 912 7973 2686 > >>>>>>>>> Find your local number: https://uci.zoom.us/u/aykHTkJBh > >>>>>>>>> < > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fu%2FaykHTkJBh&sa=D&source=calendar&usd=2&usg=AOvVaw0y_V5CisCHRyt9wsXLa9UM > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Join by Skype for Business > >>>>>>>>> https://uci.zoom.us/skype/91279732686 > >>>>>>>>> < > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://www.google.com/url?q=https%3A%2F%2Fuci.zoom.us%2Fskype%2F91279732686&sa=D&source=calendar&usd=2&usg=AOvVaw3iQwsDViu3K7-Rb_Iy6Zsy > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Botong > >>>>>>>>> > >>>>>>>>> On Tue, Apr 13, 2021 at 10:16 PM Botong Huang < > >> pku...@gmail.com > >>>>> > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> According to the preferences collected, we are tentatively > >>>>> scheduling > >>>>>>>> our > >>>>>>>>>> meeting at 9pm-10:30pm PST on 04/26 Monday. > >>>>>>>>>> > >>>>>>>>>> We will give a presentation about Tempura, followed by a free > >>>>>>>> discussion. > >>>>>>>>>> > >>>>>>>>>> Please let us know if there are new other requests. Few days > >>>>> before > >>>>>>>>>> the meeting, I will send out a zoom meeting link. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Botong > >>>>>>>>>> > >>>>>>>>>> On Wed, Apr 7, 2021 at 2:46 PM Botong Huang < > >> pku...@gmail.com> > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Julian and all, > >>>>>>>>>>> > >>>>>>>>>>> We've posted the Tempura code base below. Feel free to take > >> a > >>>>> quick > >>>>>>>> peek > >>>>>>>>>>> at the last five commits. > >>>>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://github.com/alibaba/cost-based-incremental-optimizer/commits/main > >>>>>>>>>>> > >>>>>>>>>>> I've also opened a Jira (CALCITE-4568 > >>>>>>>>>>> <https://issues.apache.org/jira/browse/CALCITE-4568>), > >> which > >>>>> will > >>>>>>>> serve > >>>>>>>>>>> as the umbrella Jira for the feature. > >>>>>>>>>>> > >>>>>>>>>>> In the meantime, we encourage everyone to enter the time > >>>>> preferences > >>>>>>>> for > >>>>>>>>>>> our first meeting here: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1wyNjB94uSGwHtVvGYDwaLlCghUJE-7aDLnCdKKXJN1o/edit?usp=sharing > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Botong > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Apr 5, 2021 at 3:59 PM Julian Hyde < > >>>>> jhyde.apa...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> I have added my time preferences to the doc. > >>>>>>>>>>>> > >>>>>>>>>>>> Before we meet, could you publish a PR for us to review? > >>>>>>>>>>>> > >>>>>>>>>>>> Initial discussions will need to be about architecture and > >>>>>>> high-level > >>>>>>>>>>>> design. So I would ask Calcite reviewers not to review the > >> PR > >>>>>>>> line-by-line > >>>>>>>>>>>> (or to leave comments in GitHub) but try to understand the > >>>>> design > >>>>>>>>>>>> holistically, and prepare questions/comments before the > >>>> meeting. > >>>>>>>>>>>> > >>>>>>>>>>>> Botong, Can you please create a Calcite JIRA case for this > >>>> task? > >>>>>>> JIRA > >>>>>>>>>>>> how we track long-running tasks such as this. > >>>>>>>>>>>> > >>>>>>>>>>>> Julian > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> On Apr 3, 2021, at 5:15 PM, Botong Huang < > >> pku...@gmail.com > >>>>> > >>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Apology for the delay. It took us some time to clean up > >> our > >>>>> code > >>>>>>>> base > >>>>>>>>>>>> and > >>>>>>>>>>>>> publicly release it (which will be out soon) for a quick > >>>> peek. > >>>>>>>>>>>>> > >>>>>>>>>>>>> We are ready to present our work. Let's schedule a time > >>>> for a > >>>>> Zoom > >>>>>>>>>>>>> meeting and discuss how to integrate Tempura into > >> Calcite. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Since some of our team members are in China, we prefer > >> the > >>>>> time > >>>>>>> slot > >>>>>>>>>>>> of > >>>>>>>>>>>>> 7:00pm-11:30pm PST any day. I've added our time > >> preference > >>>> in > >>>>> the > >>>>>>>>>>>> shared > >>>>>>>>>>>>> doc below. > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1wyNjB94uSGwHtVvGYDwaLlCghUJE-7aDLnCdKKXJN1o/edit?usp=sharing > >>>>>>>>>>>>> > >>>>>>>>>>>>> We encourage everyone to add their time preferences > >> (during > >>>>>>>>>>>> 04/15-04/30) in > >>>>>>>>>>>>> this doc. In a week or so, we will try to settle a time > >>>> that > >>>>> works > >>>>>>>> for > >>>>>>>>>>>>> most. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> Botong > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sat, Jan 30, 2021 at 9:19 PM Botong Huang < > >>>>> pku...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Julian and Rui, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sounds good to us. Please give us some time to prepare > >>>> some > >>>>>>> slides > >>>>>>>>>>>> for the > >>>>>>>>>>>>>> meeting. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I've created a doc below for discussion. Please feel > >> free > >>>> to > >>>>> add > >>>>>>>>>>>> more in > >>>>>>>>>>>>>> here: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1wyNjB94uSGwHtVvGYDwaLlCghUJE-7aDLnCdKKXJN1o/edit?usp=sharing > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> Botong > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Jan 28, 2021 at 11:18 AM Julian Hyde < > >>>>>>>> jhyde.apa...@gmail.com > >>>>>>>>>>>>> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> PS The “editable doc” that Rui refers to is also a good > >>>>> idea. I > >>>>>>>>>>>> think we > >>>>>>>>>>>>>>> should create it to continue discussion after the first > >>>>> meeting. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Julian > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Jan 28, 2021, at 11:16 AM, Julian Hyde < > >>>>>>>> jhyde.apa...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I think good next steps would be a PR and a meeting. > >>>> The > >>>>> PR > >>>>>>> will > >>>>>>>>>>>> allow > >>>>>>>>>>>>>>> us to read the code, but I think we should do the first > >>>>> round of > >>>>>>>>>>>> questions > >>>>>>>>>>>>>>> at the meeting. The meeting could perhaps start with a > >>>>>>>>>>>> presentation of the > >>>>>>>>>>>>>>> paper (do you have some slides you are planning to > >>>> present > >>>>> at > >>>>>>>> VLDB, > >>>>>>>>>>>>>>> Botong?) and then move on to questions about the > >>>> concepts, > >>>>> which > >>>>>>>>>>>>>>> alternatives were considered, and how the concepts map > >>>> onto > >>>>>>> other > >>>>>>>>>>>> current > >>>>>>>>>>>>>>> and future concepts in calcite. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I don’t think we should start “reviewing” the PR > >>>>> line-by-line > >>>>>>> at > >>>>>>>>>>>> this > >>>>>>>>>>>>>>> point. We need to understand the high-level concepts > >> and > >>>>> design > >>>>>>>>>>>> choices. If > >>>>>>>>>>>>>>> we start reviewing the PR we will get lost in the > >>>> details. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I know that integrating a major change is hard; I > >> doubt > >>>>> that we > >>>>>>>>>>>> will be > >>>>>>>>>>>>>>> able to integrate everything, but we can build > >>>> understanding > >>>>>>> about > >>>>>>>>>>>> where > >>>>>>>>>>>>>>> calcite needs to go, and I hope integrate a good amount > >>>> of > >>>>> code > >>>>>>> to > >>>>>>>>>>>> help us > >>>>>>>>>>>>>>> get there. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> As I said before, after the integration I would like > >>>>> people to > >>>>>>> be > >>>>>>>>>>>> able > >>>>>>>>>>>>>>> to experiment with it and use it in their production > >>>>> systems. > >>>>>>>> That > >>>>>>>>>>>> way, it > >>>>>>>>>>>>>>> will not be an experiment that withers, but a feature > >> set > >>>>>>>>>>>> integrates with > >>>>>>>>>>>>>>> other calcite features and gets stronger over time. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Julian > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Jan 28, 2021, at 10:54 AM, Rui Wang < > >>>>> amaliu...@apache.org> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> For me to participate in the discussion for the > >> above > >>>>>>>> questions, > >>>>>>>>>>>> I > >>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>>> need to read a lot more to know relevant context and > >>>>> likely > >>>>>>> ask > >>>>>>>>>>>> lots of > >>>>>>>>>>>>>>>>> questions :-). A editable doc is probably good for > >>>>> questions > >>>>>>>> and > >>>>>>>>>>>> back > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> forward discussion. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -Rui > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Thu, Jan 28, 2021 at 10:50 AM Rui Wang < > >>>>>>>> amaliu...@apache.org > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I am also happy to help push this work into Calcite > >>>>> (review > >>>>>>>> code > >>>>>>>>>>>> and > >>>>>>>>>>>>>>> doc, > >>>>>>>>>>>>>>>>>> etc.). > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> While you can share your code so people can have > >> more > >>>>> idea > >>>>>>> how > >>>>>>>>>>>> it is > >>>>>>>>>>>>>>>>>> implemented, I think it would be also nice to have a > >>>> doc > >>>>> to > >>>>>>>>>>>> discuss > >>>>>>>>>>>>>>> open > >>>>>>>>>>>>>>>>>> questions above. Some points that I copy those to > >>>> here: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> 1. Can this solution be compatible with existing > >>>>> solutions in > >>>>>>>>>>>> Calcite > >>>>>>>>>>>>>>>>>> Streaming, materialized view maintenance, and > >>>> multi-query > >>>>>>>>>>>> optimization > >>>>>>>>>>>>>>>>>> (Sigma and Delta relational operators, lattice, and > >>>> Spool > >>>>>>>>>>>> operator), > >>>>>>>>>>>>>>>>>> 2. Did you find that you needed two separate cost > >>>> models > >>>>> - > >>>>>>> one > >>>>>>>>>>>> for > >>>>>>>>>>>>>>> “view > >>>>>>>>>>>>>>>>>> maintenance” and another for “user queries” - since > >>>> the > >>>>>>>>>>>> objectives of > >>>>>>>>>>>>>>> each > >>>>>>>>>>>>>>>>>> activity are so different? > >>>>>>>>>>>>>>>>>> 3. whether this work will hasten the arrival of > >>>>>>> multi-objective > >>>>>>>>>>>>>>> parametric > >>>>>>>>>>>>>>>>>> query optimization [1] in Calcite. > >>>>>>>>>>>>>>>>>> 4. probably SQL shell support. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> [1]: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cacm.acm.org/magazines/2017/10/221322-multi-objective-parametric-query-optimization/fulltext > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -Rui > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> On Wed, Jan 27, 2021 at 6:52 PM Albert < > >>>>> zinki...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> it would be very nice to see a POC of your work. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Thu, Jan 28, 2021 at 10:21 AM Botong Huang < > >>>>>>>>>>>> pku...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi Julian, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Just wondering if there are any updates? We are > >>>>> wondering > >>>>>>> if > >>>>>>>> it > >>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>>>>>> help > >>>>>>>>>>>>>>>>>>>> to post our code for a quick preview. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>> Botong > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> On Fri, Jan 1, 2021 at 11:04 AM Botong Huang < > >>>>>>>> pku...@gmail.com > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi Julian, > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks for your interest! Sure let's figure out a > >>>> plan > >>>>>>> that > >>>>>>>>>>>> best > >>>>>>>>>>>>>>>>>>> benefits > >>>>>>>>>>>>>>>>>>>>> the community. Here are some clarifications that > >>>>> hopefully > >>>>>>>>>>>> answer > >>>>>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>>>> questions. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> In our work (Tempura), users specify the set of > >>>> time > >>>>>>> points > >>>>>>>> to > >>>>>>>>>>>>>>>>>>> consider > >>>>>>>>>>>>>>>>>>>>> running and a cost function that expresses users' > >>>>>>> preference > >>>>>>>>>>>> over > >>>>>>>>>>>>>>>>>>> time, > >>>>>>>>>>>>>>>>>>>>> Tempura will generate the best incremental plan > >>>> that > >>>>>>>>>>>> minimizes the > >>>>>>>>>>>>>>>>>>>> overall > >>>>>>>>>>>>>>>>>>>>> cost function. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> In this incremental plan, the sub-plans at > >>>> different > >>>>> time > >>>>>>>>>>>> points > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>> different from each other, as opposed to > >> identical > >>>>> plans > >>>>>>> in > >>>>>>>>>>>> all > >>>>>>>>>>>>>>> delta > >>>>>>>>>>>>>>>>>>>> runs > >>>>>>>>>>>>>>>>>>>>> as in streaming or IVM. As mentioned in $2.1 of > >> the > >>>>>>> Tempura > >>>>>>>>>>>> paper, > >>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>> mimic the current streaming implementation by > >>>>> specifying > >>>>>>> two > >>>>>>>>>>>>>>> (logical) > >>>>>>>>>>>>>>>>>>>> time > >>>>>>>>>>>>>>>>>>>>> points in Tempura, representing the initial run > >> and > >>>>> later > >>>>>>>>>>>> delta > >>>>>>>>>>>>>>> runs > >>>>>>>>>>>>>>>>>>>>> respectively. In general, note that Tempura > >>>> supports > >>>>>>> various > >>>>>>>>>>>> form > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>> incremental computing, not only the small-delta > >>>>>>> append-only > >>>>>>>>>>>> data > >>>>>>>>>>>>>>>>>>> model in > >>>>>>>>>>>>>>>>>>>>> streaming systems. That's why we believe Tempura > >>>>> subsumes > >>>>>>>> the > >>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>>>>>>> streaming support, as well as any IVM > >>>> implementations. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> About the cost model, we did not come up with a > >>>>> seperate > >>>>>>>> cost > >>>>>>>>>>>>>>> model, > >>>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>> rather extended the existing one. Similar to > >>>>>>> multi-objective > >>>>>>>>>>>>>>>>>>>> optimization, > >>>>>>>>>>>>>>>>>>>>> costs incurred at different time points are > >>>> considered > >>>>>>>>>>>> different > >>>>>>>>>>>>>>>>>>>>> dimensions. Tempura lets users supply a function > >>>> that > >>>>>>>>>>>> converts this > >>>>>>>>>>>>>>>>>>> cost > >>>>>>>>>>>>>>>>>>>>> vector into a final cost. So under this function, > >>>> any > >>>>> two > >>>>>>>>>>>>>>> incremental > >>>>>>>>>>>>>>>>>>>> plans > >>>>>>>>>>>>>>>>>>>>> are still comparable and there is an overall > >>>> optimum. > >>>>> I > >>>>>>>> guess > >>>>>>>>>>>> we > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>> go > >>>>>>>>>>>>>>>>>>>>> down the route of multi-objective parametric > >> query > >>>>>>>>>>>> optimization > >>>>>>>>>>>>>>>>>>> instead > >>>>>>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>>>>> there is a need. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Next on materialized views and multi-query > >>>>> optimization, > >>>>>>>>>>>> since our > >>>>>>>>>>>>>>>>>>>>> multi-time-point plan naturally involves > >>>> materializing > >>>>>>>>>>>> intermediate > >>>>>>>>>>>>>>>>>>>> results > >>>>>>>>>>>>>>>>>>>>> for later time points, we need to solve the > >>>> problem of > >>>>>>>>>>>> choosing > >>>>>>>>>>>>>>>>>>>>> materializations and include the cost of saving > >> and > >>>>>>> reusing > >>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>> materializations when costing and comparing > >> plans. > >>>> We > >>>>>>>>>>>> borrowed the > >>>>>>>>>>>>>>>>>>>>> multi-query optimization techniques to solve this > >>>>> problem > >>>>>>>> even > >>>>>>>>>>>>>>> though > >>>>>>>>>>>>>>>>>>> we > >>>>>>>>>>>>>>>>>>>>> are looking at a single query. As a result, we > >>>> think > >>>>> our > >>>>>>>> work > >>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>> orthogonal > >>>>>>>>>>>>>>>>>>>>> to Calcite's facilities around utilizing existing > >>>>> views, > >>>>>>>>>>>> lattice > >>>>>>>>>>>>>>> etc. > >>>>>>>>>>>>>>>>>>> We > >>>>>>>>>>>>>>>>>>>> do > >>>>>>>>>>>>>>>>>>>>> feel that the multi-query optimization component > >>>> can > >>>>> be > >>>>>>>>>>>> adopted to > >>>>>>>>>>>>>>>>>>> wider > >>>>>>>>>>>>>>>>>>>>> use, but probably need more suggestions from the > >>>>>>> community. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Lastly, our current implementation is set up in > >>>> java > >>>>> code, > >>>>>>>> it > >>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>>> straightforward to hook it up with SQL shell. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>>>>>>> Botong > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> On Mon, Dec 28, 2020 at 6:44 PM Julian Hyde < > >>>>>>>>>>>>>>> jhyde.apa...@gmail.com> > >>>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Botong, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> This is very exciting; congratulations on this > >>>>> research, > >>>>>>>> and > >>>>>>>>>>>> thank > >>>>>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>>>>>>>> for contributing it back to Calcite. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> The research touches several areas in Calcite: > >>>>> streaming, > >>>>>>>>>>>>>>>>>>> materialized > >>>>>>>>>>>>>>>>>>>>>> view maintenance, and multi-query optimization. > >>>> As we > >>>>>>> have > >>>>>>>>>>>> already > >>>>>>>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>>>>>>>> solutions in those areas (Sigma and Delta > >>>> relational > >>>>>>>>>>>> operators, > >>>>>>>>>>>>>>>>>>> lattice, > >>>>>>>>>>>>>>>>>>>>>> and Spool operator), it will be interesting to > >> see > >>>>>>> whether > >>>>>>>>>>>> we can > >>>>>>>>>>>>>>>>>>> make > >>>>>>>>>>>>>>>>>>>> them > >>>>>>>>>>>>>>>>>>>>>> compatible, or whether one concept can subsume > >>>>> others. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Your work differs from streaming queries in that > >>>> your > >>>>>>>>>>>> relations > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>> used > >>>>>>>>>>>>>>>>>>>>>> by “external” user queries, whereas in pure > >>>> streaming > >>>>>>>>>>>> queries, the > >>>>>>>>>>>>>>>>>>> only > >>>>>>>>>>>>>>>>>>>>>> activity is the change propagation. Did you find > >>>>> that you > >>>>>>>>>>>> needed > >>>>>>>>>>>>>>> two > >>>>>>>>>>>>>>>>>>>>>> separate cost models - one for “view > >> maintenance” > >>>> and > >>>>>>>>>>>> another for > >>>>>>>>>>>>>>>>>>> “user > >>>>>>>>>>>>>>>>>>>>>> queries” - since the objectives of each activity > >>>> are > >>>>> so > >>>>>>>>>>>> different? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I wonder whether this work will hasten the > >>>> arrival of > >>>>>>>>>>>>>>> multi-objective > >>>>>>>>>>>>>>>>>>>>>> parametric query optimization [1] in Calcite. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I will make time over the next few days to read > >>>> and > >>>>>>> digest > >>>>>>>>>>>> your > >>>>>>>>>>>>>>>>>>> paper. > >>>>>>>>>>>>>>>>>>>>>> Then I expect that we will have a back-and-forth > >>>>> process > >>>>>>> to > >>>>>>>>>>>> create > >>>>>>>>>>>>>>>>>>>>>> something that will be useful for the broader > >>>>> community. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> One thing will be particularly useful: making > >> this > >>>>>>>>>>>> functionality > >>>>>>>>>>>>>>>>>>>>>> available from a SQL shell, so that people can > >>>>> experiment > >>>>>>>>>>>> with > >>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>>>>>> functionality without writing Java code or > >>>> setting up > >>>>>>>> complex > >>>>>>>>>>>>>>>>>>> databases > >>>>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>>>> metadata. I have in mind something like the > >> simple > >>>>> DDL > >>>>>>>>>>>> operations > >>>>>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>>>>> available in Calcite’s ’server’ module. I wonder > >>>>> whether > >>>>>>> we > >>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>>> devise > >>>>>>>>>>>>>>>>>>>>>> some kind of SQL syntax for a “multi-query”. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Julian > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cacm.acm.org/magazines/2017/10/221322-multi-objective-parametric-query-optimization/fulltext > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Dec 23, 2020, at 8:55 PM, Botong Huang < > >>>>>>>> pku...@gmail.com > >>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thanks Aron for pointing this out. To see the > >>>>> figure, > >>>>>>>> please > >>>>>>>>>>>>>>> refer > >>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> Fig > >>>>>>>>>>>>>>>>>>>>>>> 3(a) in our paper: > >>>>>>>>>>>>>>>>>>>>>> > >>>>> https://kai-zeng.github.io/papers/tempura-vldb2021.pdf > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>> Botong > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> On Wed, Dec 23, 2020 at 7:20 PM JiaTao Tao < > >>>>>>>>>>>> taojia...@gmail.com> > >>>>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Seems interesting, the pic can not be seen in > >>>> the > >>>>> mail, > >>>>>>>>>>>> may you > >>>>>>>>>>>>>>>>>>> open > >>>>>>>>>>>>>>>>>>>> a > >>>>>>>>>>>>>>>>>>>>>> JIRA > >>>>>>>>>>>>>>>>>>>>>>>> for this, people who are interested in this > >> can > >>>>>>> subscribe > >>>>>>>>>>>> to the > >>>>>>>>>>>>>>>>>>>> JIRA? > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Regards! > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Aron Tao > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Botong Huang <bot...@apache.org> > >> 于2020年12月24日周四 > >>>>>>>> 上午3:18写道： > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> This is a proposal to extend the Calcite > >>>> optimizer > >>>>>>> into > >>>>>>>> a > >>>>>>>>>>>>>>> general > >>>>>>>>>>>>>>>>>>>>>>>>> incremental query optimizer, based on our > >>>> research > >>>>>>> paper > >>>>>>>>>>>>>>>>>>> published > >>>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> VLDB > >>>>>>>>>>>>>>>>>>>>>>>>> 2021: > >>>>>>>>>>>>>>>>>>>>>>>>> Tempura: a general cost-based optimizer > >>>> framework > >>>>> for > >>>>>>>>>>>>>>> incremental > >>>>>>>>>>>>>>>>>>>> data > >>>>>>>>>>>>>>>>>>>>>>>>> processing > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> We also have a demo in SIGMOD 2020 > >> illustrating > >>>>> how > >>>>>>>>>>>> Alibaba’s > >>>>>>>>>>>>>>>>>>> data > >>>>>>>>>>>>>>>>>>>>>>>>> warehouse is planning to use this incremental > >>>>> query > >>>>>>>>>>>> optimizer > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>> alleviate > >>>>>>>>>>>>>>>>>>>>>>>>> cluster-wise resource skewness: > >>>>>>>>>>>>>>>>>>>>>>>>> Grosbeak: A Data Warehouse Supporting > >>>>> Resource-Aware > >>>>>>>>>>>>>>> Incremental > >>>>>>>>>>>>>>>>>>>>>>>> Computing > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> To our best knowledge, this is the first > >>>> general > >>>>>>>>>>>> cost-based > >>>>>>>>>>>>>>>>>>>>>> incremental > >>>>>>>>>>>>>>>>>>>>>>>>> optimizer that can find the best plan across > >>>>> multiple > >>>>>>>>>>>> families > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>> incremental computing methods, including IVM, > >>>>>>> Streaming, > >>>>>>>>>>>>>>>>>>> DBToaster, > >>>>>>>>>>>>>>>>>>>>>> etc. > >>>>>>>>>>>>>>>>>>>>>>>>> Experiments (in the paper) shows that the > >>>>> generated > >>>>>>> best > >>>>>>>>>>>> plan > >>>>>>>>>>>>>>> is > >>>>>>>>>>>>>>>>>>>>>>>>> consistently much better than the plans from > >>>> each > >>>>>>>>>>>> individual > >>>>>>>>>>>>>>>>>>> method > >>>>>>>>>>>>>>>>>>>>>>>> alone. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> In general, incremental query planning is > >>>> central > >>>>> to > >>>>>>>>>>>> database > >>>>>>>>>>>>>>>>>>> view > >>>>>>>>>>>>>>>>>>>>>>>>> maintenance and stream processing systems, > >> and > >>>> are > >>>>>>> being > >>>>>>>>>>>>>>> adopted > >>>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>>>>>> active > >>>>>>>>>>>>>>>>>>>>>>>>> databases, resumable query execution, > >>>> approximate > >>>>>>> query > >>>>>>>>>>>>>>>>>>> processing, > >>>>>>>>>>>>>>>>>>>>>> etc. > >>>>>>>>>>>>>>>>>>>>>>>> We > >>>>>>>>>>>>>>>>>>>>>>>>> are hoping that this feature can help > >> widening > >>>> the > >>>>>>>>>>>> spectrum of > >>>>>>>>>>>>>>>>>>>>>> Calcite, > >>>>>>>>>>>>>>>>>>>>>>>>> solicit more use cases and adoption of > >> Calcite. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Below is a brief description of the technical > >>>>> details. > >>>>>>>>>>>> Please > >>>>>>>>>>>>>>>>>>> refer > >>>>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> Tempura paper for more details. We are also > >>>>> working > >>>>>>> on a > >>>>>>>>>>>>>>> journal > >>>>>>>>>>>>>>>>>>>>>> version > >>>>>>>>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>>>>>>>> the paper with more implementation details. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Currently the query plan generated by Calcite > >>>> is > >>>>> meant > >>>>>>>> to > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>>> executed > >>>>>>>>>>>>>>>>>>>>>>>>> altogether at once. In the proposal, > >> Calcite’s > >>>>> memo > >>>>>>> will > >>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>>> extended > >>>>>>>>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>>>>>>>>>>> temporal information so that it is capable of > >>>>>>> generating > >>>>>>>>>>>>>>>>>>> incremental > >>>>>>>>>>>>>>>>>>>>>>>> plans > >>>>>>>>>>>>>>>>>>>>>>>>> that include multiple sub-plans to execute at > >>>>>>> different > >>>>>>>>>>>> time > >>>>>>>>>>>>>>>>>>> points. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> The main idea is to view each table as one > >> that > >>>>>>> changes > >>>>>>>>>>>> over > >>>>>>>>>>>>>>> time > >>>>>>>>>>>>>>>>>>>>>> (Time > >>>>>>>>>>>>>>>>>>>>>>>>> Varying Relations (TVR)). To achieve that we > >>>>>>> introduced > >>>>>>>>>>>>>>>>>>> TvrMetaSet > >>>>>>>>>>>>>>>>>>>>>> into > >>>>>>>>>>>>>>>>>>>>>>>>> Calcite’s memo besides RelSet and RelSubset > >> to > >>>>> track > >>>>>>>>>>>> related > >>>>>>>>>>>>>>>>>>> RelSets > >>>>>>>>>>>>>>>>>>>>>> of a > >>>>>>>>>>>>>>>>>>>>>>>>> changing table (e.g. snapshot of the table at > >>>>> certain > >>>>>>>>>>>> time, > >>>>>>>>>>>>>>>>>>> delta of > >>>>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>>> table between two time points, etc.). > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> [image: image.png] > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> For example in the above figure, each > >> vertical > >>>>> line > >>>>>>> is a > >>>>>>>>>>>>>>>>>>> TvrMetaSet > >>>>>>>>>>>>>>>>>>>>>>>>> representing a TVR (S, R, S left outer join > >> R, > >>>>> etc.). > >>>>>>>>>>>>>>> Horizontal > >>>>>>>>>>>>>>>>>>>> lines > >>>>>>>>>>>>>>>>>>>>>>>>> represent time. Each black dot in the grid > >> is a > >>>>>>> RelSet. > >>>>>>>>>>>> Users > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>>>> write > >>>>>>>>>>>>>>>>>>>>>>>> TVR > >>>>>>>>>>>>>>>>>>>>>>>>> Rewrite Rules to describe valid > >> transformations > >>>>>>> between > >>>>>>>>>>>> these > >>>>>>>>>>>>>>>>>>> dots. > >>>>>>>>>>>>>>>>>>>>>> For > >>>>>>>>>>>>>>>>>>>>>>>>> example, the blues lines are inter-TVR rules > >>>> that > >>>>>>>>>>>> describe how > >>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>>>>>> compute > >>>>>>>>>>>>>>>>>>>>>>>>> certain RelSet of a TVR from RelSets of other > >>>>> TVRs. > >>>>>>> The > >>>>>>>>>>>> red > >>>>>>>>>>>>>>> lines > >>>>>>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>>>>>>>>>>> intra-TVR rules that describe transformations > >>>>> within a > >>>>>>>>>>>> TVR. All > >>>>>>>>>>>>>>>>>>> TVR > >>>>>>>>>>>>>>>>>>>>>>>> rewrite > >>>>>>>>>>>>>>>>>>>>>>>>> rules are logical rules. All existing Calcite > >>>>> rules > >>>>>>>> still > >>>>>>>>>>>> work > >>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>>>>>>>>>>> volcano system without modification. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> All changes in this feature will consist of > >>>> four > >>>>>>> parts: > >>>>>>>>>>>>>>>>>>>>>>>>> 1. Memo extension with TvrMetaSet > >>>>>>>>>>>>>>>>>>>>>>>>> 2. Rule engine upgrade, capable of matching > >>>>> TvrMetaSet > >>>>>>>> and > >>>>>>>>>>>>>>>>>>> RelNodes, > >>>>>>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>>>>>>>>> well as links in between the nodes. > >>>>>>>>>>>>>>>>>>>>>>>>> 3. A basic set of TvrRules, written using the > >>>>> upgraded > >>>>>>>>>>>> rule > >>>>>>>>>>>>>>>>>>> engine > >>>>>>>>>>>>>>>>>>>>>> API. > >>>>>>>>>>>>>>>>>>>>>>>>> 4. Multi-query optimization, used to find the > >>>> best > >>>>>>>>>>>> incremental > >>>>>>>>>>>>>>>>>>> plan > >>>>>>>>>>>>>>>>>>>>>>>>> involving multiple time points. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Note that this feature is an extension in > >>>> nature > >>>>> and > >>>>>>>> thus > >>>>>>>>>>>> when > >>>>>>>>>>>>>>>>>>>>>> disabled, > >>>>>>>>>>>>>>>>>>>>>>>>> does not change any existing Calcite > >> behavior. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Other than scenarios in the paper, we also > >>>> applied > >>>>>>> this > >>>>>>>>>>>>>>>>>>>>>> Calcite-extended > >>>>>>>>>>>>>>>>>>>>>>>>> incremental query optimizer to a type of > >>>> periodic > >>>>>>> query > >>>>>>>>>>>> called > >>>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>>>>>>>> ‘‘range > >>>>>>>>>>>>>>>>>>>>>>>>> query’’ in Alibaba’s data warehouse. It > >>>> achieved > >>>>> cost > >>>>>>>>>>>> savings > >>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>>> 80% > >>>>>>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>>>>>>>>> total CPU and memory consumption, and 60% on > >>>>>>> end-to-end > >>>>>>>>>>>>>>> execution > >>>>>>>>>>>>>>>>>>>>>> time. > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> All comments and suggestions are welcome. > >>>> Thanks > >>>>> and > >>>>>>>> happy > >>>>>>>>>>>>>>>>>>> holidays! > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>>> Botong > >>>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~ > >>>>>>>>>>>>>>>>>>> no mistakes > >>>>>>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Viliam Durina > >>>>>>> Jet Developer > >>>>>>> hazelcast® > >>>>>>> > >>>>>>> <https://www.hazelcast.com> 2 W 5th Ave, Ste 300 | San Mateo, > >> CA > >>>>> 94402 | > >>>>>>> USA > >>>>>>> +1 (650) 521-5453 <(650)%20521-5453> <(650)%20521-5453> | > hazelcast.com < > >> https://www.hazelcast.com> > >>>>>>> > >>>>>>> -- > >>>>>>> This message contains confidential information and is intended > >> only > >>>> for > >>>>>>> the > >>>>>>> individuals named. If you are not the named addressee you should > >> not > >>>>>>> disseminate, distribute or copy this e-mail. Please notify the > >>>> sender > >>>>>>> immediately by e-mail if you have received this e-mail by mistake > >>>> and > >>>>>>> delete this e-mail from your system. E-mail transmission cannot be > >>>>>>> guaranteed to be secure or error-free as information could be > >>>>> intercepted, > >>>>>>> corrupted, lost, destroyed, arrive late or incomplete, or contain > >>>>> viruses. > >>>>>>> The sender therefore does not accept liability for any errors or > >>>>> omissions > >>>>>>> in the contents of this message, which arise as a result of e-mail > >>>>>>> transmission. If verification is required, please request a > >>>> hard-copy > >>>>>>> version. -Hazelcast > >>>>>>> > >>>>> > >>>> > >>> > >> > >
