In order to use the planner you can instantiate directly without necessarily passing by Frameworks. For an end to end example on how to parse, validate, plan, execute a query using the basic primitives provided by Calcite have a look in [1].
[1] https://github.com/zabetak/calcite/blob/livecodingdemo/core/src/test/java/org/apache/calcite/examples/foodmart/java/EndToEndExampleEnumerable.java On Wed, May 8, 2019 at 9:41 AM Mark Pasterkamp <[email protected]> wrote: > I don't necessarily want to pass relnodes from one planner to another, but > I see no other way how to do it. > From my understanding, if you have a sql string you need a planner to parse > and validate it into a relnode. > The PlannerImpl created by Frameworks.getPlanner() has an empty internal > planner which gets initialized via the Frameworks.withPlanner function[1]. > > Frameworks.withPlanner then subsequently calls the Frameworks.withPrepare > [2] which calls the CalcitePrepareImpl.perform() [3]. > There the planner gets created using the createPlanner method [4]. It is > there that the planner is finally created as a VolcanoPlanner [5]. > Maybe it can be considered to make this planner configurable in one of the > config settings instead of it being a hardcoded VolcanoPlanner. > > The one thing I can still try is to use the RelOptCostImpl.Factory [6] to > make the VolcanoPlanner return RelOptCostImpl instead of VolcanoCost. > > Or perhaps there is another way how I can parse a string into RelNodes that > I do not know about, then I would love to know. > > [1] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L137 > [2] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L114 > [3] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L157 > [4] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1038 > [5] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L520 > [6] > > https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/plan/RelOptCostImpl.java#L122 > > > > On Tue, 7 May 2019 at 13:21, Stamatis Zampetakis <[email protected]> > wrote: > > > Thanks for looking more into this Mark. > > > > Passing RelNodes from one planner to another is not a good idea. > > Ideally this should be possible but I guess it requires CALCITE-1536 [1] > to > > be solved. > > > > Another way to avoid problems like the one you observed is to copy > RelNodes > > from one cluster to another [2]. > > > > Best, > > Stamatis > > > > [1] https://issues.apache.org/jira/browse/CALCITE-1536 > > [2] https://issues.apache.org/jira/browse/CALCITE-1681 > > > > > > On Mon, May 6, 2019 at 12:02 PM Mark Pasterkamp < > > [email protected]> wrote: > > > > > I think I found the origin of the exception. > > > To convert a single query to a rel node I am using the PlannerImp > > provided > > > by Frameworks.getPlanner(). > > > This planner provides a VolcanoPlanner to the relnode clusters. > > > When calling the HepPlanner.findBestExp() it will eventually call > > > the applyTransformationResults which will compute a cost using the > > > rel.getCluster().getPlanner() (which is a VolcanoPlanner). > > > > > > Mark > > > > > > On Fri, 3 May 2019 at 19:56, Mark Pasterkamp < > > [email protected] > > > > > > > wrote: > > > > > > > I did not know this is how it works, I just copied the example above. > > > > Would there be an easy way to create a RelNode containg a tablescan > > over > > > > the materialized view "mv"? > > > > Trying to create one using for instance a relbuilder gives a calcite > > > > exception. > > > > Otherwise I might just look for another test file in which I can get > > > > access to the schema and use the materiaqlizationservice. > > > > > > > > Mark > > > > > > > > On Fri, 3 May 2019 at 18:35, Jesus Camacho Rodriguez > > > > <[email protected]> wrote: > > > > > > > >> bq . Since we are talking about materialized views, I think in most > > > cases > > > >> tableRel should be simply a LogicalTableScan. > > > >> > > > >> Stamatis is correct about this, I had not realized tableRel == > > queryRel > > > >> in > > > >> your sample code. > > > >> > > > >> Thanks, > > > >> Jesús > > > >> > > > >> On Fri, May 3, 2019 at 6:12 AM Stamatis Zampetakis < > [email protected] > > > > > > >> wrote: > > > >> > > > >> > I think the main problem comes from the fact that tableRel == > > queryRel > > > >> in > > > >> > the test case you provided. > > > >> > Defining the materialized view like that basically says that when > > you > > > >> find > > > >> > a part of the query that satisfies queryRel replace it with > itself. > > > >> > In conjunction with the rule that is used, which allows partial > > > >> rewritings > > > >> > using union, you end up with a rule that matches infinite number > of > > > >> times. > > > >> > Since we are talking about materialized views, I think in most > cases > > > >> > tableRel should be simply a LogicalTableScan. > > > >> > The idea is that expression represented by queryRel is > materialized > > > >> into a > > > >> > table so in order to retrieve the results we only need to scan the > > > >> table. > > > >> > > > > >> > Regarding the "if (true)" statements that you observed, most > likely > > > they > > > >> > were introduced as release toggles [1]. > > > >> > However, since the last commit was in 2013 I think by now it is > safe > > > to > > > >> > refactor that part and remove dead code. > > > >> > > > > >> > [1] https://www.martinfowler.com/articles/feature-toggles.html > > > >> > > > > >> > Best, > > > >> > Stamatis > > > >> > > > > >> > On Fri, May 3, 2019 at 12:50 PM Mark Pasterkamp < > > > >> > [email protected]> wrote: > > > >> > > > > >> > > Dear Jesus, > > > >> > > > > > >> > > I think your intuition in this regard is correct. > > > >> > > After executing the main program in the HepPlanner the resulting > > > plan > > > >> > > contains a lot of circular references. > > > >> > > Changing the matching order does not influence this behaviour. > > > >> > > > > > >> > > > > > >> > > Mark > > > >> > > > > > >> > > On Thu, 2 May 2019 at 22:14, Jesus Camacho Rodriguez > > > >> > > <[email protected]> wrote: > > > >> > > > > > >> > > > Mark, > > > >> > > > > > > >> > > > I have an intuition that this happens because the rule > creates a > > > >> > > partially > > > >> > > > contained rewriting with a union, where one side contains a > scan > > > >> over > > > >> > the > > > >> > > > materialized view and the other side contains the query itself > > > with > > > >> a > > > >> > > > filter on top excluding the data that is coming from the > > > >> materialized > > > >> > > view. > > > >> > > > Then the rule is triggered on the plan representing the > original > > > >> query > > > >> > > > again and the process is repeated. Have you tried changing the > > > >> matching > > > >> > > > order for your hep program? > > > >> > > > > > > >> > > > Thanks, > > > >> > > > Jesús > > > >> > > > > > > >> > > > On Thu, May 2, 2019 at 8:53 AM Mark Pasterkamp < > > > >> > > > [email protected]> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Hi Stamatis, > > > >> > > > > > > > >> > > > > I have tried to recreate the issue but I have not been able > to > > > do > > > >> > > that. I > > > >> > > > > was however able to create a new exception which I don't > quite > > > >> > > > understand. > > > >> > > > > The error happened when calcite was creating a union > rewriting > > > >> using > > > >> > > > > materialized views. But trying to recreate this situation > gave > > > me > > > >> > > another > > > >> > > > > interesting one. > > > >> > > > > This time, the planner rewrites one of the children nodes > into > > > >> > itself I > > > >> > > > > would assume which causes a stack overflow. The method > itself > > > can > > > >> be > > > >> > > > found > > > >> > > > > here: > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/mpasterkamp/calcite/blob/768b7928dbde5f6f9775a1119e7466d8eafafb4b/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java#L312 > > > >> > > > > > > > >> > > > > Perhaps I am doing something wrong, perhaps not? I am not > > > >> > knowledgeable > > > >> > > > > enough about this to understand why this is happening. Wish > I > > > >> could > > > >> > > help > > > >> > > > > more for that. > > > >> > > > > > > > >> > > > > Also, while investigating this issue I found another > > interesting > > > >> > > artifact > > > >> > > > > in de source code of the VolcanoCost. A lot of methods in > this > > > >> class > > > >> > > have > > > >> > > > > an "if (true)"-statement like here: > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/calcite/blob/4b4d8037c5073e4eb5702b12bc4ecade31476616/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoCost.java#L100 > > > >> > > > > > > > >> > > > > Now I was just curious, is there any reason for this to be > > there > > > >> that > > > >> > > you > > > >> > > > > know of? > > > >> > > > > > > > >> > > > > Thank you for responding and congratulations for your recent > > > >> > > promotions. > > > >> > > > > > > > >> > > > > > > > >> > > > > With kind regards, > > > >> > > > > > > > >> > > > > Mark > > > >> > > > > > > > >> > > > > > > > >> > > > > On Thu, 2 May 2019 at 14:58, Stamatis Zampetakis < > > > >> [email protected]> > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > Said like that it looks like a bug. > > > >> > > > > > > > > >> > > > > > I think the best would be to reproduce the exception as a > > unit > > > >> test > > > >> > > in > > > >> > > > > > HepPlannerTest [1], RelOptRulesTest [2], or PlannerTest > [3] > > so > > > >> that > > > >> > > we > > > >> > > > > > could understand better the use case. > > > >> > > > > > > > > >> > > > > > [1] > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java > > > >> > > > > > [2] > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java > > > >> > > > > > [3] > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/tools/PlannerTest.java > > > >> > > > > > > > > >> > > > > > On Thu, May 2, 2019 at 7:56 AM Mark Pasterkamp < > > > >> > > > > > [email protected]> > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > I don't, I would assume that the > HepPlanner.findBestExp() > > > >> > > calculates > > > >> > > > > the > > > >> > > > > > > cost somewhere down the line > > > >> > > > > > > > > > >> > > > > > > On Thu, May 2, 2019, 03:31 Yuzhao Chen < > > > [email protected]> > > > >> > > wrote: > > > >> > > > > > > > > > >> > > > > > > > Why you care about cost when use HepPlanner ? The > > > >> HepPlanner is > > > >> > > > aimed > > > >> > > > > > for > > > >> > > > > > > > some deterministic planning rules, we usually do not > > need > > > >> cost > > > >> > in > > > >> > > > > Hep. > > > >> > > > > > > Some > > > >> > > > > > > > exceptions like Join reorder may need a cost. > > > >> > > > > > > > > > > >> > > > > > > > What kind of planning promotion you did ? I'm kind of > > > >> curious > > > >> > > about > > > >> > > > > it. > > > >> > > > > > > > > > > >> > > > > > > > Best, > > > >> > > > > > > > Danny Chan > > > >> > > > > > > > 在 2019年5月1日 +0800 PM9:27，Mark Pasterkamp < > > > >> > > > > [email protected] > > > >> > > > > > > > >，写道： > > > >> > > > > > > > > Dear all, > > > >> > > > > > > > > > > > >> > > > > > > > > While playing around with the HepPlanner I ran into > an > > > >> issue > > > >> > > > where > > > >> > > > > > the > > > >> > > > > > > > > planner wants to rewrite a query with a union > rewrite. > > > >> When > > > >> > the > > > >> > > > > > > > > RelMetaDataQuery computes the cost, the cost > instance > > > is a > > > >> > > > > > VolcanoCost. > > > >> > > > > > > > > Then when it tries to calculate the cost of one of > the > > > >> > union's > > > >> > > > > > operands > > > >> > > > > > > > it > > > >> > > > > > > > > is a RelCostImpl which results in the > > > ClassCastException. > > > >> > > > > > > > > > > > >> > > > > > > > > How would I go about solving this issue? As far as > my > > > >> > knowledge > > > >> > > > > > goes, I > > > >> > > > > > > > am > > > >> > > > > > > > > not able to change the costhandler of the > > > >> RelMetaDataQuery. > > > >> > > > Another > > > >> > > > > > > > > approach I could see is removing the cast in the > > > >> VolcanoCost > > > >> > > > class, > > > >> > > > > > > but I > > > >> > > > > > > > > would hope I do not have to do that. > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > With kind regards, > > > >> > > > > > > > > > > > >> > > > > > > > > Mark > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
