So, the question is whether materialized views need to be “registered” with the planner before they can be considered. If they are “registered” with a Volcano planner this means that they are included in equivalence classes (RelSets and RelSubsets) and canonized.
A weaker form of registration is to make sure that the types used (in both the row-type of a RelNode and in the various RexNodes contained therein) all come from the same type factory. Clearly there are advantages to registration (if objects are canonized they use less memory and can be compared using ==) and there are negatives (significant copying is involved). So, the question is whether we can use some kind of compromise: work on un-registered RelNodes at an early stage (while figuring out which materialized views might pertain to a query) and register only when we have narrowed down the set of materialized views. Maryann, Since you did https://issues.apache.org/jira/browse/CALCITE-1500 <https://issues.apache.org/jira/browse/CALCITE-1500>, can you comment on this change? Julian > On Mar 9, 2017, at 1:09 PM, Remus Rusanu <rrus...@hortonworks.com> wrote: > > Moving to calcite-dev > > From: Remus Rusanu <rrus...@hortonworks.com> > Date: Thursday, March 9, 2017 at 1:04 PM > To: Ashutosh Chauhan <ashut...@hortonworks.com>, Julian Hyde > <jh...@hortonworks.com> > Cc: "sql...@hortonworks.com" <sql...@hortonworks.com> > Subject: Why Calcite 1.10 is not hitting the assert > > The 1.12 relevant assert stack is this: > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:368) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:592) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1467) > > In 1.10 the names are a bit different, but VolcanoPlanner.findBestExp() calls > useApplicableMaterializations() which exists immediately, because > context.unwrap(CalciteConnectionConfig.class) returns null. So no > ‘registration’ occurs (registerImpl is never called with the provided > materialization plan, as per my debugging). > > However, when needed, the materialization is found. This stack bellow finds > it, and uses it, despite not being ‘registered’: > at > org.apache.calcite.plan.volcano.VolcanoPlanner.getMaterializations(VolcanoPlanner.java:348) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewFilterScanRule.apply(HiveMaterializedViewFilterScanRule.java:71) > at > org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewFilterScanRule.onMatch(HiveMaterializedViewFilterScanRule.java:64) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:213) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:819) > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1455) > > The result is the desired one: > > hive> create materialized view srcm enable rewrite as select key from src > where key=10; > … > hive> explain extended select key from src where key=10; > OK > STAGE DEPENDENCIES: > Stage-0 is a root stage > > STAGE PLANS: > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > TableScan > alias: default.srcm > GatherStats: false > Select Operator > expressions: key (type: string) > outputColumnNames: _col0 > ListSink > > The big changes came with CALCITE-1500 >
