Steve Carlin has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/22870 )
Change subject: IMPALA-14061: Calcite Planner: added Calcite rules ...................................................................... IMPALA-14061: Calcite Planner: added Calcite rules This commit adds Calcite optimization rules to create more efficient plans. These rules should be considered a work in progress. These were tested against a 3TB tpcds database so they are fairly efficient as/is, but we can make improvements as we see them along the way. Most of the changes have been added to the CalciteOptimizer file. There are several phases of rules that are applied, which are as follows: - expand nodes: These rules change the plan to a plan that can be handled by Impala. For instance, there are RelNodes such as "LogicalIntersect" which are not directly applicable to the Impala physical nodes so they need to be expanded. - coerce nodes: This module changes the nodes so they have the correct datatype values (e.g. literal strings in Calcite are char but need to be varchar for Impala) - optimize nodes: first pass on reordering the logical RelNode ordering. - join: Squishes the join RelNodes together, pushes them into one "multiJoin" and then lets Calcite's join optimizer reorder the joins into a more optimal plan. A note on this: with this iteration, statistics are still not being applied. This will come in with later commits to make better plans. - post join optimize nodes: Reruns the optimize nodes since the join ordering may present new optimization opportunities - pre Impala commit: Extra massaging after optimization that is done at the end - conversion to Impala RelNodes: Maps Calcite RelNodes into Impala RelNodes which will then be mapped to Impala PlanNodes In addition to this general change, there is also a change with removing the "toCNF" rule. Calcite has multiple places where it creates a SEARCH operator via "simplifying" the RexNodes within various rules. This operator is not supported directly in Impala and we need to call "expandSearch" to handle this. Because Impala does this under the covers in the rules, this has been fixed by overriding the RexBuilder (with ImpalaRexBuilder) and expanding the SEARCH operator whenever it is called (sidenote: we could have changed the rules that called simplify, but that would have resulted in too much code duplication). The toCNF rule was removed and placed as a call within the CoerceOperandShuttle, which already manipulates all the RexNodes, so all that code is now in one place. Change-Id: I6671f7ed298a18965ef0b7a5fc10f4912333a52b --- M java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceNodes.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaRexBuilder.java D java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ConvertToCNFRules.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteSingleNodePlanner.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/util/LogUtil.java M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q01.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q02.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q03.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q04.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q05.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q06.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q07.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q08.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q09.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q10a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q11.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q12.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q13.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14b.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q15.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q16.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q17.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q18.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q19.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q20.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q21.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q22.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23b.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24b.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q25.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q26.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q27.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q29.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q30.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q31.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q32.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q33.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q34.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q35a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q36.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q37.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q38.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39a.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39b.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q40.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q41.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q42.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43-verbose.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q44.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q45.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q46.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q47.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q48.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q49.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q50.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q51.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q52.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q53.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q54.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q55.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q56.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q57.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q58.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q59.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q60.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q61.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q62.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q63.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q64.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q65.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q66.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q67.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q68.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q69.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q70.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q71.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q72.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q73.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q74.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q75.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q76.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q77.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q78.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q79.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q80.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q81.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q82.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q83.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q84.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q85.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q86.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q87.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q88.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q89.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q90.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q91.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q92.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q93.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q94.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q95.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q96.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q97.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q98.test M testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q99.test 111 files changed, 48,658 insertions(+), 55,571 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/22870/8 -- To view, visit http://gerrit.cloudera.org:8080/22870 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6671f7ed298a18965ef0b7a5fc10f4912333a52b Gerrit-Change-Number: 22870 Gerrit-PatchSet: 8 Gerrit-Owner: Steve Carlin <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Anonymous Coward (816) Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]>
