Hello Aman Sinha, Quanlong Huang, Joe McDonnell, Csaba Ringhofer, Michael
Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21109
to look at the new patch set (#25).
Change subject: IMPALA-12872: Use Calcite for optimization - part 1: simple
queries
......................................................................
IMPALA-12872: Use Calcite for optimization - part 1: simple queries
This is the first commit to use the Calcite library to parse,
analyze, and optimize queries.
The hook for the planner is through an override of the JniFrontend. The
CalciteJniFrontend class is the driver that walks through each of the
Calcite steps which are as follows:
CalciteQueryParser: Takes the string query and outputs an AST in the
form of Calcite's SqlNode object.
CalciteMetadataHandler: Iterate through the SqlNode from the previous step
and make sure all essential table metadata is retrieved from catalogd.
CalciteValidator: Validate the SqlNode tree, akin to the Impala Analyzer.
CalciteRelNodeConverter: Change the AST into a logical plan. In this first
commit, the only logical nodes used are LogicalTableScan and LogicalProject.
The LogicalTableScan will serve as the node that reads from an Hdfs Table and
the LogicalProject will only project out the used columns in the query. In
later versions, the LogicalProject will also handle function changes.
CalciteOptimizer: This step is to optimize the query. In this cut, it will be
a nop, but in later versions, it will perform logical optimizations via
Calcite's rule mechanism.
CalcitePhysPlanCreator: Converts the Calcite RelNode logical tree into
Impala's PlanNode physical tree
ExecRequestCreator: Implement the existing Impala steps that turn a Single
Node Plan into a Distributed Plan. It will also create the TExecRequest object
needed by the runtime server.
Only some very basic queries will work with this commit. These include:
select * from tbl <-- only needs the LogicalTableScan
select c1 from tbl <-- Also uses the LogicalProject
In the CalciteJniFrontend, there is some basic checks to make sure only
select statements will get processed. Any non-query statement will revert
back to the current Impala planner.
In this iteration, any queries besides the minimal ones listed above will
result in a caught exception which will then be run through the current
Impala planner. The tests that do work can be found in calcite.test and
run through the custom cluster test test_experimental_planner.py
This iteration should support all types with the exception of complex
types. Calcite does not have a STRING type, so the string type is
represented as VARCHAR(MAXINT) similar to how Hive represents their
STRING type.
The ImpalaTypeConverter file is used to convert the Impala Type object
to corresponding Calcite objects.
Authorization is not yet working with this current commit. A Jira has been
filed (IMPALA-13011) to deal with this.
Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98
---
M bin/set-classpath.sh
M bin/start-impala-cluster.py
M fe/src/main/java/org/apache/impala/analysis/TableName.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A java/calcite-planner/pom.xml
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaProjectRel.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/NodeWithExprs.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/phys/ImpalaHdfsScanNode.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaCalciteCatalogReader.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CompilerStep.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaConformance.java
M java/pom.xml
A testdata/workloads/functional-query/queries/QueryTest/calcite.test
A tests/custom_cluster/test_calcite_planner.py
33 files changed, 3,102 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/21109/25
--
To view, visit http://gerrit.cloudera.org:8080/21109
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I453fd75b7b705f4d7de1ed73c3e24cafad0b8c98
Gerrit-Change-Number: 21109
Gerrit-PatchSet: 25
Gerrit-Owner: Steve Carlin <scar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com>
