Hello Aman Sinha, Anonymous Coward (816), Joe McDonnell, Michael Smith, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21575
to look at the new patch set (#13).
Change subject: IMPALA-13221: Calcite: Enable tpcds and tpch queries
......................................................................
IMPALA-13221: Calcite: Enable tpcds and tpch queries
As a minor milestone, the tpcds and tpch queries now work
which shows that a use case for queries is enabled for
the Calcite framework.
The two test case files were created rather than reused. These
test case files are temporary. In a future commit, the
test framework files will be able to run through both the
current framework and the Calcite framework.
The changes that had to be made to support these queries are:
- Added the word "value" as a non-reserved key word since it
is used as a column alias in tpch
- Added support for datetime interval operators. A dummy IntervalExpr
was added to support logical to physical representation. The
IntervalExpr is removed once the actual "+" operation for datetime
is created, but the translation framework is simplified by working
on the Calcite interval type in a separate temporary Expr.
- Added new maps from Calcite SqlKind description of functions to
Impala functions. Added a map for Calcite function names to Impala
function names.
- Hacked around an issue with casting in the analysis framework. When
the "analyze" is done, it does not recursive enter "implicit" casts.
But since all casts created by Calcite are implicit, the analysis
has to be done before the cast is created.
- Added functions that adjust the scale. Impala has its own way
of deriving the return type so the new operator
ImpalaAdjustScaleFunction was created to call into Impala's code.
- Added a ImpalaGroupingFunction operator since the return type of
the "grouping" function differs in Calcite.
- Created bypass for some extract time operators to use
ImpalaOperator directly instead of Calcite operators because the
validation step failed in Calcite.
- Change HdfsTable to more generic FeFsTable
- Created an ImpalaBaseTableRef class to deal with multiple table
references to the same name in a query. Impala deals with this
through the Analyzer, but the code for Calcite has to ensure that
there is a unique table alias for every TableRef.
- Added conjunctive normal form rules because performance was too
slow without them when running some tpcds queries.
- Added ExtractLiteralAgg rule. Calcite creates a LITERAL_AGG
operator which puts a literal inside of an aggregate. Impala doesn't
support this, so a rule is created to change this in a way that
is supported in Impala.
- Added the MinusToDistinct rules. This rule exists in Calcite and
changes a Minus RelNode into supported RelNodes. However, one rule
within Calcite did not exactly match what is needed for Impala. The
rule was copied over into Impala and changed to suit Impala's needs
which is documented in the code.
- Fixed a bug on how comments were being parsed.
- Added decorrelator step when doing the SqlNode to RelNode conversion.
Change-Id: I3107d336ac07ecd89530b640165798ec6a574f41
---
M java/calcite-planner/pom.xml
M java/calcite-planner/src/main/codegen/config.fmpp
M java/calcite-planner/src/main/codegen/templates/Parser.jj
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/IntervalExpr.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaAdjustScaleFunction.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCustomOperatorTable.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaGroupingFunction.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAnalyticRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/ImpalaBaseTableRef.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ConvertToCNFRules.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ExtractLiteralAgg.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaMinusToDistinctRule.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteQueryParser.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/util/SimplifiedAnalyzer.java
A tests/custom_cluster/test_calcite_tpcds.py
A tests/custom_cluster/test_calcite_tpch.py
28 files changed, 1,344 insertions(+), 77 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/21575/13
--
To view, visit http://gerrit.cloudera.org:8080/21575
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3107d336ac07ecd89530b640165798ec6a574f41
Gerrit-Change-Number: 21575
Gerrit-PatchSet: 13
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Anonymous Coward (816)
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
