Steve Carlin has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/21575 )
Change subject: IMPALA-13221: Calcite: Enable tpcds and tpch queries ...................................................................... IMPALA-13221: Calcite: Enable tpcds and tpch queries As a minor milestone, the tpcds and tpch queries now work which shows that a use case for queries is enabled for the Calcite framework. The two test case files were created rather than reused. These test case files are temporary. In a future commit, the test framework files will be able to run through both the current framework and the Calcite framework. The changes that had to be made to support these queries are: - Added the word "value" as a non-reserved key word since it is used as a column alias in tpch - Added support for datetime interval operators. A dummy IntervalExpr was added to support logical to physical representation. The IntervalExpr is removed once the actual "+" operation for datetime is created, but the translation framework is simplified by working on the Calcite interval type in a separate temporary Expr. - Added new maps from Calcite SqlKind description of functions to Impala functions. Added a map for Calcite function names to Impala function names. - Hacked around an issue with casting in the analysis framework. When the "analyze" is done, it does not recursive enter "implicit" casts. But since all casts created by Calcite are implicit, the analysis has to be done before the cast is created. - Added functions that adjust the scale. Impala has its own way of deriving the return type so the new operator ImpalaAdjustScaleFunction was created to call into Impala's code. - Added a ImpalaGroupingFunction operator since the return type of the "grouping" function differs in Calcite. - Created bypass for some extract time operators to use ImpalaOperator directly instead of Calcite operators because the validation step failed in Calcite. - Change HdfsTable to more generic FeFsTable - Created an ImpalaBaseTableRef class to deal with multiple table references to the same name in a query. Impala deals with this through the Analyzer, but the code for Calcite has to ensure that there is a unique table alias for every TableRef. - Added conjunctive normal form rules because performance was too slow without them when running some tpcds queries. - Added ExtractLiteralAgg rule. Calcite creates a LITERAL_AGG operator which puts a literal inside of an aggregate. Impala doesn't support this, so a rule is created to change this in a way that is supported in Impala. - Added the MinusToDistinct rules. These ruleis exist in Calcite and changes a Minus RelNode into supported RelNodes. However, one rule within Calcite did not exactly match what is needed for Impala. The rule was copied over into Impala and changed to suit Impala's needs which is documented in the code. - Fixed a bug on how comments were being parsed. - Added decorrelator step when doing the SqlNode to RelNode conversion. Change-Id: I3107d336ac07ecd89530b640165798ec6a574f41 --- M java/calcite-planner/pom.xml M java/calcite-planner/src/main/codegen/config.fmpp M java/calcite-planner/src/main/codegen/templates/Parser.jj M java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/IntervalExpr.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaAdjustScaleFunction.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCustomOperatorTable.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaGroupingFunction.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAnalyticRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/ImpalaBaseTableRef.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ConvertToCNFRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ExtractLiteralAgg.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaMinusToDistinctRule.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteJniFrontend.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/util/SimplifiedAnalyzer.java A tests/custom_cluster/test_calcite_tpcds.py A tests/custom_cluster/test_calcite_tpch.py 28 files changed, 1,399 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/21575/7 -- To view, visit http://gerrit.cloudera.org:8080/21575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3107d336ac07ecd89530b640165798ec6a574f41 Gerrit-Change-Number: 21575 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin <[email protected]>
