Steve Carlin has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24577
Change subject: IMPALA-15178: Calcite Planner: support Iceberg time travel
feature.
......................................................................
IMPALA-15178: Calcite Planner: support Iceberg time travel feature.
This commit supports the Iceberg time travel feature for queries.
Iceberg uses either the "FOR SYSTEM_TIME AS OF" or "FOR
SYSTEM_VERSION AS OF" sql phrase to retrieve data (with its proper
schema) at a given time.
Calcite does provide some code to support snapshots with its
SqlSnapshotNode (validation time) and LogicalSnapshot (optimization
time) objects. However, the Calcite implementation did not quite
fit the architecture used by Impala.
The SqlSnapshot object has a timestamp type "Period" and directly
creates a LogicalSnapshot RelNode on top of a LogicalTableScan. The
reason this didn't work for Impala is because:
- Iceberg works off a long value of snapshotId. While a timestamp
can be provided at query time, sometimes it is the snapshotId
itself that is provided through the SYSTEM_VERSION feature.
- Impala creates a pseudo-catalog table at analysis time called
ImpalaTimeTravelTable. This table holds the schema at the given
time instance for the table. The problem for Calcite is that this
schema needs to be known at validation time to handle validation
steps like expanding a select star operation. When Calcite calls
back into Impala, it needs a unique table name to fetch this.
Some creative engineering was needed to handled these issues. The
architecture here is most likely safe from Calcite upgrades.
Highlights of these changes include:
- In the Parser.jj file, the Impala TimeTravelSpec object is created.
A JavaCCParserUtils utility class helps grab the special token after
the "AS OF" keyword. The Impala CUP parser has the code which parses
the time travel SQL. So some code was added to the CUP parser which
allows the object to be created, located in the ".cup" file and the
Parser java file.
- The SqlSnapshot Calcite class was extended (ImpalaSnapshotSqlNode
to hold onto the Impala specific TimeTravelSpec. The Period variable
is ignored. The tableRef name contains the table name concatenated
with "_tt_<TimeTravelSpec hash code>" to generate a unique table name
that will be found in the CalciteCatalogReader in the validation
stage.
- The CalciteMetadataHandler.TableVisitor is still called after the
parsing and before the validation. It walks through the SqlNode tree.
It still tracks all the tables as before. When it encounters the
ImpalaSnapshotSqlNode, it keeps a map of a table with a list of
TimeTravelSpecs which need a catalog table object. For normal tables
without a TimeTravelSpec, the list will contain a "null" for the
TimeTravelSpec. Duplicates are ok here as will be discussed in the
next bullet point.
- While there may be duplicates, the TableName objects duplicates
are removed via the TableVisitor.getTableNames() by the caller of
the visitor. These unique table names eventually get into the
StatementMetadataLoader class.
- The CalciteMetadataHandler.populateCalciteSchema is a callback
from the Impala framework called by the StatementMetadataLoader
before the Calcite validation step. In this phase, the tables get
loaded from catalogd. At this point, the Calcite schema gets loaded
with all the catalogd objects. For time travel objects, as mentioned
earlier, an IcebergTimeTravelTable object gets created. There may
be some TimeTravel tables that are duplicates, but it is not worth
the coding headache to make them unique since they only exist at
analysis time. The unique time travel name uses the same TimeTravelSpec
hash code in the ImpalaSnapshotSqlNode object which is how Calcite
can fetch the correct schema.
- At Logical node creation time, a LogicalSnapshot node is created.
This is not needed by Impala since the LogicalTableScan object will
contain the CalciteIcebergTable which contains the
IcebergTimeTravelTable, so it is removed from the logical RelNode
tree via the RemoveSnapshotRule.
- The existing ScanNodeHelper interface has a new getTimeTravelSpec
method which allows the IcebergScanPlanner to fetch the
TimeTravelSpec.
Testing: This was run against the time_travel e2e test in
test_iceberg.py and manually checked to ensure that there was never
any fallback. In a later commit, there will be tests to ensure that
fallback does not happen.
Change-Id: I6ab3466d6c453f8e030763749dd64903b8c41264
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/Parser.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTimeTravelTable.java
M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java
M fe/src/main/java/org/apache/impala/planner/ScanNodeHelper.java
M fe/src/main/java/org/apache/impala/planner/ScanNodeHelperImpl.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M java/calcite-planner/src/main/codegen/templates/Parser.jj
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaHdfsScanRel.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/RemoveSnapshotRule.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteDb.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteIcebergTable.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/CalciteTable.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteAnalysisDriver.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteParsedStatement.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/util/JavaCCParserUtils.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/validate/ImpalaSnapshotSqlNode.java
19 files changed, 471 insertions(+), 61 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/24577/7
--
To view, visit http://gerrit.cloudera.org:8080/24577
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6ab3466d6c453f8e030763749dd64903b8c41264
Gerrit-Change-Number: 24577
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <[email protected]>
