Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17765 )
Change subject: IMPALA-10840: Add support for "FOR SYSTEM_TIME AS OF" and "FOR SYSTEM_VERSION AS OF" for Iceberg tables ...................................................................... IMPALA-10840: Add support for "FOR SYSTEM_TIME AS OF" and "FOR SYSTEM_VERSION AS OF" for Iceberg tables This patch adds support "FOR SYSTEM_TIME AS OF" and "FOR SYSTEM_VERSION AS OF" clauses for Iceberg tables. The new clauses are part of the table ref. FOR SYSTEM_TIME AS OF conforms to the SQL2011 standard: https://cs.ulb.ac.be/public/_media/teaching/infoh415/tempfeaturessql2011.pdf With FOR SYSTEM_TIME AS OF we can query a table at a specific time point, e.g. we can retrieve what was the table content 1 day ago. The timestamp given to "FOR SYSTEM_TIME AS OF" is interpreted in the local timezone. The local timezone can be set via the query option TIMEZONE. By default the timezone being used is the coordinator node's local timezone. The timestamp is translated to UTC because table snapshots are tagged with a UTC timestamps. "FOR SYSTEM_VERSION AS OF" is a non-standard extension. It works similarly to FOR SYSTEM_TIME AS OF, but with this clause we can query a table via a snapshot ID instead of a timestamp. HIVE-25344 also added support for these clauses to Hive. Table snapshot IDs and timestamp information can be queried with the help of the DESCRIBE HISTORY command. Sample queries: SELECT * FROM t FOR SYSTEM_TIME AS OF now(); SELECT * FROM t FOR SYSTEM_TIME AS OF '2021-08-10 11:02:34'; SELECT * FROM t FOR SYSTEM_TIME AS OF now() - interval 10 days + interval 3 hours; SELECT * FROM t FOR SYSTEM_VERSION AS OF 7080861547601448759; SELECT * FROM t FOR SYSTEM_TIME AS OF now() MINUS SELECT * FROM t FOR SYSTEM_TIME AS OF now() - interval 1 days; This patch uses some parts of the in-progress IMPALA-9773 (https://gerrit.cloudera.org/#/c/13342/) developed by Todd Lipcon and Grant Henke. This patch also resolves some TODOs of IMPALA-9773, i.e. after this patch it'll be easier to add time travel for Kudu tables as well. Testing: * added parser tests (ParserTest.java) * added analyzer tests (AnalyzeStmtsTest.java) * added e2e tests (test_iceberg.py) Change-Id: Ib523c5e47b8d9c377bea39a82fe20249177cf824 Reviewed-on: http://gerrit.cloudera.org:8080/17765 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/common/init.cc M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java A fe/src/main/java/org/apache/impala/analysis/TimeTravelSpec.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java A fe/src/main/java/org/apache/impala/util/ExprUtil.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/main/java/org/apache/impala/util/KuduUtil.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M tests/query_test/test_iceberg.py 19 files changed, 576 insertions(+), 64 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib523c5e47b8d9c377bea39a82fe20249177cf824 Gerrit-Change-Number: 17765 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Amogh Margoor <[email protected]> Gerrit-Reviewer: Attila Jeges <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Qifan Chen <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
