Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19145 )
Change subject: IMPALA-11339: Add Iceberg LOAD DATA INPATH statement ...................................................................... IMPALA-11339: Add Iceberg LOAD DATA INPATH statement Extend LOAD DATA INPATH statement to support Iceberg tables. Native parquet tables need Iceberg field ids, therefore to add files this change uses child queries to load and rewrite the data. The child queries create > insert > drop the temporary table over the specified directory. The create part depends on LIKE PARQUET/ORC clauses to infer the file format. This requires identifying a file in the directory and using that to create the temporary table. The target file or directory is moved to a staging directory before ingestion similar to native file formats. In case of a query failure the files are moved back to the original location. Child query executor will return the error message of the failing query and the child query profiles will be available through the WebUI. At this point the PARTITION clause it not supported because it would require analysis of the PartitionSpec (IMPALA-11750). Testing: - Added e2e tests - Added fe unit tests Change-Id: I8499945fa57ea0499f65b455976141dcd6d789eb Reviewed-on: http://gerrit.cloudera.org:8080/19145 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java A fe/src/main/java/org/apache/impala/analysis/QueryStringBuilder.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-load.test M tests/query_test/test_iceberg.py 10 files changed, 607 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/19145 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8499945fa57ea0499f65b455976141dcd6d789eb Gerrit-Change-Number: 19145 Gerrit-PatchSet: 13 Gerrit-Owner: Tamas Mate <[email protected]> Gerrit-Reviewer: Anonymous Coward <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
