wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/15797 )
Change subject: IMPALA-9688: Support create iceberg table by impala ...................................................................... Patch Set 18: (5 comments) Hi Zoltan, I'm preparing to implement the function of query iceberg table by impala recently, and I will update here if there is any progess: IMPALA-9741, including my design or solution. http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java File fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java: http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java@569 PS17, Line 569: a managed Icebe > "a managed Iceberg" Done http://gerrit.cloudera.org:8080/#/c/15797/11/fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java File fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java: http://gerrit.cloudera.org:8080/#/c/15797/11/fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java@236 PS11, Line 236: return true; > Do you have a design doc about it? E.g. a google doc that you could share w Thanks for your patient review again, Zoltan. I didn't have a design doc about query iceberg table by impala now. But I am considering how to implement this recently, I will discussed with my colleague who research the iceberg. And I've already create a jira: IMPALA-9741, I will update here if there is progress. http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java File fe/src/main/java/org/apache/impala/catalog/IcebergTable.java: http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/catalog/IcebergTable.java@64 PS17, Line 64: icebergTableLocat > Shouldn't this be renamed to icebergTableLocation_ here and in the thrift s Done, yes, you are right, I will use table location to replace table name here, including thrift file to identify iceberg file system table. http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/util/IcebergUtil.java File fe/src/main/java/org/apache/impala/util/IcebergUtil.java: http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@24 PS17, Line 24: import org.apache.iceberg.hadoop.HadoopTableOperations; : import org.apache.iceberg.hadoop.HadoopTables; > So initially it'll only support "Files System Tables", and not "Metastore t Iceberg indeed support HiveCatalog and HadoopTables to create 'Metastore tables' and 'Files System Tables' as you mentioned above, and you can find a quick start here: https://iceberg.apache.org/api-quickstart/#using-a-hive-catalog. I use HadoopTables in this patch to implement this function: impala create hms table and iceberg create file system table, just like impala supporting kudu. Thus impala can manage hms table and iceberg manage file system table. I think HiveCatalog is not very suitable for impala due to HiveCatalog creating both table in hms and hdfs. We cannot use this in external iceberg table situation: if we use HiveCatalog to create a metastore table, we cannot create external table by impala, because there is already a table in hms created by iceberg. But if we use HadoopTables to create a file system table, we can use create external table by impala through a table location. http://gerrit.cloudera.org:8080/#/c/15797/17/fe/src/main/java/org/apache/impala/util/IcebergUtil.java@66 PS17, Line 66: tableLoca > nit: tableLocation? Done, I already modify this as tableLocation to identity the iceberg file system table. -- To view, visit http://gerrit.cloudera.org:8080/15797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8d85db4c904a8c758c4cfb4f19cfbdab7e6ea284 Gerrit-Change-Number: 15797 Gerrit-PatchSet: 18 Gerrit-Owner: wangsheng <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Sahil Takiar <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: wangsheng <[email protected]> Gerrit-Comment-Date: Tue, 12 May 2020 03:16:08 +0000 Gerrit-HasComments: Yes
