[
https://issues.apache.org/jira/browse/IMPALA-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195455#comment-17195455
]
WangSheng commented on IMPALA-10164:
------------------------------------
Hi [~boroknagyz], I've do some work for this jira, and it's not very difficult
to implement this function. But HadoopCatalog is quite different from
HadoopTables:
# We just need Configuration to construct HadoopTables, but HadoopCatalog need
another param location, such as hdfs://xxx/warehouse/, and this location used
to reserve table. When using HadoopCatalog, we need to provide TableIdentifier
which mainly contains database and table, then Iceberg will create table use
location 'hdfs://xxx/warehouse/database/table' to storage table info;
# When create external table, we cannot use
'hdfs://xxx/warehouse/database/table' to loading table directly, we need use
TableIdentifier.of(database, table) and 'hdfs://xxx/warehouse/' instead.
So here is the problem: when creating external table with HadoopCatalog, how to
define the location?
* If we use 'hdfs://xxx/warehouse' in sql, we can simply use this location and
TableIdentifier.of(database, table) to loading table, but this usage is
different from HdfsTable, a little wired;
* If we use 'hdfs://xxx/warehouse/database/table' in sql, we need to extract
'hdfs://xxx/warehouse', 'database', 'table' from this location, and compare
with database, table with from 'create external table xxx', if same, we can
loading table, otherwise maybe throw exception.
How do you think?Here is my simple patch, I use first method to just verify, in
this patch, we need to create table like this:
{code:java}
create external database.table
stored as ICEBERG
location 'hdfs://test-warehouse'{code}
Here is the Gerrit url: https://gerrit.cloudera.org/#/c/16446/
> Support HadoopCatalog for Iceberg table
> ---------------------------------------
>
> Key: IMPALA-10164
> URL: https://issues.apache.org/jira/browse/IMPALA-10164
> Project: IMPALA
> Issue Type: Improvement
> Reporter: WangSheng
> Assignee: WangSheng
> Priority: Minor
> Labels: impala-iceberg
>
> We just supported HadoopTable api to create Iceberg table in Impala now, it's
> apparently not enough, so we preparing to support HadoopCatalog. The main
> design is to add a new table property named 'iceberg.catalog', and default
> value is 'hadoop.tables', we implement 'hadoop.catalog' to supported
> HadoopCatalog api. We may even support 'hive.catalog' in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
