Peter Rozsa created IMPALA-15501:
------------------------------------
Summary: Support CTAS on Iceberg REST catalogs
Key: IMPALA-15501
URL: https://issues.apache.org/jira/browse/IMPALA-15501
Project: IMPALA
Issue Type: Sub-task
Components: Backend, Catalog, Frontend
Reporter: Peter Rozsa
CREATE TABLE AS SELECT into a database served by an Iceberg REST catalog cannot
be analyzed or executed. Besides sharing every problem of plain CREATE TABLE
(IMPALA-15499), CTAS has its own blockers in {{IcebergCtasTarget}}:
* {{IcebergCtasTarget.setLocations()}}
({{fe/.../catalog/iceberg/IcebergCtasTarget.java:152-178}})
asserts via {{Preconditions.checkState}} that the catalog is one of
{{HADOOP_TABLES}},
{{HIVE_CATALOG}} or {{CATALOGS}}. A REST catalog has no {{TIcebergCatalog}}
value, so
{{IcebergUtil.getTIcebergCatalog(msTbl)}} (called at {{:100}}) falls through
to
{{CATALOGS}} and the table location is taken from
{{msTable_.getSd().getLocation()}},
which the analyzer filled from
{{MetastoreShim.getPathForNewTable(db.getMetaStoreDb(),
msTbl)}} ({{CreateTableAsSelectStmt.java:233}}).
{{IcebergMetaProvider.loadDb()}}
({{IcebergMetaProvider.java:155-159}}) returns a {{Database}} with only the
name set,
so there is no location to derive from.
* {{IcebergCtasTarget.getIcebergApiTable()}} returns {{null}} ({{:277-278}}) and
{{transformToTHdfsTable(boolean, ...)}} throws
{{IllegalStateException("not implemented here")}} ({{:260-262}}). Nothing on
the REST
path can obtain an Iceberg {{Transaction}} for a table that does not exist
yet.
* {{createFsTable()}} ({{:180-187}}) dispatches on {{instanceof Db}} /
{{LocalDb}}; for
the REST provider the db is a {{LocalDb}}, which is fine, but it must not try
to load
HMS partition metadata.
* CTAS is two-phase in the backend: {{client-request-state.cc:886-892}} waits
for
{{ddl_exec_response()->new_table_created}} before running the INSERT, and
{{:2015}}
drops the table again if the INSERT fails. Both phases go through
{{CatalogOpExecutor}} and die at {{catalog-op-executor.cc:111-113}}
("Operation is not supported without CatalogD.") in standalone mode, or are
sent to a
catalogd that does not know the REST table in hybrid mode.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
