Peter Rozsa has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/25039 )
Change subject: IMPALA-15493: Coordinator-local DDL execution for Iceberg REST catalogs ...................................................................... IMPALA-15493: Coordinator-local DDL execution for Iceberg REST catalogs DDL on a table loaded from an Iceberg REST catalog has always been built as a TDdlExecRequest and sent to CatalogD, which does not know the table in hybrid mode and does not exist in standalone mode. This adds the DDL counterpart of the coordinator-side DML finalization introduced in IMPALA-15310. TCatalogOpRequest carries an optional rest_catalog_name. When it is set, CatalogOpExecutor hands the DDL to the coordinator frontend through the new TIcebergDdlExecRequest JNI call instead of CatalogD, and ClientRequestState skips the catalog update wait, since nothing is cached on the coordinator for REST tables. The new IcebergDdlExecutor executes the operation through the named catalog without loading the table first, so tables with unreadable metadata can still be dropped. Like CatalogD, it never drops a blacklisted table: DROP TABLE IF EXISTS on one, which the analyzer hides, is a no-op. The frontend routes a DDL by the catalog that serves its target table, whether or not the table could be loaded or exists. The new CatalogRef names that catalog: CatalogD, or a REST catalog, addressed by its name when the name is unique. MetaProvider.getCatalogRef(db, table) answers without loading the table, and FeCatalog exposes it. A single provider serves all of its tables. MultiMetaProvider keeps merging databases across providers as introduced in IMPALA-14016, and records which provider serves each table when it lists the tables of a database, which analysis does before the DDL is routed. Tables of the primary provider are not recorded, as the primary provider serves every table that no secondary provider lists. A missing table is served by the first provider that serves its database, and a missing database by the primary provider: CatalogD when it is deployed, otherwise the first REST catalog. Either way, IF EXISTS on them is a no-op. IcebergMetaProvider now reports a failure to load a table, including the unchecked exceptions of the REST client, as TException. LocalDb then turns the table into a failed table that DROP TABLE can still analyze, as it already did for tables served through MultiMetaProvider. DROP TABLE [IF EXISTS] [PURGE] is the first operation carried by the new path: IcebergRESTCatalog.dropTable() is implemented and the analyzer lets DROP TABLE through for REST tables whose catalog has a unique name. DROP VIEW IF EXISTS routed to a REST catalog is a no-op, as REST catalogs serve no views. Other DDL is still rejected at analysis. Since the catalog name now routes DDL as well, the DML-specific names are generalized: getIcebergDmlCatalogName() becomes getCatalogName(), and FeCatalogManager looks a catalog up by that name through getCatalog() (for loading its tables) and getIcebergCatalog() (for DROP TABLE). Testing: - New IcebergDdlExecutorTest covers execution, blacklisted tables and the error paths without a live catalog. - New MultiMetaProviderTest covers which provider serves a table of a merged database, missing tables and databases, and failing providers. - New IcebergMetaProviderTest covers reporting REST failures to load a table as TException. - New AnalyzerCapabilityTest case; MetaProviderDecoratorTest covers the new MetaProvider method. - New custom cluster tests drop REST tables with and without CatalogD, across two REST catalogs, tables that fail to load with one and with two REST catalogs sharing a database name, blacklisted tables, and IF EXISTS on missing tables and databases. - Updated the REST catalog documentation. Change-Id: I932a211b25ea4e24a607d047c35da186d8922d02 Assisted-by: Claude Fable 5.1 (Claude Code) Assisted-by: Claude Opus 5.5 (Claude Code) Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --- M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M common/thrift/Frontend.thrift M docs/topics/impala_iceberg_rest_catalog.xml M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java A fe/src/main/java/org/apache/impala/catalog/CatalogRef.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProviderDecorator.java M fe/src/main/java/org/apache/impala/catalog/local/MultiMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java A fe/src/main/java/org/apache/impala/service/IcebergDdlExecutor.java M fe/src/main/java/org/apache/impala/service/IcebergDmlFinalizer.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/catalogmanager/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/catalogmanager/LocalImpl.java M fe/src/main/java/org/apache/impala/service/catalogmanager/TestImpl.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerCapabilityTest.java A fe/src/test/java/org/apache/impala/catalog/local/IcebergMetaProviderTest.java M fe/src/test/java/org/apache/impala/catalog/local/MetaProviderDecoratorTest.java A fe/src/test/java/org/apache/impala/catalog/local/MultiMetaProviderTest.java A fe/src/test/java/org/apache/impala/service/IcebergDdlExecutorTest.java M fe/src/test/java/org/apache/impala/service/IcebergRestDmlOperationTest.java M fe/src/test/java/org/apache/impala/service/catalogmanager/LocalImplTest.java M tests/custom_cluster/test_iceberg_rest_catalog.py 35 files changed, 1,327 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/25039/4 -- To view, visit http://gerrit.cloudera.org:8080/25039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I932a211b25ea4e24a607d047c35da186d8922d02 Gerrit-Change-Number: 25039 Gerrit-PatchSet: 4 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Aleksandr Efimov <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
