Hello Daniel Becker, Zoltan Borok-Nagy, Peter Rozsa, Noemi Pap-Takacs, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24682
to look at the new patch set (#9).
Change subject: IMPALA-15310: Finalize REST catalog INSERT
......................................................................
IMPALA-15310: Finalize REST catalog INSERT
Enable INSERT INTO for Iceberg tables loaded from named REST catalogs.
Carry the opaque catalog name through DML finalization and reload the
target from the same provider. This keeps routing correct with CatalogD
or multiple REST catalogs without exposing endpoints or credentials.
Reuse the shared Iceberg finalizer and planning-time table UUID.
Existing HMS/CatalogD tables retain their current finalization route.
REST catalog writes remain limited to INSERT INTO.
The finalizer is identified by the catalog's configured name. It only
has to be unique within one coordinator, since the same process plans
the query and finalizes the write, so an ordinal would serve equally
well. The name is what a reader sees in the log line and in the error
a user gets, which an ordinal would not be. A name shared by two
catalogs, or left unset, therefore disables INSERT INTO for the
catalogs involved rather than picking one of them.
The write checks ask the provider what the table supports rather than
testing what kind of catalog it came from. A provider that loads a
table without HMS behind it has to report the access type as READ, so
hasProviderDerivedCapabilities() says the access type is not the thing
to ask, and the next provider in that position needs no second test.
MERGE, UPDATE, DELETE and OPTIMIZE get the refusal from a call of their
own rather than from checkTableCapability(), which they do not use at
all today. Routing them through it is IMPALA-15320: it would start
running ensureTableWriteSupported() on statements that have never run
it, which is a change to make on its own.
Document the new property and drop the blanket "write operations are
not supported" from the REST catalog docs.
Testing:
- Java, generated Thrift, impalad, catalogd and statestored builds
- Analyzer, provider routing, UUID and cleanup tests
- Standalone, mixed and multi-provider REST INSERT
- Known and unknown commit failure cleanup
- Fixture ownership, qualified paths and shutdown cleanup
Change-Id: I7d17216fb8d12e5bd007b93cb9b3bc153438213b
Assisted-by: gpt-5.6-sol (OpenAI Codex), Claude Opus 5 (Anthropic)
---
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M common/thrift/CatalogService.thrift
M common/thrift/Query.thrift
M docs/topics/impala_iceberg_rest_catalog.xml
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/MergeStmt.java
M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java
M fe/src/main/java/org/apache/impala/analysis/OptimizeStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M
fe/src/main/java/org/apache/impala/service/catalogmanager/FeCatalogManager.java
M fe/src/main/java/org/apache/impala/service/catalogmanager/LocalImpl.java
M fe/src/main/java/org/apache/impala/service/catalogmanager/TestImpl.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
A fe/src/test/java/org/apache/impala/analysis/AnalyzerCapabilityTest.java
A fe/src/test/java/org/apache/impala/service/IcebergRestDmlOperationTest.java
A fe/src/test/java/org/apache/impala/service/catalogmanager/LocalImplTest.java
M fe/src/test/java/org/apache/impala/util/DebugUtilsTest.java
M
java/iceberg-rest-catalog-test/src/main/java/org/apache/iceberg/rest/IcebergRestCatalogTest.java
M testdata/bin/run-iceberg-rest-server.sh
M testdata/configs/catalog_configs/iceberg_rest_config/rest.properties
M testdata/configs/catalog_configs/multicatalog_rest_config/rest-1.properties
M testdata/configs/catalog_configs/multicatalog_rest_config/rest-2.properties
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-multicatalog.test
M
testdata/workloads/functional-query/queries/QueryTest/iceberg-multiple-rest-catalogs.test
M tests/common/iceberg_rest_server.py
M tests/custom_cluster/test_iceberg_rest_catalog.py
35 files changed, 882 insertions(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/24682/9
--
To view, visit http://gerrit.cloudera.org:8080/24682
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d17216fb8d12e5bd007b93cb9b3bc153438213b
Gerrit-Change-Number: 24682
Gerrit-PatchSet: 9
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Daniel Becker <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
