Peter Rozsa has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24838 )
Change subject: IMPALA-15145: Propagate credential metadata to Scan Nodes / Execution Plan ...................................................................... IMPALA-15145: Propagate credential metadata to Scan Nodes / Execution Plan Wires the credentials vended by an Iceberg REST catalog (fetched and translated to Hadoop config keys by IMPALA-15144) through to the S3 connections used during planning and execution. Backend: - HdfsTableDescriptor reads TIcebergTable.credentials into a list of CredentialEntry (prefix, fs.s3a.* config, expiry). - New per-query QueryCredentials, owned by QueryState: scan nodes register their table's credentials at fragment init; lookups are a longest-prefix match across the registered tables, where a prefix only covers paths below it (on a path-component boundary, so a table's credential is never applied to a sibling location whose name merely starts with the table name), and hand out shared ownership of immutable entries. An existing entry for a prefix is only replaced by one that expires later. - HdfsFsCache::GetConnection() takes an optional QueryState and resolves the credential for the path: vended entry first, then the process-global S3ConnCredentials, then core-site.xml defaults. For a vended entry it applies the entry's Hadoop config (key material and credential provider) to the connection it builds. Such connections are keyed by the credential's prefix plus a digest of its material, so every query reading a prefix with the same credential shares one connection, a rotated credential gets a new one, and no secret is embedded in a cache key. HdfsConnOptions becomes a plain std::map so options and credentials share one representation. - QueryState is threaded into every GetConnection() call on the query path: scan init, delete-vector/Puffin blob reads, table sinks, DML finalization and the bulk HDFS operations it issues. Spill-to-S3 keeps passing its upload-tuning options per call. Frontend: - New VendedCredentialsFileIO, a HadoopFileIO wrapper set as the catalog's io-impl when vending is enabled (an explicitly configured io-impl is left alone). Iceberg's RESTCatalog instantiates it per loaded table and hands it the table's credentials through SupportsStorageCredentials. Every open resolves the credential whose prefix covers the path and uses a FileSystem instance created with that credential's Hadoop config; Hadoop's FileSystem cache is keyed by bucket and user only, so these instances are created with FileSystem.newInstance() and cached process-wide by bucket and credential identity, and closed once their credential has expired. Manifest and stats reads therefore authenticate with the vended keys without any change to Iceberg's planning code, and IcebergFileMetadataLoader lists files through the same instances. Paths not covered by a credential fall through to HadoopFileIO. - Server-returned credentials are ignored when iceberg.rest-catalog.vended-credentials-enabled is false; some catalogs (Lakekeeper) vend them regardless of the access-delegation header. Tests: - tmp-file-mgr-test seeds its fake S3 connection under the same key GetConnection() now computes. - test_iceberg_rest_catalog gains a run against the in-tree HDFS-backed REST server with vended-credentials-enabled=true; that server vends nothing, so this verifies the plumbing is a no-op for the common case. Change-Id: I73592a1521f29374316ed001341f956b40f9c0d5 Assisted-by: Claude Fable 5.1 <[email protected]> --- M be/src/exec/blob-reader.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/iceberg-delete-builder.cc M be/src/exec/puffin/puffin-writer.cc M be/src/exec/table-sink-base.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/coordinator.cc M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/hdfs-fs-cache.cc M be/src/runtime/hdfs-fs-cache.h A be/src/runtime/query-credentials.cc A be/src/runtime/query-credentials.h M be/src/runtime/query-state.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/hdfs-bulk-ops.cc M be/src/util/hdfs-bulk-ops.h M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergRESTCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/RESTCatalogProperties.java A fe/src/main/java/org/apache/impala/catalog/iceberg/VendedCredentialsFileIO.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/test/java/org/apache/impala/catalog/iceberg/TestRESTCatalogProperties.java A testdata/configs/catalog_configs/iceberg_rest_vended_config/rest.properties M tests/custom_cluster/test_iceberg_rest_catalog.py 29 files changed, 865 insertions(+), 78 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/24838/4 -- To view, visit http://gerrit.cloudera.org:8080/24838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73592a1521f29374316ed001341f956b40f9c0d5 Gerrit-Change-Number: 24838 Gerrit-PatchSet: 4 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
