Peter Rozsa has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24839 )
Change subject: IMPALA-15146: Extend Impala Minicluster with an S3-compatible object store for testing vended credentials ...................................................................... IMPALA-15146: Extend Impala Minicluster with an S3-compatible object store for testing vended credentials Adds a Docker Compose stack under testdata/bin/minicluster_lakekeeper_s3 that gives the minicluster an S3-compatible object store and a REST catalog that vends per-table credentials, so credential vending can be tested end to end without a cloud account: - RustFS as the S3 endpoint. It is Apache-2.0 licensed and MinIO-compatible, and its STS AssumeRole endpoint is what lets Lakekeeper vend scoped, expiring session credentials. RustFS enforces the session policy, so a credential vended for one table's prefix is rejected for another table in the same bucket. - Lakekeeper as the Iceberg REST catalog (s3-compat storage flavor), backed by Postgres and authenticating against the Keycloak realm shared with the existing minicluster_lakekeeper stack. - A one-shot bootstrap container (setup.sh + seed-table.py) that creates the test bucket with a SigV4-signed PUT via curl (no vendor CLI needed), creates the warehouse and the ice_s3 namespace, and seeds the ice_s3.nation and ice_s3.many_files tables via pyiceberg. run-lakekeeper-s3.sh / kill-lakekeeper-s3.sh start and stop the stack; the start script waits for the bootstrap container to finish so Impala never connects before the warehouse exists. Minicluster config: core-site.xml.py honours S3_ENDPOINT (and S3_CONNECTION_SSL_ENABLED) to point fs.s3a at RustFS with path-style access. Two catalog configs are added: iceberg_s3_vended_config with vending enabled and iceberg_s3_novend_config as the negative variant. tests/custom_cluster/test_iceberg_credential_vending.py brings the stack up around the test class (skipped when Docker is unavailable) and verifies that - a scan of the S3-backed table succeeds with vended credentials, - vended credentials coexist with, and win over, the process-global --s3a_*_key_cmd credentials for the table's prefix, - the scan fails to authenticate when vending is disabled and no other S3 credentials are configured (Lakekeeper vends regardless of the access-delegation header, so this also covers Impala ignoring them), - two tables in one bucket with different prefix-scoped credentials can be planned and scanned in the same query. Change-Id: I913b43300f8e0c7052b0b1fea609dc0ad50bce9b Co-Authored-By: Claude Fable 5.1 <[email protected]> --- A testdata/bin/kill-lakekeeper-s3.sh A testdata/bin/minicluster_lakekeeper_s3/Dockerfile A testdata/bin/minicluster_lakekeeper_s3/create-s3-warehouse.json A testdata/bin/minicluster_lakekeeper_s3/docker-compose.yaml A testdata/bin/minicluster_lakekeeper_s3/seed-table.py A testdata/bin/minicluster_lakekeeper_s3/setup.sh A testdata/bin/run-lakekeeper-s3.sh M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.py A testdata/configs/catalog_configs/iceberg_s3_novend_config/s3-novend.properties A testdata/configs/catalog_configs/iceberg_s3_vended_config/s3-vended.properties A tests/custom_cluster/test_iceberg_credential_vending.py 11 files changed, 827 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/24839/2 -- To view, visit http://gerrit.cloudera.org:8080/24839 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I913b43300f8e0c7052b0b1fea609dc0ad50bce9b Gerrit-Change-Number: 24839 Gerrit-PatchSet: 2 Gerrit-Owner: Peter Rozsa <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
