This is an automated email from the ASF dual-hosted git repository.
roryqi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new cf1b173ca8 [MINOR] docs: update credential vending, add vended
credentials to Trino (#12165)
cf1b173ca8 is described below
commit cf1b173ca8bf99b719573a407c2923a95d7127fa
Author: Mark Hoerth <[email protected]>
AuthorDate: Thu Jul 23 03:15:53 2026 -0700
[MINOR] docs: update credential vending, add vended credentials to Trino
(#12165)
### What changes were proposed in this pull request?
Rewrites `docs/security/credential-vending.md` and fixes
`docs/iceberg-rest-engine/trino.md`.
**docs/security/credential-vending.md**
- Restructure around where credential vending properties are set. The
two property columns held the same setting twice, differing only by a
`gravitino.iceberg-rest.` prefix, so they collapse to one column plus a
single stated rule.
- Add a quick start showing a working catalog and engine configuration.
- Add the IAM trust and permission policies for `s3-token`, and the
equivalent role requirements for `oss-token`, `adls-token`, and
`gcs-token`.
- Correct provider auto-detection. `s3-role-arn` does not select
`s3-token`. `BaseCatalog.addStorageCredentialProviders` never inspects
the role ARN, so a catalog with the key pair and a role ARN but no
explicit `credential-providers` falls back to `s3-secret-key`.
- Remove `azure-storage-account-key` from `adls-token`.
`ADLSTokenGenerator.initialize` reads only `storageAccountName`,
`tenantId`, `clientId`, and `clientSecret`.
- Correct the Iceberg GCP and Aliyun bundle links, which pointed at each
other's artifacts.
- Correct `credential-cache-max-size`. The right-hand column gave
`gravitino.iceberg-rest.cache-max-size`; `CredentialConstants` defines
`credential-cache-max-size`, so the prefix rule holds with no exception.
- Cover the Iceberg, Hive, Glue, and Paimon catalog classpaths in
Deployment, which previously covered only the Iceberg REST server and
Fileset.
- State that `s3-secret-key`, `oss-secret-key`, and `azure-account-key`
return the configured long-lived key to the client.
**docs/iceberg-rest-engine/trino.md**
- Add `iceberg.rest-catalog.vended-credentials-enabled` and
`iceberg.rest-catalog.prefix`. Neither appeared anywhere in the docs,
while the page's comparison table advertised credential vending.
- Separate storage credentials from authentication so vended credentials
can be configured without static keys. Every previous example set static
S3 keys.
- Warn that `s3.aws-access-key` and `s3.aws-secret-key` silently
override vended credentials, so queries succeed while vending is not in
use.
- Scope the access control and credential vending claims in the
comparison table to what applies.
### Why are the changes needed?
The credential vending page carried four factual errors. The most
consequential is provider auto-detection: the page states that
`s3-role-arn` enables the `s3-token` provider. It does not. A catalog
configured with the access key pair and a role ARN, but without an
explicit `credential-providers`, falls back to `s3-secret-key` and vends
the long-lived access key. Users following the current page believe they
have scoped STS credentials and are shipping permanent ones to every
engine that loads a table.
The Trino page advertised credential vending in its comparison table
while never documenting `vended-credentials-enabled`, and set static S3
keys in all four of its configuration examples. Those keys cause Trino
to ignore vended credentials while queries continue to succeed, so the
setup appears correct and no vending occurs.
### Does this PR introduce _any_ user-facing change?
Documentation only. No API or property changes.
### How was this patch tested?
Docs build.
Behavioral claims were checked against `BaseCatalog`, `CredentialUtils`,
`CredentialConfig`, `CredentialConstants`, `ADLSTokenGenerator`, and the
catalog `build.gradle.kts` copy tasks.
The documented Trino configuration was verified end to end against
Gravitino 1.3.0 with AWS S3: Spark wrote an Iceberg table and Trino 478
read it back through the Iceberg REST endpoint using vended STS
credentials, with OAuth2 client-credentials authentication and
authorization enabled.
Flagged for review: the Azure role names for `adls-token`, the GCS role
for `gcs-token`, the Alibaba RAM policy syntax for `oss-token`, and
whether the Hive and Glue catalogs take `gravitino-{cloud}` or the
`-bundle` variant. These follow each vendor's documented model and the
Paimon precedent, but were not confirmed against a running deployment.
Co-authored-by: Mark Hoerth <[email protected]>
---
docs/iceberg-rest-engine/trino.md | 320 ++++++++++++-----------
docs/security/credential-vending.md | 505 +++++++++++++++++++++++-------------
2 files changed, 490 insertions(+), 335 deletions(-)
diff --git a/docs/iceberg-rest-engine/trino.md
b/docs/iceberg-rest-engine/trino.md
index 2a438830d6..83b9b4aad0 100755
--- a/docs/iceberg-rest-engine/trino.md
+++ b/docs/iceberg-rest-engine/trino.md
@@ -1,244 +1,268 @@
---
-title: "Connect Trino to Iceberg REST"
+title: "Connect Trino to the Iceberg REST Catalog"
sidebar_label: "Trino"
---
## Introduction
-Apache Gravitino exposes an [Iceberg REST catalog](../iceberg-rest-service.md)
endpoint that any
-Iceberg-compatible engine can connect to directly — without installing a
Gravitino-specific
-connector plugin. This page describes how to configure Trino to use
Gravitino's Iceberg REST
-(IRC) endpoint.
+Apache Gravitino exposes an Iceberg REST Catalog (IRC) endpoint that any
Iceberg-compatible engine
+can connect to directly, without installing a Gravitino-specific connector
plugin. This page
+describes how to configure Trino to use it.
-:::note
-This integration uses the standard Apache Iceberg REST catalog specification.
Gravitino enforces
-its full access-control model on all IRC requests.
-:::
+## Quick Start
-## Prerequisites
-
-- Apache Gravitino running with the Iceberg REST service enabled. See
- [Iceberg REST catalog service](../iceberg-rest-service.md) for setup
instructions.
-- The Gravitino IRC endpoint is accessible from the Trino coordinator and all
workers. The default
- port is `9001`.
-- Trino 469 or later recommended.
-
-## Configuration
-
-Create a catalog properties file in your Trino `etc/catalog/` directory. The
filename determines
-the catalog name in Trino — `gravitino_irc.properties` creates a catalog named
`gravitino_irc`.
-
-:::note
-The `warehouse` property is managed by the Gravitino IRC server and does not
need to be set in
-the Trino catalog configuration.
-:::
-
-### No Authentication
+A complete Trino catalog file using vended credentials with OAuth2
authentication. Place it in
+`etc/catalog/` and restart Trino. The sections below explain each part and the
alternatives.
```properties
connector.name=iceberg
iceberg.catalog.type=rest
-iceberg.rest-catalog.uri=http://<gravitino-host>:9001/iceberg
+iceberg.rest-catalog.uri=http://{gravitino_host}:9001/iceberg
+iceberg.rest-catalog.prefix={catalog}
-# Native S3 filesystem (Trino 430+)
+# Vended credentials
+iceberg.rest-catalog.vended-credentials-enabled=true
fs.native-s3.enabled=true
-s3.region=us-east-1
-s3.aws-access-key=<access-key>
-s3.aws-secret-key=<secret-key>
+s3.region={region_name}
+
+# OAuth2
+iceberg.rest-catalog.security=OAUTH2
+iceberg.rest-catalog.oauth2.credential={client_id}:{client_secret}
+iceberg.rest-catalog.oauth2.server-uri={oauth_server_uri}
+iceberg.rest-catalog.oauth2.scope={scope}
# Table defaults
iceberg.file-format=PARQUET
iceberg.compression-codec=ZSTD
```
-### Basic Authentication
+## Prerequisites
-Requires Trino **481+**. Trino has no native Basic mode for Iceberg REST; pass
`Authorization`
-via HTTP headers.
+- Apache Gravitino running with the Iceberg REST service enabled. See
+ [Iceberg REST catalog service](../iceberg-rest-service.md) for setup
instructions.
+- The IRC endpoint reachable from the Trino coordinator and all workers. The
default port is `9001`.
+- Trino 430 or later, required for the native S3 filesystem. Verified on Trino
478. Some features
+ below need a higher version, noted where they apply.
-```shell
-echo -n '<username>:<password>' | base64
-```
+## Configuration
-```properties
-connector.name=iceberg
-iceberg.catalog.type=rest
-iceberg.rest-catalog.uri=http://<gravitino-host>:9001/iceberg
+Create a catalog properties file in your Trino `etc/catalog/` directory. The
filename determines the
+catalog name in Trino, so `gravitino_irc.properties` creates a catalog named
`gravitino_irc`.
-# Basic authentication
-iceberg.rest-catalog.http-headers=Authorization: Basic <base64-credentials>
+The Quick Start above is a complete working file. The rest of this page
explains each part and the
+alternatives. Storage credentials and authentication are independent choices,
so pick one option from
+each of the two sections below.
-# Native S3 filesystem (Trino 430+)
-fs.native-s3.enabled=true
-s3.region=us-east-1
-s3.aws-access-key=<access-key>
-s3.aws-secret-key=<secret-key>
+`iceberg.rest-catalog.prefix` selects which Gravitino catalog to use and must
match the catalog name
+in Gravitino. Confirm the expected value against the server:
-# Table defaults
-iceberg.file-format=PARQUET
-iceberg.compression-codec=ZSTD
+```bash
+curl -s "http://{gravitino_host}:9001/iceberg/v1/config?warehouse={catalog}";
```
-Replace `<base64-credentials>` with the output of `echo -n
'<username>:<password>' | base64`.
+Gravitino returns the prefix under `defaults`, so setting it explicitly in
Trino matches rather than
+overrides it.
-### OAuth2 Authentication
+The `warehouse` property is managed by the IRC and does not need to be set in
the Trino catalog file.
-```properties
-connector.name=iceberg
-iceberg.catalog.type=rest
-iceberg.rest-catalog.uri=http://<gravitino-host>:9001/iceberg
+## Storage Credentials
-# OAuth2 authentication
-iceberg.rest-catalog.security=OAUTH2
-iceberg.rest-catalog.oauth2.token=<your-token>
+Trino needs credentials to read and write the underlying object storage.
Choose one of the two
+options below. Do not configure both.
-# Native S3 filesystem (Trino 430+)
-fs.native-s3.enabled=true
-s3.region=us-east-1
-s3.aws-access-key=<access-key>
-s3.aws-secret-key=<secret-key>
+### Vended Credentials
-# Table defaults
-iceberg.file-format=PARQUET
-iceberg.compression-codec=ZSTD
-```
+Used in the Quick Start above. Gravitino mints short-lived, path-scoped
credentials at query time and
+returns them to Trino, so no long-lived storage keys live in the Trino
configuration. See
+[Credential vending](../security/credential-vending.md) for the catalog-side
configuration this
+requires.
+
+Do not set `s3.aws-access-key` or `s3.aws-secret-key` alongside vended
credentials. When static keys
+are present, Trino uses them and ignores the vended credentials. Queries still
succeed, so the
+catalog appears correctly configured while vending is not actually in use.
+
+`fs.native-s3.enabled=true` is required. The native S3 filesystem performs the
request signing that
+consumes vended credentials. Without it, metadata operations succeed and data
reads fail with
+`ICEBERG_FILESYSTEM_ERROR`.
-See [How to authenticate](../security/how-to-authenticate.md) for Gravitino
authentication
-configuration options.
+Trino implements vended credentials for S3 only. Vended credential support for
GCS and Azure are
+open Trino feature requests
([trinodb/trino#24518](https://github.com/trinodb/trino/issues/24518),
+[trinodb/trino#23238](https://github.com/trinodb/trino/issues/23238)), so use
static credentials for
+those backends.
-:::tip Local development
-For local development with MinIO, replace the S3 section with:
+### Static Credentials
+
+Configure storage keys directly in Trino. Simpler to set up, but the keys are
long-lived, are not
+scoped to a table path, and are managed outside Gravitino.
```properties
fs.native-s3.enabled=true
-s3.endpoint=http://<minio-host>:9000
+s3.region={region_name}
+s3.aws-access-key={access_key_id}
+s3.aws-secret-key={secret_access_key}
+```
+
+For local development against MinIO:
+
+```properties
+fs.native-s3.enabled=true
+s3.endpoint=http://{minio_host}:9000
s3.path-style-access=true
-s3.aws-access-key=<minio-access-key>
-s3.aws-secret-key=<minio-secret-key>
+s3.aws-access-key={minio_access_key}
+s3.aws-secret-key={minio_secret_key}
s3.region=us-east-1
```
-See
[gravitino-irc-quickstart](https://github.com/markhoerth/gravitino-irc-quickstart)
for a
-complete local development environment using MinIO.
-:::
+## Authentication
-## Start Trino
+How Trino identifies itself to Gravitino. Independent of the storage
credential choice above.
-Trino is a server process — the catalog is picked up automatically when Trino
starts. After
-placing `gravitino_irc.properties` in `etc/catalog/`, restart Trino:
+### No Authentication
-```bash
-$TRINO_HOME/bin/launcher restart
-```
+Add nothing. Suitable only when Gravitino authentication is disabled.
-Once Trino is running, connect using the Trino CLI:
+### Basic Authentication
+
+Requires Trino 481 or later. Trino has no native Basic mode for Iceberg REST,
so pass the
+`Authorization` header directly. On earlier releases,
`iceberg.rest-catalog.http-headers` is not
+available and Basic authentication cannot be used.
```bash
-trino --server http://<trino-host>:8080 --catalog gravitino_irc
+echo -n '{username}:{password}' | base64
```
-Or connect without specifying a default catalog and qualify queries fully:
-
-```bash
-trino --server http://<trino-host>:8080
+```properties
+iceberg.rest-catalog.http-headers=Authorization: Basic {base64_credentials}
```
-## Examples
+### OAuth2 Authentication
-Once connected, use the Trino CLI or any Trino-compatible client.
+The Quick Start above uses the client credentials flow, which obtains and
renews tokens rather than
+carrying a static one that eventually expires. Prefer it.
-### List Schemas
+To carry a static token instead:
-```sql
-SHOW SCHEMAS FROM gravitino_irc;
+```properties
+iceberg.rest-catalog.security=OAUTH2
+iceberg.rest-catalog.oauth2.token={token}
```
-### List Tables
+On Trino 479 and later, add the following to avoid token-exchange behavior
that can cause repeated
+token requests:
-```sql
-SHOW TABLES FROM gravitino_irc.<namespace>;
+```properties
+iceberg.rest-catalog.session=NONE
+iceberg.rest-catalog.oauth2.token-exchange-enabled=false
```
-### Query a Table
+`iceberg.rest-catalog.session=NONE` is already the default and can be omitted.
-```sql
-SELECT * FROM gravitino_irc.<namespace>.<table> LIMIT 10;
+See [How to authenticate](../security/how-to-authenticate.md) for the
Gravitino side of this
+configuration.
+
+## Starting Trino
+
+Trino is a server process, and the catalog is picked up when Trino starts.
After placing
+`gravitino_irc.properties` in `etc/catalog/`, restart Trino:
+
+```bash
+$TRINO_HOME/bin/launcher restart
```
-### Create a Schema
+Trino needs roughly 20 seconds to accept queries after a restart, which is
long enough to produce
+misleading connection errors in scripted runs.
-When creating a schema in Trino, a storage location must be specified:
+Once Trino is running, connect using the Trino CLI:
-```sql
-CREATE SCHEMA gravitino_irc.<namespace>
-WITH (location = 's3://<bucket>/<namespace>/');
+```bash
+trino --server http://{trino_host}:8080 --catalog gravitino_irc
```
-### Create a Table
+Or connect without a default catalog and qualify queries fully:
-```sql
-CREATE TABLE gravitino_irc.<namespace>.new_table (
- id INTEGER,
- name VARCHAR,
- created_at TIMESTAMP
-)
-WITH (
- format = 'PARQUET',
- format_version = 2
-);
+```bash
+trino --server http://{trino_host}:8080
```
+## Verifying Credential Vending
+
+Confirm the server vends credentials before assuming Trino is using them. The
`storage-credentials`
+block in the `loadTable` response is the direct evidence:
+
+```bash
+curl -s -H "X-Iceberg-Access-Delegation: vended-credentials" \
+ -H "Authorization: Bearer {token}" \
+
http://{gravitino_host}:9001/iceberg/v1/{catalog}/namespaces/{namespace}/tables/{table}
+```
+
+Three markers distinguish genuine vending from static credentials passed
through: the access key
+begins with `ASIA` rather than `AKIA`, a session token is present, and the
prefix is scoped to the
+table path rather than the whole bucket.
+
## Known Issues
-### `SHOW SCHEMAS` Fails on Gravitino IRC (OAuth2, Nested Namespaces)
+### Vended Credentials Are Not Refreshed During Long-Running Queries
-**Cause:** When Trino connects to Gravitino IRC with
`iceberg.rest-catalog.security=OAUTH2`,
-`iceberg.rest-catalog.nested-namespace-enabled=true`, and
`iceberg.rest-catalog.session=NONE`
-(the default), `SHOW SCHEMAS` recursively calls Iceberg REST `listNamespaces`.
On Trino releases
-before 482, each recursive call creates a separate OAuth session, which can
trigger excessive token
-requests and cause errors such as `Connection pool shut down` or
`StackOverflowError`.
+**Cause:** Gravitino advertises a refresh endpoint in the `loadTable` response
as
+`client.refresh-credentials-endpoint`, but Trino does not call it when vended
credentials expire
+mid-query
([trinodb/trino#25827](https://github.com/trinodb/trino/issues/25827)). Scans
running past
+the STS session lifetime fail.
-**Solution:** Upgrade to Trino 482+.
+**Solution:** Raise `s3-token-expire-in-secs` on the Gravitino catalog,
together with the maximum
+session duration on the IAM role, or keep individual queries shorter than the
session lifetime.
+
+### `SHOW SCHEMAS` Fails With OAuth2 and Nested Namespaces
+
+**Cause:** With `iceberg.rest-catalog.security=OAUTH2`,
+`iceberg.rest-catalog.nested-namespace-enabled=true`, and
`iceberg.rest-catalog.session=NONE` (the
+default), `SHOW SCHEMAS` recursively calls Iceberg REST `listNamespaces`. On
Trino releases before
+482, each recursive call creates a separate OAuth session, which can trigger
excessive token requests
+and cause errors such as `Connection pool shut down` or `StackOverflowError`.
+
+**Solution:** Upgrade to Trino 482 or later.
### `TIMESTAMP WITH TIME ZONE` Values Are Not Adjusted to the Client Session
Time Zone
-**Cause:** For `TIMESTAMP WITH TIME ZONE` values, Trino does not adjust query
results according to
-the client session time zone. Unlike Spark and Flink, Trino displays these
values based on the
-stored timestamp-with-time-zone value.
+**Cause:** Trino does not adjust `TIMESTAMP WITH TIME ZONE` results according
to the client session
+time zone. Unlike Spark and Flink, it displays these values based on the stored
+timestamp-with-time-zone value.
-**Solution:** To convert a `TIMESTAMP WITH TIME ZONE` value to the current
client session time
-zone, use `at_timezone` together with `current_timezone()`:
+**Solution:** Convert with `at_timezone` and `current_timezone()`:
```sql
SELECT
id,
at_timezone(timestamp_with_timezone_column, current_timezone())
-FROM <catalog>.<namespace>.<table>;
+FROM {catalog}.{namespace}.{table};
```
-## Gravitino Connector vs. Iceberg REST
+### Trino Identifiers Are Not Case Sensitive
-| Feature | Gravitino Engine Connector | Iceberg REST
|
-|:-------------------------|:----------------------------|:------------------------------|
-| Engine plugin required | Yes | No
|
-| Gravitino access control | Yes | Yes
|
-| Supported engines | Trino, Spark, Flink, Daft | Any
Iceberg-compatible engine |
-| Credential vending | Varies | Yes (S3, GCS, OSS,
ADLS) |
+**Cause:** Trino identifiers are not treated as case sensitive, so metadata
names that differ only by
+letter case cannot be distinguished. See [Trino identifier
+documentation](https://trino.io/docs/current/language/reserved.html#language-identifiers).
The
+limitation comes from Trino itself and is not specific to Gravitino.
-### Trino Identifiers Are Not Treated as Case Sensitive
+**Solution:** Use lowercase metadata names, and avoid creating objects whose
names differ only by
+letter case.
-Trino identifiers are not treated as case sensitive. As a result, metadata
names that differ
-only by letter case cannot be distinguished. See [Trino identifier
-documentation](https://trino.io/docs/current/language/reserved.html#language-identifiers).
This
-limitation comes from Trino itself and is not specific to Gravitino.
+## Gravitino Connector vs. the IRC
-For the best compatibility with Trino:
+| Feature | Gravitino engine connector | IRC
|
+|:-------------------------|:---------------------------|:------------------------------|
+| Engine plugin required | Yes | No
|
+| Gravitino access control | Yes | Yes, for API-created
catalogs |
+| Supported engines | Trino, Spark, Flink, Daft | Any
Iceberg-compatible engine |
+| Credential vending | Varies | Yes, S3 only in
Trino |
-- Use lowercase metadata names.
-- Avoid creating objects whose names differ only by letter case.
+Catalogs created through the Gravitino REST catalog API are registered in a
metalake, so privileges
+can be granted on them and Gravitino access control applies to queries that
reach them over the IRC.
+Catalogs defined instead in the Iceberg REST service configuration file are
not registered in a
+metalake, so there is nothing to grant privileges on.
## Related
+- [Credential vending](../security/credential-vending.md)
- [Iceberg REST catalog service](../iceberg-rest-service.md)
- [Connect Spark to Iceberg REST](./spark.md)
- [Connect Flink to Iceberg REST](./flink.md)
diff --git a/docs/security/credential-vending.md
b/docs/security/credential-vending.md
old mode 100644
new mode 100755
index c4969cd262..1a4298b9be
--- a/docs/security/credential-vending.md
+++ b/docs/security/credential-vending.md
@@ -7,269 +7,400 @@ license: "This software is licensed under the Apache
License version 2."
## Background
-Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides an
unified way to control the access to diverse data sources in different
platforms.
-
-### Capabilities
-
-- Supports Gravitino Iceberg REST server.
-- Supports Gravitino server with the following catalog types:
-
- | Catalog type | Supported credential types |
Since version |
-
|-----------------|----------------------------------------------------|------------------|
- | Hadoop (Fileset)| S3, OSS, GCS, ADLS |
0.7.0-incubating |
- | Hive | S3, OSS, GCS, ADLS |
1.3.0 |
- | Iceberg | S3, OSS, GCS, ADLS |
1.3.0 |
- | Glue | S3 |
1.3.0 |
- | JDBC | JDBC user/password (`jdbc-user-password`) |
1.3.0 |
- | Paimon | S3, OSS, JDBC user/password (`jdbc-user-password`) |
1.3.0 |
-- Supports pluggable credentials with build-in credentials:
- - S3: `S3TokenCredential`, `S3SecretKeyCredential`, `AwsIrsaCredential`
- - GCS: `GCSTokenCredential`
- - ADLS: `ADLSTokenCredential`, `AzureAccountKeyCredential`
- - OSS: `OSSTokenCredential`, `OSSSecretKeyCredential`
-- Spark, Flink, and Trino connectors automatically consume vended credentials
for Hive, Iceberg, Glue, JDBC, and Paimon catalogs since 1.3.0.
+Gravitino credential vending is used to generate temporary or static
credentials for accessing data. With credential vending, Gravitino provides a
unified way to control access to diverse data sources across different
platforms.
+
+## Supported Catalogs
+
+| Catalog type | Vends |
+|--------------|---------------------------------|
+| Fileset | S3, OSS, GCS, ADLS |
+| Hive | S3, OSS, GCS, ADLS |
+| Iceberg | S3, OSS, GCS, ADLS |
+| Glue | S3 |
+| JDBC | JDBC user and password |
+| Paimon | S3, OSS, JDBC user and password |
+
+S3 is Amazon S3, OSS is Alibaba Cloud OSS, GCS is Google Cloud Storage, and
ADLS is Azure Data Lake Storage. The Gravitino Spark, Flink, and Trino
connectors consume vended credentials automatically for these catalogs.
+
+## Quick Start
+
+Vend scoped S3 credentials to Spark through the IRC. Create the catalog
through the Gravitino REST catalog API:
+
+```shell
+curl -X POST http://localhost:8090/api/metalakes/{metalake}/catalogs \
+-H "Content-Type: application/json" \
+-d '{
+ "name": "iceberg_catalog",
+ "type": "RELATIONAL",
+ "provider": "lakehouse-iceberg",
+ "properties": {
+ "catalog-backend": "jdbc",
+ "uri": "jdbc:postgresql://{postgres_host}:5432/{database}",
+ "jdbc-driver": "org.postgresql.Driver",
+ "jdbc-user": "{jdbc_user}",
+ "jdbc-password": "{jdbc_password}",
+ "jdbc-initialize": "true",
+ "warehouse": "s3://{bucket_name}/{warehouse_path}",
+ "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
+ "credential-providers": "s3-token",
+ "s3-access-key-id": "{access_key_id}",
+ "s3-secret-access-key": "{secret_access_key}",
+ "s3-region": "{region_name}",
+ "s3-role-arn": "{role_arn}"
+ }
+}'
+```
+
+Point Spark at the IRC and request vended credentials with the delegation
header:
+
+```shell
+./bin/spark-sql -v \
+--packages
org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.0,org.apache.iceberg:iceberg-aws-bundle:1.10.0
\
+--conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
+--conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
+--conf spark.sql.catalog.rest.type=rest \
+--conf spark.sql.catalog.rest.uri=http://127.0.0.1:9001/iceberg/ \
+--conf spark.sql.catalog.rest.prefix=iceberg_catalog \
+--conf
spark.sql.catalog.rest.header.X-Iceberg-Access-Delegation=vended-credentials
+```
+
+The role in `s3-role-arn` needs a trust policy and S3 permissions before this
works. See [`s3-token`](#s3-token).
+
+For Trino instead of Spark:
+
+```properties
+connector.name=iceberg
+iceberg.catalog.type=rest
+iceberg.rest-catalog.uri=http://127.0.0.1:9001/iceberg/
+iceberg.rest-catalog.prefix=iceberg_catalog
+iceberg.rest-catalog.vended-credentials-enabled=true
+fs.native-s3.enabled=true
+s3.region={region_name}
+```
+
+For the full Trino setup, see [Connect Trino to the
IRC](../iceberg-rest-engine/trino.md).
+
+## Setting Properties
+
+Credential vending properties go in the catalog's `properties` map when you
create it through the Gravitino REST catalog API, alongside the warehouse
location and the other catalog settings.
+
+The Gravitino Iceberg REST Catalog (IRC) is the exception. It can also read a
catalog straight from `gravitino.conf`, using the same property names prefixed
with `gravitino.iceberg-rest.`:
+
+```properties
+s3-role-arn # as a catalog property
+gravitino.iceberg-rest.s3-role-arn # in gravitino.conf
+```
+
+Catalogs defined in `gravitino.conf` are not registered in a metalake, so
Gravitino access control does not apply to them. Privileges are granted on
catalogs in a metalake, and there is nothing to grant them on.
## General Configurations
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since
Version |
-|-------------------------------------|--------------------------------------------------------|--------------------------------------------------------------------------------------------|---------------|----------|------------------|
-| `credential-provider-type` |
`gravitino.iceberg-rest.credential-provider-type` | Deprecated; use
`credential-providers` instead. |
(none) | Yes | 0.7.0-incubating |
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | The credential
provider types, separated by comma. |
(none) | Yes | 0.8.0-incubating |
-| `credential-cache-expire-ratio` |
`gravitino.iceberg-rest.credential-cache-expire-ratio` | Ratio of the
credential's expiration time when Gravitino remove credential from the cache. |
0.15 | No | 0.8.0-incubating |
-| `credential-cache-max-size` |
`gravitino.iceberg-rest.cache-max-size` | Max size for the
credential cache. |
10000 | No | 0.8.0-incubating |
+| Property | Description
| Default value | Required |
+|---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|---------------|----------------------|
+| `credential-providers` | The credential provider types, separated
by comma. If omitted, Gravitino infers some providers from the other properties
present. | (none) | Yes, unless inferred |
+| `credential-cache-expire-ratio` | Ratio of the credential's expiration time
when Gravitino removes the credential from the cache.
| 0.15 | No |
+| `credential-cache-max-size` | Max size for the credential cache.
| 10000 | No |
-## Build-in Credentials Configurations
+### Values for `credential-providers`
-### S3 Credentials
+| Value | Storage | Vends
|
+|----------------------|---------|------------------------------------------------------------|
+| `s3-token` | S3 | Temporary STS credentials, scoped to the
table path |
+| `aws-irsa` | S3 | Credentials from an IAM role for service
accounts, for EKS |
+| `s3-secret-key` | S3 | The configured static access key and secret
|
+| `oss-token` | OSS | Temporary STS credentials, scoped to the
table path |
+| `oss-secret-key` | OSS | The configured static access key and secret
|
+| `adls-token` | ADLS | A user delegation SAS token
|
+| `azure-account-key` | ADLS | The configured static storage account key
|
+| `gcs-token` | GCS | A downscoped access token
|
+| `jdbc-user-password` | JDBC | The configured JDBC username and password
|
-#### S3 IRSA Credential
+Each value has its own properties, listed in the sections below. To vend for
more than one storage type on a catalog, separate values with a comma. Custom
providers can be added by implementing `CredentialProvider`, described under
[Custom Credentials](#custom-credentials).
-A credential using AWS IAM Roles for Service Accounts (IRSA) to access S3 with
temporary credentials, typically used in EKS environments. This provider
supports both basic IRSA credentials and fine-grained path-based access control
with dynamically generated IAM policies.
+### When `credential-providers` Is Omitted
-**Features:**
-- **Basic IRSA mode**: Returns credentials with full permissions of the
associated IAM role (for non-path-based contexts)
-- **Fine-grained mode**: Generates path-specific credentials with minimal
required permissions (for table access with `X-Iceberg-Access-Delegation:
vended-credentials`)
-- **Automatic policy generation**: Creates custom IAM policies scoped to
specific table paths including data, metadata, and write locations
-- **EKS integration**: Leverages existing IRSA setup while providing enhanced
security through path-based restrictions
+If a catalog does not set `credential-providers`, Gravitino infers providers
from the credential properties present:
+| Properties present | Provider
enabled |
+|--------------------------------------------------------------|---------------------|
+| `s3-access-key-id` and `s3-secret-access-key` |
`s3-secret-key` |
+| `oss-access-key-id` and `oss-secret-access-key` |
`oss-secret-key` |
+| `azure-storage-account-name` and `azure-storage-account-key` |
`azure-account-key` |
+| `gcs-service-account-file` | `gcs-token`
|
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required |
Since Version |
-|-------------------------------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------|---------------|----------|---------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `aws-irsa` for AWS IRSA
credential provider.
| (none) | Yes | 1.0.0 |
-| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the IAM role to assume. Required for fine-grained
path-based access control. | (none) | Yes* | 1.0.0
|
-| `s3-region` | `gravitino.iceberg-rest.s3-region`
| The AWS region for STS operations. Used for fine-grained access
control. | (none) | No | 1.0.0
|
-| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | Token expiration time in
seconds for fine-grained credentials. Cannot exceed role's max session
duration. | 3600 | No | 1.0.0 |
-| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | Alternative STS endpoint
for fine-grained credential generation. Useful for S3-compatible services.
| (none) | No | 1.0.0 |
+JDBC catalogs additionally infer `jdbc-user-password` from `jdbc-user` and
`jdbc-password`.
-**Note**: `s3-role-arn` is required only when using fine-grained path-based
access control with vended credentials. For basic IRSA usage without path
restrictions, only `credential-providers=aws-irsa` is needed.
+Four providers have no inference rule and must always be set explicitly:
`s3-token`, `oss-token`, `adls-token`, and `aws-irsa`. In particular, setting
`s3-role-arn` without `credential-providers` does not enable `s3-token`. The
catalog falls back to `s3-secret-key` and vends the static access key instead,
which is long-lived and not scoped to the table path. Set
`credential-providers` explicitly whenever you want token-based vending.
-**Prerequisites for fine-grained mode:**
-- EKS cluster with IRSA properly configured
-- `AWS_WEB_IDENTITY_TOKEN_FILE` environment variable pointing to the service
account token
-- IAM role with permissions to assume the target role specified in
`s3-role-arn`
-- Target IAM role with necessary S3 permissions for the data locations
+## S3
-#### S3 Secret Key Credential
+### `s3-token`
-A credential with static S3 access key id and secret access key.
+Gravitino calls STS
[AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html)
and returns temporary credentials scoped to the table path.
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description |
Default value | Required | Since Version |
-|-------------------------------------|---------------------------------------------------|--------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-secret-key` for S3
secret key credential provider. | (none) | Yes | 0.8.0-incubating |
-| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data. | (none) | Yes | 0.6.0-incubating |
-| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data. | (none) | Yes | 0.6.0-incubating |
+| Property | Description
| Default value | Required |
+|-----------------------------|------------------------------------------------------------------------------------------------|---------------|----------|
+| `s3-role-arn` | ARN of the role Gravitino assumes, in the form
`arn:aws:iam::{account_id}:role/{role_name}`. | (none) | Yes |
+| `s3-region` | Region of the S3 service, like `us-west-2`.
| (none) | No |
+| `s3-token-expire-in-secs` | Session lifetime of the vended credentials.
Cannot exceed the role's maximum session duration. | 3600 | No |
+| `s3-external-id` | External ID passed on AssumeRole, for
cross-account trust policies that require one. | (none) | No
|
+| `s3-token-service-endpoint` | Alternative STS endpoint, for S3-compatible
storage such as MinIO. | (none) | No |
-#### S3 Token Credential
+Also set `s3-access-key-id` and `s3-secret-access-key`. Gravitino uses them to
call AssumeRole, not to reach data, and they are never sent to the engine.
-An S3 token is a token credential with scoped privileges, by leveraging STS
[Assume
Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html).
To use an S3 token credential, you should create a role and grant it proper
privileges.
+#### Trust Policy on the Role
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
-|-------------------------------------|----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `s3-token` for S3 token
credential provider.
| (none) | Yes
| 0.8.0-incubating |
-| `s3-access-key-id` |
`gravitino.iceberg-rest.s3-access-key-id` | The static access key ID
used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
-| `s3-secret-access-key` |
`gravitino.iceberg-rest.s3-secret-access-key` | The static secret access
key used to access S3 data.
| (none) | Yes |
0.6.0-incubating |
-| `s3-role-arn` | `gravitino.iceberg-rest.s3-role-arn`
| The ARN of the role to access the S3 data.
| (none) | Yes | 0.7.0-incubating |
-| `s3-region` | `gravitino.iceberg-rest.s3-region`
| The region of the S3 service, like `us-west-2`.
| (none) | No | 0.6.0-incubating |
-| `s3-external-id` |
`gravitino.iceberg-rest.s3-external-id` | The S3 external id to
generate token.
| (none) | No
| 0.7.0-incubating |
-| `s3-token-expire-in-secs` |
`gravitino.iceberg-rest.s3-token-expire-in-secs` | The S3 session token
expire time in secs, it couldn't exceed the max session time of the assumed
role. | 3600 | No
| 0.7.0-incubating |
-| `s3-token-service-endpoint` |
`gravitino.iceberg-rest.s3-token-service-endpoint` | An alternative endpoint of
the S3 token service, This could be used with s3-compatible object storage
service like MINIO that has a different STS endpoint. | (none) | No
| 0.8.0-incubating |
+The role in `s3-role-arn` must allow the `s3-access-key-id` principal to
assume it. Without this, AssumeRole is rejected and no credential is vended.
-### OSS Credentials
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [{
+ "Effect": "Allow",
+ "Principal": { "AWS": "arn:aws:iam::{account_id}:user/{gravitino_user}" },
+ "Action": "sts:AssumeRole"
+ }]
+}
+```
-#### OSS Secret Key Credential
+#### Permission Policy on the Role
+
+The vended credentials inherit this policy, narrowed to the table path.
Without S3 access to the warehouse prefix, the credentials are vended but
cannot read or write.
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
+ "Resource": "arn:aws:s3:::{bucket_name}/{warehouse_path}/*"
+ },
+ {
+ "Effect": "Allow",
+ "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
+ "Resource": "arn:aws:s3:::{bucket_name}"
+ }
+ ]
+}
+```
-A credential with static OSS access key id and secret access key.
+### `aws-irsa`
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
-|-------------------------------------|---------------------------------------------------|-------------------------------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-secret-key` for OSS
secret credential. | (none) | Yes
| 0.8.0-incubating |
-| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
-| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data. | (none) | Yes
| 0.7.0-incubating |
+For Gravitino running on EKS. Instead of an access key, Gravitino uses its
pod's IAM role to call AssumeRole, so no static keys exist anywhere in the
setup.
-#### OSS Token Credential
+| Property | Description
| Default value | Required
|
+|-----------------------------|------------------------------------------------------------------------------------------------|---------------|-----------------------------|
+| `s3-role-arn` | ARN of the role to assume, in the form
`arn:aws:iam::{account_id}:role/{role_name}`. | (none) | For
path-scoped credentials |
+| `s3-region` | AWS region for STS operations.
| (none) | No
|
+| `s3-token-expire-in-secs` | Session lifetime of the vended credentials.
Cannot exceed the role's maximum session duration. | 3600 | No
|
+| `s3-token-service-endpoint` | Alternative STS endpoint, for S3-compatible
storage. | (none) | No
|
-An OSS token is a token credential with scoped privileges, by leveraging STS
[Assume
Role](https://www.alibabacloud.com/help/en/oss/developer-reference/use-temporary-access-credentials-provided-by-sts-to-access-oss).
To use an OSS token credential, you should create a role and grant it proper
privileges.
+Set `s3-role-arn` to get credentials scoped to the table path, with an IAM
policy generated per table covering its data, metadata, and write locations.
Without it, the vended credentials carry the full permissions of the pod's role.
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required |
Since Version |
-|-------------------------------------|---------------------------------------------------|--------------------------------------------------------------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `oss-token` for s3 token
credential.
| (none) | Yes | 0.8.0-incubating |
-| `oss-access-key-id` |
`gravitino.iceberg-rest.oss-access-key-id` | The static access key ID
used to access OSS data.
| (none) | Yes | 0.7.0-incubating |
-| `oss-secret-access-key` |
`gravitino.iceberg-rest.oss-secret-access-key` | The static secret access
key used to access OSS data.
| (none) | Yes | 0.7.0-incubating |
-| `oss-role-arn` | `gravitino.iceberg-rest.oss-role-arn`
| The ARN of the role to access the OSS data.
| (none) | Yes |
0.8.0-incubating |
-| `oss-region` | `gravitino.iceberg-rest.oss-region`
| The region of the OSS service, like `oss-cn-hangzhou`, only used
when `credential-providers` is `oss-token`. | (none) | No |
0.8.0-incubating |
-| `oss-external-id` |
`gravitino.iceberg-rest.oss-external-id` | The OSS external id to
generate token.
| (none) | No | 0.8.0-incubating |
-| `oss-token-expire-in-secs` |
`gravitino.iceberg-rest.oss-token-expire-in-secs` | The OSS security token
expire time in secs.
| 3600 | No | 0.8.0-incubating |
+The role in `s3-role-arn` needs the same two policies as `s3-token` above,
except the trust policy names the pod's IAM role rather than an IAM user.
-### ADLS Credentials
+IRSA itself must already be configured on the pod's Kubernetes service
account. When it is, EKS injects a signed service account token into the pod
and sets `AWS_WEB_IDENTITY_TOKEN_FILE` to its path, which is what the AWS SDK
uses to obtain credentials. If vending fails, check that this variable is
present in the pod.
-#### Azure Account Key Credential
+### `s3-secret-key`
-A credential with static Azure storage account name and key.
+Returns the catalog's configured access key and secret to the client,
unchanged, in the `loadTable` response.
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
-|-------------------------------------|-----------------------------------------------------|-----------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `azure-account-key` for
Azure account key credential. | (none) | Yes | 0.8.0-incubating
|
-| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
-| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
+The key is long-lived, carries whatever permissions its IAM user has, and is
not scoped to the table path. Any client that can load a table receives it, and
it stays valid after the query finishes. Prefer `s3-token`, which returns
temporary credentials scoped to the table path. Use `s3-secret-key` to confirm
the vending path works before configuring a role.
-#### ADLS Token Credential
+| Property | Description
| Default value | Required |
+|------------------------|------------------------------------------------------|---------------|----------|
+| `s3-access-key-id` | The static access key ID used to access S3 data.
| (none) | Yes |
+| `s3-secret-access-key` | The static secret access key used to access S3
data. | (none) | Yes |
-An ADLS token is a token credential with scoped privileges, by leveraging
Azure [User Delegation
Sas](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas).
To use an ADLS token credential, you should create a Microsoft Entra ID
service principal and grant it proper privileges.
+## OSS
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
-|-------------------------------------|-----------------------------------------------------|---------------------------------------------------------------------|---------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `adls-token` for ADLS
token credential. | (none) | Yes |
0.8.0-incubating |
-| `azure-storage-account-name` |
`gravitino.iceberg-rest.azure-storage-account-name` | The static storage
account name used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
-| `azure-storage-account-key` |
`gravitino.iceberg-rest.azure-storage-account-key` | The static storage
account key used to access ADLS data. | (none) | Yes |
0.8.0-incubating |
-| `azure-tenant-id` |
`gravitino.iceberg-rest.azure-tenant-id` | Azure Active Directory
(AAD) tenant ID. | (none) | Yes |
0.8.0-incubating |
-| `azure-client-id` |
`gravitino.iceberg-rest.azure-client-id` | Azure Active Directory
(AAD) client ID used for authentication. | (none) | Yes |
0.8.0-incubating |
-| `azure-client-secret` |
`gravitino.iceberg-rest.azure-client-secret` | Azure Active Directory
(AAD) client secret used for authentication. | (none) | Yes |
0.8.0-incubating |
-| `adls-token-expire-in-secs` |
`gravitino.iceberg-rest.adls-token-expire-in-secs` | The ADLS SAS token expire
time in secs. | 3600 | No |
0.8.0-incubating |
+### `oss-token`
-### GCS Credentials
+Gravitino calls Alibaba Cloud STS
[AssumeRole](https://www.alibabacloud.com/help/en/oss/developer-reference/use-temporary-access-credentials-provided-by-sts-to-access-oss)
and returns temporary credentials scoped to the table path.
-#### GCS Token Credential
+Also set `oss-access-key-id` and `oss-secret-access-key`. Gravitino uses them
to call AssumeRole, not to reach data, and they are never sent to the engine.
-An GCS token is a token credential with scoped privileges, by leveraging GCS
[Credential Access
Boundaries](https://cloud.google.com/iam/docs/downscoping-short-lived-credentials).
To use an GCS token credential, you should create an GCS service account and
grant it proper privileges.
+| Property | Description
| Default value |
Required |
+|----------------------------|--------------------------------------------------------------------------------------------------------------|---------------|----------|
+| `oss-access-key-id` | The static access key ID used to access OSS
data. | (none)
| Yes |
+| `oss-secret-access-key` | The static secret access key used to access OSS
data. | (none) |
Yes |
+| `oss-role-arn` | The ARN of the role to access the OSS data.
| (none) |
Yes |
+| `oss-region` | The region of the OSS service, like
`oss-cn-hangzhou`, only used when `credential-providers` is `oss-token`. |
(none) | No |
+| `oss-external-id` | The OSS external id to generate the token.
| (none) |
No |
+| `oss-token-expire-in-secs` | The OSS security token expire time in secs.
| 3600 |
No |
-| Gravitino server catalog properties | Gravitino Iceberg REST server
configurations | Description
| Default value | Required | Since Version |
-|-------------------------------------|---------------------------------------------------|------------------------------------------------------------|-------------------------------------|----------|------------------|
-| `credential-providers` |
`gravitino.iceberg-rest.credential-providers` | `gcs-token` for GCS token
credential. | (none) | Yes
| 0.8.0-incubating |
-| `gcs-credential-file-path` |
`gravitino.iceberg-rest.gcs-credential-file-path` | Deprecated; use
`gcs-service-account-file` instead. | GCS Application default
credential. | No | 0.7.0-incubating |
-| `gcs-service-account-file` |
`gravitino.iceberg-rest.gcs-service-account-file` | The location of GCS
credential file. | GCS Application default credential. |
No | 0.8.0-incubating |
+#### Trust Policy on the RAM Role
-:::note
-For the Gravitino Iceberg REST server, ensure that the credential file is
accessible by the server. For example, the server may be running on a GCE
machine, or you may set the environment variable `export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even
when `gcs-service-account-file` is already configured.
-:::
+The role in `oss-role-arn` must allow the `oss-access-key-id` principal to
assume it.
-## Custom Credentials
+```json
+{
+ "Version": "1",
+ "Statement": [{
+ "Effect": "Allow",
+ "Action": "sts:AssumeRole",
+ "Principal": { "RAM": ["acs:ram::{account_id}:user/{gravitino_user}"] }
+ }]
+}
+```
-Gravitino supports custom credentials, you can implement the
`org.apache.gravitino.credential.CredentialProvider` interface to support
custom credentials, and place the corresponding jar to the classpath of Iceberg
catalog server or Fileset catalog.
+#### Permission Policy on the RAM Role
+
+The vended credentials inherit this policy, narrowed to the table path.
+
+```json
+{
+ "Version": "1",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": ["oss:GetObject", "oss:PutObject", "oss:DeleteObject"],
+ "Resource": "acs:oss:*:*:{bucket_name}/{warehouse_path}/*"
+ },
+ {
+ "Effect": "Allow",
+ "Action": ["oss:ListObjects", "oss:GetBucketInfo"],
+ "Resource": "acs:oss:*:*:{bucket_name}"
+ }
+ ]
+}
+```
-## Deployment
+### `oss-secret-key`
-Besides setting credentials-related configuration, download the related cloud
bundle jar and place it in the classpath of the Iceberg REST server or Fileset
catalog.
+Returns the catalog's configured access key and secret to the client,
unchanged.
-For the Fileset catalog, use the Gravitino cloud bundle jar with Hadoop and
cloud packages:
+The key is long-lived, carries whatever permissions its RAM user has, and is
not scoped to the table path. Any client that can load a table receives it, and
it stays valid after the query finishes. Prefer `oss-token`. Use
`oss-secret-key` to confirm the vending path works before configuring a role.
-- [Gravitino AWS bundle jar with Hadoop and cloud
packages](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle)
-- [Gravitino Aliyun bundle jar with Hadoop and cloud
packages](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle)
-- [Gravitino GCP bundle jar with Hadoop and cloud
packages](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle)
-- [Gravitino Azure bundle jar with Hadoop and cloud
packages](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle)
+| Property | Description
| Default value | Required |
+|-------------------------|-------------------------------------------------------|---------------|----------|
+| `oss-access-key-id` | The static access key ID used to access OSS data.
| (none) | Yes |
+| `oss-secret-access-key` | The static secret access key used to access OSS
data. | (none) | Yes |
-For the Iceberg REST catalog server, download the corresponding Gravitino
cloud packages.
+## ADLS
-- [Gravitino Iceberg AWS bundle
JAR](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-aws-bundle)
-- [Gravitino Iceberg GCP bundle
JAR](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-aliyun-bundle)
-- [Gravitino Iceberg Aliyun bundle
JAR](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-gcp-bundle)
-- [Gravitino Iceberg Azure bundle
JAR](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-azure-bundle)
+### `adls-token`
-:::note
-Since Gravitino 1.1.0, the above Gravitino Iceberg cloud bundle jars have
already included the Iceberg cloud bundle jars, no need to download and include
them separately.
-:::
+Gravitino requests an Azure [user delegation
SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas)
and returns it to the client, scoped to the table path.
-The classpath of the server:
+Azure grants access through role assignments rather than policy documents. The
Microsoft Entra ID service principal identified by `azure-tenant-id`,
`azure-client-id`, and `azure-client-secret` needs two roles:
-- Iceberg REST server: the classpath differs by deployment mode; see the
[Deployment](../iceberg-rest-service.md#deployment) section.
-- Fileset catalog: `catalogs/fileset/libs/`
+- **Storage Blob Delegator**, assigned at the storage account, which allows it
to request the user delegation key that signs the SAS.
+- **Storage Blob Data Contributor**, assigned on the container or the
warehouse path, which determines what the vended SAS can do. Use **Storage Blob
Data Reader** for read-only access.
-## Credential Vending for Catalogs
+Without the delegator role the SAS cannot be issued at all. Without a data
role the SAS is issued but grants nothing.
-Hive, Iceberg, Glue, JDBC, and Paimon catalogs support server-side credential
vending since Gravitino 1.4.0. This section explains how credential vending
works for these catalog types and how it differs from the Hadoop Fileset and
Iceberg REST server cases.
+| Property | Description
| Default value | Required |
+|------------------------------|---------------------------------------------------------------------|---------------|----------|
+| `azure-storage-account-name` | The static storage account name used to
access ADLS data. | (none) | Yes |
+| `azure-tenant-id` | Azure Active Directory (AAD) tenant ID.
| (none) | Yes |
+| `azure-client-id` | Azure Active Directory (AAD) client ID used
for authentication. | (none) | Yes |
+| `azure-client-secret` | Azure Active Directory (AAD) client secret
used for authentication. | (none) | Yes |
+| `adls-token-expire-in-secs` | The ADLS SAS token expire time in secs.
| 3600 | No |
-### Auto-Detection of Credential Providers
+### `azure-account-key`
-Relational catalogs **do not** require an explicit `credential-providers`
property in the catalog configuration. Instead, Gravitino automatically detects
which credential providers to enable based on the catalog properties you supply:
+Returns the catalog's configured storage account key to the client, unchanged.
-- If `s3-access-key-id` / `s3-secret-access-key` are present, the S3
secret-key credential provider is enabled.
-- If `s3-role-arn` is also present, the S3 token (STS) credential provider is
enabled instead.
-- If `oss-access-key-id` / `oss-secret-access-key` are present, the OSS
secret-key or token credential provider is enabled.
-- For JDBC catalogs (and Paimon with `catalog-backend=jdbc`), if `jdbc-user` /
`jdbc-password` are set, the JDBC credential provider is enabled.
-- Paimon additionally supports OSS and S3 credential vending when the
corresponding cloud properties are set.
+A storage account key grants full access to every container in the storage
account, not just the warehouse path, and it does not expire. Any client that
can load a table receives it. Prefer `adls-token`, which is scoped and
time-limited. Use `azure-account-key` to confirm the vending path works before
configuring a service principal.
-No extra configuration is required beyond the catalog's normal properties.
+| Property | Description
| Default value | Required |
+|------------------------------|-----------------------------------------------------------|---------------|----------|
+| `azure-storage-account-name` | The static storage account name used to
access ADLS data. | (none) | Yes |
+| `azure-storage-account-key` | The static storage account key used to access
ADLS data. | (none) | Yes |
-### Security: Sensitive Properties Hidden from REST API
+## GCS
-To protect credentials, all sensitive catalog properties (such as
`s3-access-key-id`, `s3-secret-access-key`, `jdbc-user`, `jdbc-password`, etc.)
are excluded from the `GET /api/metalakes/{metalake}/catalogs/{catalog}`
response. Clients retrieve credentials through the dedicated credential vending
endpoint instead.
+### `gcs-token`
-### Credential Vending REST API
+Gravitino downscopes its own credentials using GCS [credential access
boundaries](https://cloud.google.com/iam/docs/downscoping-short-lived-credentials)
and returns a token scoped to the table path.
-Clients retrieve vended credentials from:
+There is no role to assume. The identity is the service account in
`gcs-service-account-file`, or the application default credentials when that is
unset. Grant that service account **Storage Object User**
(`roles/storage.objectUser`) on the warehouse bucket, or **Storage Object
Viewer** for read-only access. Downscoping narrows from those permissions, so
the vended token can never exceed what the service account itself holds.
-```
-GET /api/metalakes/{metalake}/objects/catalog/{catalog}/credentials
-```
+| Property | Description |
Default value | Required |
+|----------------------------|------------------------------------------|-------------------------------------|----------|
+| `gcs-service-account-file` | The location of the GCS credential file. | GCS
Application default credential. | No |
-The server returns short-lived or static credentials that the client can use
to access the underlying storage directly.
+For the IRC, ensure that the credential file is accessible by that server. For
example, the server may be running on a GCE machine, or you may set the
environment variable `export
GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json` even
when `gcs-service-account-file` is already configured.
-### Spark and Flink Connector Integration
+## Requesting Vended Credentials
-The Gravitino Spark and Flink connectors automatically call the credential
vending API and inject the returned credentials into the connector's
configuration, so no connector-side credential configuration is needed. For
example, `GravitinoHiveCatalog`, `GravitinoGlueCatalog`,
`GravitinoJdbcCatalog`, and `GravitinoIcebergCatalog` all consume the vended
credentials transparently.
+How a client asks depends on which interface it uses.
-### Backward Compatibility: `gravitino.catalog.credential.backfillToProperties`
+### Over the IRC
-During a rolling upgrade from Gravitino < 1.4.0 to 1.4.0, older clients that
read catalog properties directly (rather than calling `/credentials`) would
lose access to credentials because the properties are now hidden. To allow a
zero-downtime migration, set the following property in `gravitino.conf`:
+Credentials are vended only when the client asks for them. Spark, Flink, and
other IRC clients ask with a header:
-```properties
-gravitino.catalog.credential.backfillToProperties = true
+```
+X-Iceberg-Access-Delegation: vended-credentials
```
-When enabled, the server re-includes hidden credential properties in `GET
/catalogs/{catalog}` responses for backward compatibility.
+In Spark, set it as a catalog config key:
-:::caution
-`gravitino.catalog.credential.backfillToProperties = true` exposes credentials
in plaintext in catalog GET responses. Disable it once all clients have been
upgraded to use the credential vending API.
-:::
+```properties
+spark.sql.catalog.{name}.header.X-Iceberg-Access-Delegation=vended-credentials
+```
-## Example
+Trino asks with a catalog property instead, and sends the header for you:
+
+```properties
+iceberg.rest-catalog.vended-credentials-enabled=true
+```
-### Credential Vending for Iceberg REST Server
+### Over the Gravitino REST Catalog API
-Suppose the Iceberg table data is stored in S3, follow the steps below:
+Hive, Glue, JDBC, Paimon, and Fileset catalogs are reached through the
Gravitino REST catalog API, which has no delegation header. Credential
properties are hidden from the catalog GET response, so clients fetch them from
the Gravitino credential endpoint instead. It works for any metadata object:
-1. Download the [Gravitino Iceberg AWS bundle
JAR](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-aws-bundle),
and place it in the classpath of Iceberg REST server.
+```
+GET /api/metalakes/{metalake}/objects/{type}/{full_name}/credentials
+```
-2. Add s3 token credential configurations.
+For a catalog, `{type}` is `catalog` and `{full_name}` is the catalog name:
```
-gravitino.iceberg-rest.warehouse = s3://{bucket_name}/{warehouse_path}
-gravitino.iceberg-rest.io-impl= org.apache.iceberg.aws.s3.S3FileIO
-gravitino.iceberg-rest.credential-providers = s3-token
-gravitino.iceberg-rest.s3-access-key-id = xxx
-gravitino.iceberg-rest.s3-secret-access-key = xxx
-gravitino.iceberg-rest.s3-region = {region_name}
-gravitino.iceberg-rest.s3-role-arn = {role_arn}
+GET /api/metalakes/{metalake}/objects/catalog/{catalog}/credentials
```
-3. Explore the Iceberg table with a Spark client with credential vending
enabled.
+The Gravitino Spark and Flink connectors call this for you and inject the
returned credentials, so no client configuration is needed.
-```shell
-./bin/spark-sql -v \
---packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.3.1 \
---conf spark.jars={path}/iceberg-aws-bundle-1.5.2.jar \
---conf
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
\
---conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog \
---conf spark.sql.catalog.rest.type=rest \
---conf spark.sql.catalog.rest.uri=http://127.0.0.1:9001/iceberg/ \
---conf
spark.sql.catalog.rest.header.X-Iceberg-Access-Delegation=vended-credentials
+## Custom Credentials
+
+Gravitino supports custom credentials. You can implement the
`org.apache.gravitino.credential.CredentialProvider` interface to support
custom credentials, and place the corresponding jar in the classpath of the IRC
or the Fileset catalog.
+
+## Deployment
+
+The credential provider implementations ship in separate jars. Whichever
component vends the credentials needs the right jar on its classpath, or the
provider cannot be created and no credentials are vended.
+
+| Vending component | Jar |
Classpath
|
+|-------------------|----------------------------------------------------|----------------------------------------------------------------------------------------|
+| IRC | `gravitino-iceberg-{cloud}-bundle` | See
[Deployment](../iceberg-rest-service.md#deployment); it differs by deployment
mode |
+| Iceberg catalog | `gravitino-iceberg-{cloud}-bundle` |
`catalogs/lakehouse-iceberg/libs/`
|
+| Fileset catalog | `gravitino-{cloud}-bundle` |
`catalogs/fileset/libs/`
|
+| Hive catalog | `gravitino-{cloud}` |
`catalogs/hive/libs/`
|
+| Glue catalog | `gravitino-aws` |
`catalogs/glue/libs/`
|
+| Paimon catalog | `gravitino-aws` for S3, `gravitino-aliyun` for OSS |
`catalogs/lakehouse-paimon/libs/`
|
+
+Substitute `{cloud}` with `aws`, `gcp`, `aliyun`, or `azure`. Note the two jar
families: the `-bundle` variants also carry Hadoop and cloud SDK packages,
which the Fileset catalog and the IRC need. The Hive, Glue, and Paimon catalogs
only vend credentials, so they take the plain `gravitino-{cloud}` jar.
+
+Since Gravitino 1.1.0, the Gravitino Iceberg cloud bundle jars already include
the Iceberg cloud bundle jars, so there is no need to download and include
those separately.
+
+Vending JDBC user and password requires no additional jar.
+
+Bundle jars on Maven Central:
+
+-
[gravitino-aws-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle),
[gravitino-gcp-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle),
[gravitino-aliyun-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle),
[gravitino-azure-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle)
+-
[gravitino-iceberg-aws-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-aws-bundle),
[gravitino-iceberg-gcp-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-gcp-bundle),
[gravitino-iceberg-aliyun-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-aliyun-bundle),
[gravitino-iceberg-azure-bundle](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-iceberg-azure-bundle)
+-
[gravitino-aws](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws),
[gravitino-aliyun](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun)
+
+## Upgrading From a Release Earlier Than 1.3.0
+
+Since 1.3.0, sensitive catalog properties such as `s3-access-key-id`,
`s3-secret-access-key`, `jdbc-user`, and `jdbc-password` are excluded from `GET
/api/metalakes/{metalake}/catalogs/{catalog}`. Clients written against earlier
releases that read those properties directly lose access to them.
+
+For a zero-downtime migration, set the following in `gravitino.conf`:
+
+```properties
+gravitino.catalog.credential.backfillToProperties = true
```
+
+The Gravitino server then re-includes the hidden properties in its catalog GET
responses. Turn it off once all clients use the Gravitino credential endpoint,
since it exposes credentials in plaintext.
