morazow commented on code in PR #3515: URL: https://github.com/apache/fluss/pull/3515#discussion_r3467612974
########## website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/polaris.md: ########## @@ -0,0 +1,144 @@ +--- +title: Polaris +sidebar_position: 2 +--- + +# Polaris + +## Introduction + +[Apache Polaris](https://polaris.apache.org/) is an open-source, fully-featured catalog for Apache Iceberg. It implements Iceberg's [REST Catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) interface, making Iceberg tables discoverable and queryable by any Iceberg-compatible engine, with role-based access control and credential vending built in. + +This guide explains how to configure Fluss to use Polaris as its Iceberg catalog. For general Iceberg integration details (table mapping, data types, limitations), see [Iceberg](../formats/iceberg.md). + +## How It Works + +When Fluss is configured with Polaris as its Iceberg REST catalog: + +1. Fluss creates and manages Iceberg table metadata through Polaris's REST API +2. The [tiering service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service) writes data to object storage and commits snapshots via Polaris +3. Any Iceberg-compatible engine (Flink, Spark, Trino, StarRocks, etc.) can discover and query the tiered tables through Polaris + +## Prerequisites + +### Running Polaris Instance + +You need a running Polaris instance with a catalog and a principal (a `client_id` / `client_secret` pair) that can access it. The fastest way to get started is the [Polaris Quickstart](https://polaris.apache.org/), which starts Polaris and automatically creates a `quickstart_catalog` plus a `quickstart_user` principal, printing the principal's credentials in the container logs. + +To create a catalog manually, first obtain an access token with your root credentials: + +```bash +export TOKEN=$(curl -s http://<polaris-host>:8181/api/catalog/v1/oauth/tokens \ + -d 'grant_type=client_credentials' \ + -d 'client_id=<root-client-id>' \ + -d 'client_secret=<root-client-secret>' \ + -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r '.access_token') +``` + +Then create a catalog backed by your object storage: + +```bash +curl -X POST http://<polaris-host>:8181/api/management/v1/catalogs \ + -H "Authorization: Bearer ${TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "catalog": { + "name": "my_catalog", + "type": "INTERNAL", + "properties": { "default-base-location": "s3://my-bucket/iceberg" }, + "storageConfigInfo": { + "storageType": "S3", + "allowedLocations": ["s3://my-bucket/iceberg"], + "roleArn": "<your-role-arn>" + } + } + }' +``` + +> **NOTE**: Adjust the `storageConfigInfo` to match your storage backend. Polaris supports S3, Azure, and GCS. You also need a principal with a role granting `TABLE_WRITE_DATA` on the catalog — see the [Polaris documentation](https://polaris.apache.org/) for catalog, principal, and access-control setup. Review Comment: A note on the vended credentials: - the catalog's storageConfigInfo must carry a roleArn - and Polaris must be able to AssumeRole against it The example guide already includes `"roleArn": "<your-role-arn>"`, so it is consistent. However, without roleArn table operations fail with `Failed to get subscoped credentials: roleArn must not be null`. For STS-less or no roleArn setups (e.g, MinIO, NooBaa), the working path is to skip credential subscoping (`SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION`) and supply static `fs.s3a.*` keys instead. Maybe good idea to add a short note distinguishing the two paths (vended w/ roleArn vs. static keys). ########## website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/polaris.md: ########## @@ -0,0 +1,144 @@ +--- +title: Polaris +sidebar_position: 2 +--- + +# Polaris + +## Introduction + +[Apache Polaris](https://polaris.apache.org/) is an open-source, fully-featured catalog for Apache Iceberg. It implements Iceberg's [REST Catalog](https://iceberg.apache.org/concepts/catalog/#decoupling-using-the-rest-catalog) interface, making Iceberg tables discoverable and queryable by any Iceberg-compatible engine, with role-based access control and credential vending built in. + +This guide explains how to configure Fluss to use Polaris as its Iceberg catalog. For general Iceberg integration details (table mapping, data types, limitations), see [Iceberg](../formats/iceberg.md). + +## How It Works + +When Fluss is configured with Polaris as its Iceberg REST catalog: + +1. Fluss creates and manages Iceberg table metadata through Polaris's REST API +2. The [tiering service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service) writes data to object storage and commits snapshots via Polaris +3. Any Iceberg-compatible engine (Flink, Spark, Trino, StarRocks, etc.) can discover and query the tiered tables through Polaris + +## Prerequisites + +### Running Polaris Instance + +You need a running Polaris instance with a catalog and a principal (a `client_id` / `client_secret` pair) that can access it. The fastest way to get started is the [Polaris Quickstart](https://polaris.apache.org/), which starts Polaris and automatically creates a `quickstart_catalog` plus a `quickstart_user` principal, printing the principal's credentials in the container logs. + +To create a catalog manually, first obtain an access token with your root credentials: + +```bash +export TOKEN=$(curl -s http://<polaris-host>:8181/api/catalog/v1/oauth/tokens \ + -d 'grant_type=client_credentials' \ + -d 'client_id=<root-client-id>' \ + -d 'client_secret=<root-client-secret>' \ + -d 'scope=PRINCIPAL_ROLE:ALL' | jq -r '.access_token') Review Comment: The token calls assume the default Polaris realm. On a non-default realm, requests may a Polaris-Realm header e.g. `-H "Polaris-Realm: <REALM>"` or they'll resolve against the wrong/default realm. Worth a one-line note so readers running a custom realm don't get silently misrouted. Same for the below `curl` command. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
