This is an automated email from the ASF dual-hosted git repository.
emaynard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/polaris.git
The following commit(s) were added to refs/heads/main by this push:
new 645553840 Use env var in spark container (#1522)
645553840 is described below
commit 645553840f32f354d3f16516d61c5cc48e077ce8
Author: Owen Lin (You-Cheng Lin)
<106612301+owenoweni...@users.noreply.github.com>
AuthorDate: Tue May 13 15:31:45 2025 +0800
Use env var in spark container (#1522)
* added
Signed-off-by: owenowenisme <mses010...@gmail.com>
* fix
Signed-off-by: owenowenisme <mses010...@gmail.com>
* add export
Signed-off-by: owenowenisme <mses010...@gmail.com>
* update docs using .env
Signed-off-by: owenowenisme <mses010...@gmail.com>
* update docs
Signed-off-by: owenowenisme <mses010...@gmail.com>
* change back from using .env to export
Signed-off-by: owenowenisme <mses010...@gmail.com>
* Apply suggestions from code review
Co-authored-by: Adnan Hemani <adna...@berkeley.edu>
---------
Signed-off-by: owenowenisme <mses010...@gmail.com>
Co-authored-by: Adnan Hemani <adna...@berkeley.edu>
---
getting-started/assets/polaris/create-catalog.sh | 2 +-
.../assets/trino-config/catalog/iceberg.properties | 2 +-
getting-started/eclipselink/docker-compose.yml | 8 ++++++-
.../unreleased/getting-started/using-polaris.md | 28 ++++++++++------------
4 files changed, 22 insertions(+), 18 deletions(-)
diff --git a/getting-started/assets/polaris/create-catalog.sh
b/getting-started/assets/polaris/create-catalog.sh
index a4e828769..c35a07cc5 100755
--- a/getting-started/assets/polaris/create-catalog.sh
+++ b/getting-started/assets/polaris/create-catalog.sh
@@ -22,7 +22,7 @@ set -e
apk add --no-cache jq
token=$(curl -s http://polaris:8181/api/catalog/v1/oauth/tokens \
- --user root:s3cr3t \
+ --user ${CLIENT_ID}:${CLIENT_SECRET} \
-d grant_type=client_credentials \
-d scope=PRINCIPAL_ROLE:ALL | sed -n 's/.*"access_token":"\([^"]*\)".*/\1/p')
diff --git a/getting-started/assets/trino-config/catalog/iceberg.properties
b/getting-started/assets/trino-config/catalog/iceberg.properties
index 28c3c61fa..176a623b7 100644
--- a/getting-started/assets/trino-config/catalog/iceberg.properties
+++ b/getting-started/assets/trino-config/catalog/iceberg.properties
@@ -21,7 +21,7 @@ connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=http://polaris:8181/api/catalog
iceberg.rest-catalog.security=OAUTH2
-iceberg.rest-catalog.oauth2.credential=root:s3cr3t
+iceberg.rest-catalog.oauth2.credential=${ENV:CLIENT_ID}:${ENV:CLIENT_SECRET}
iceberg.rest-catalog.oauth2.scope=PRINCIPAL_ROLE:ALL
iceberg.rest-catalog.warehouse=quickstart_catalog
# Required to support local filesystem:
https://trino.io/docs/current/object-storage.html#configuration
diff --git a/getting-started/eclipselink/docker-compose.yml
b/getting-started/eclipselink/docker-compose.yml
index 5589dfb3a..68a682c62 100644
--- a/getting-started/eclipselink/docker-compose.yml
+++ b/getting-started/eclipselink/docker-compose.yml
@@ -36,6 +36,7 @@ services:
polaris.persistence.eclipselink.configuration-file:
/deployments/config/eclipselink/persistence.xml
polaris.realm-context.realms: POLARIS
quarkus.otel.sdk.disabled: "true"
+ POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,${CLIENT_ID},${CLIENT_SECRET}
volumes:
- ../assets/eclipselink/:/deployments/config/eclipselink
healthcheck:
@@ -54,6 +55,8 @@ services:
- STORAGE_LOCATION=${STORAGE_LOCATION}
- AWS_ROLE_ARN=${AWS_ROLE_ARN}
- AZURE_TENANT_ID=${AZURE_TENANT_ID}
+ - CLIENT_ID=${CLIENT_ID}
+ - CLIENT_SECRET=${CLIENT_SECRET}
volumes:
- ../assets/polaris/:/polaris
entrypoint: '/bin/sh -c "chmod +x /polaris/create-catalog.sh &&
/polaris/create-catalog.sh"'
@@ -79,7 +82,7 @@ services:
--conf, "spark.sql.catalog.quickstart_catalog.type=rest",
--conf,
"spark.sql.catalog.quickstart_catalog.warehouse=quickstart_catalog",
--conf,
"spark.sql.catalog.quickstart_catalog.uri=http://polaris:8181/api/catalog";,
- --conf, "spark.sql.catalog.quickstart_catalog.credential=root:s3cr3t",
+ --conf,
"spark.sql.catalog.quickstart_catalog.credential=${USER_CLIENT_ID}:${USER_CLIENT_SECRET}",
--conf, "spark.sql.catalog.quickstart_catalog.scope=PRINCIPAL_ROLE:ALL",
--conf, "spark.sql.defaultCatalog=quickstart_catalog",
--conf, "spark.sql.catalogImplementation=in-memory",
@@ -91,6 +94,9 @@ services:
depends_on:
polaris-setup:
condition: service_completed_successfully
+ environment:
+ - CLIENT_ID=${CLIENT_ID}
+ - CLIENT_SECRET=${CLIENT_SECRET}
stdin_open: true
tty: true
ports:
diff --git a/site/content/in-dev/unreleased/getting-started/using-polaris.md
b/site/content/in-dev/unreleased/getting-started/using-polaris.md
index 75004e3ea..d452a3522 100644
--- a/site/content/in-dev/unreleased/getting-started/using-polaris.md
+++ b/site/content/in-dev/unreleased/getting-started/using-polaris.md
@@ -21,7 +21,12 @@ Title: Using Polaris
type: docs
weight: 400
---
-
+## Setup
+Define your `CLIENT_ID` & `CLIENT_SECRET` and export them for future use.
+```shell
+export CLIENT_ID=YOUR_CLIENT_ID
+export CLIENT_SECRET=YOUR_CLIENT_SECRET
+```
## Defining a Catalog
In Polaris, the [catalog]({{% relref "../entities#catalog" %}}) is the
top-level entity that objects like [tables]({{% relref "../entities#table" %}})
and [views]({{% relref "../entities#view" %}}) are organized under. With a
Polaris service running, you can create a catalog like so:
@@ -79,11 +84,13 @@ With a catalog created, we can create a [principal]({{%
relref "../entities#prin
Be sure to provide the necessary credentials, hostname, and port as before.
-When the `principals create` command completes successfully, it will return
the credentials for this new principal. Be sure to note these down for later.
For example:
+When the `principals create` command completes successfully, it will return
the credentials for this new principal. Export them for future use. For example:
-```
+```shell
./polaris ... principals create example
{"clientId": "XXXX", "clientSecret": "YYYY"}
+export USER_CLIENT_ID=XXXX
+export USER_CLIENT_SECRET=YYYY
```
Now, we grant the principal the [principal role]({{% relref
"../entities#principal-role" %}}) we created, and grant the [catalog role]({{%
relref "../entities#catalog-role" %}}) the principal role we created. For more
information on these entities, please refer to the linked documentation.
@@ -154,27 +161,21 @@ bin/spark-sql \
--conf
spark.sql.catalog.quickstart_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf
spark.sql.catalog.quickstart_catalog.catalog-impl=org.apache.iceberg.rest.RESTCatalog
\
--conf
spark.sql.catalog.quickstart_catalog.uri=http://localhost:8181/api/catalog \
---conf spark.sql.catalog.quickstart_catalog.credential='XXXX:YYYY' \
+--conf
spark.sql.catalog.quickstart_catalog.credential='${USER_CLIENT_ID}:${USER_CLIENT_SECRET}'
\
--conf spark.sql.catalog.quickstart_catalog.scope='PRINCIPAL_ROLE:ALL' \
--conf spark.sql.catalog.quickstart_catalog.token-refresh-enabled=true \
--conf spark.sql.catalog.quickstart_catalog.client.region=us-west-2
```
-Replace `XXXX` and `YYYY` with the client ID and client secret generated when
you created the `quickstart_user` principal.
-
Similar to the CLI commands above, this configures Spark to use the Polaris
running at `localhost:8181`. If your Polaris server is running elsewhere, but
sure to update the configuration appropriately.
Finally, note that we include the `hadoop-aws` package here. If your table is
using a different filesystem, be sure to include the appropriate dependency.
#### Using Spark SQL from a Docker container
-Replace the credentials used in the Docker container using the following code:
-
+Refresh the Docker container with the user's credentials:
```shell
-USER_CLIENT_ID="XXXX"
-USER_CLIENT_SECRET="YYYY"
-sed -i
"s/^\(.*spark\.sql\.catalog\.quickstart_catalog\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}\",/"
getting-started/eclipselink/docker-compose.yml
docker compose -f getting-started/eclipselink/docker-compose.yml up -d
```
@@ -233,12 +234,9 @@ org.apache.iceberg.exceptions.ForbiddenException:
Forbidden: Principal 'quicksta
### Connecting with Trino
-Replace the credentials used in the Docker container using the following code:
+Refresh the Docker container with the user's credentials:
```shell
-USER_CLIENT_ID="XXXX"
-USER_CLIENT_SECRET="YYYY"
-sed -i
"s/^\(iceberg\.rest-catalog\.oauth2\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}/"
getting-started/eclipselink/trino-config/catalog/iceberg.properties
docker compose -f getting-started/eclipselink/docker-compose.yml down trino
docker compose -f getting-started/eclipselink/docker-compose.yml up -d
```
