dimas-b commented on code in PR #257: URL: https://github.com/apache/polaris-tools/pull/257#discussion_r3639173505
########## polaris-synchronizer/manual-test/README.md: ########## @@ -0,0 +1,122 @@ +# Manual Test Setup + +Brings up two independent Apache Polaris instances (source + target) to +manually validate `polaris-synchronizer`, including the `--skip-iceberg-content` +flag, without needing Postgres/RDS or S3/MinIO: + +* Both instances use Polaris's default **in-memory metastore** (no datasource + configured). +* Both instances share a single **FILE storage volume**, mounted at the same + path in each container — so table metadata written by source is readable + by target, and a full sync (including namespace/table content) actually + succeeds. This lets you run the full sync and the `--skip-iceberg-content` + sync back-to-back in the same session and compare the results. + +`polaris-init` seeds SOURCE with a catalog, principal, principal-role, +catalog-role, grant, namespace, and table. TARGET is left completely empty — +`polaris-synchronizer` is responsible for creating everything there. + +## Usage + +```bash +cd polaris-synchronizer/manual-test +docker compose up --build -d +docker compose ps # wait for polaris-init to show "Exited (0)" +cat seed/credentials.json Review Comment: Why advise users to dump credentials to STDOUT? ########## polaris-synchronizer/manual-test/docker-compose.yml: ########## @@ -0,0 +1,88 @@ +services: + + # ── Polaris instances ──────────────────────────────────────────────────────── + # Official Apache Polaris image. No Postgres/RDS — Polaris defaults to its + # in-memory metastore when no datasource is configured. Both instances share + # a single FILE storage volume mounted at the same path, so metadata written + # by source is readable by target — this lets a full sync (including + # namespace/table content) actually succeed, so you can compare it against + # a `--skip-iceberg-content` run in this example. + + polaris-source: + image: apache/polaris:1.1.0-incubating + container_name: polaris-source + hostname: polaris-source + ports: + - "8181:8181" + - "8182:8182" + environment: + POLARIS_BOOTSTRAP_CREDENTIALS: POLARIS,root,s3cr3t + JAVA_OPTS_APPEND: >- + -Dpolaris.readiness.ignore-severe-issues=true + -Dpolaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE"] + -Dpolaris.features."ALLOW_INSECURE_STORAGE_TYPES"=true + -Dpolaris.features."ALLOW_UNSTRUCTURED_TABLE_LOCATION"=true + healthcheck: + test: ["CMD", "curl", "-sf", "http://localhost:8182/q/health/ready";] + interval: 5s + timeout: 5s + retries: 20 + start_period: 20s + volumes: + - shared-storage:/tmp/polaris + networks: + - polaris-net + + polaris-target: + image: apache/polaris:1.1.0-incubating Review Comment: Why not 1.6.0? 😉 ########## polaris-synchronizer/manual-test/init/init.sh: ########## @@ -0,0 +1,162 @@ +#!/bin/bash +# Seeds SOURCE with: catalog + principal + principal-role + catalog-role + +# grant + namespace + table. TARGET is left completely empty so that +# polaris-synchronizer creates entities on target itself. + +set -euo pipefail + +SOURCE="http://polaris-source:8181"; +TARGET="http://polaris-target:8181"; +CATALOG="test-catalog" +ROOT_CLIENT_ID="root" +ROOT_SECRET="s3cr3t" + +wait_ready() { + local base=$1 + echo "[init] Waiting for $base..." + until curl -sf "$base/api/catalog/v1/oauth/tokens" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${ROOT_CLIENT_ID}&client_secret=${ROOT_SECRET}&scope=PRINCIPAL_ROLE:ALL" \ + >/dev/null 2>&1; do sleep 3; done + echo "[init] $base is ready" +} + +get_token() { + local base=$1 + curl -sf -X POST "$base/api/catalog/v1/oauth/tokens" \ + -H "Content-Type: application/x-www-form-urlencoded" \ + -d "grant_type=client_credentials&client_id=${ROOT_CLIENT_ID}&client_secret=${ROOT_SECRET}&scope=PRINCIPAL_ROLE:ALL" \ + | jq -r '.access_token' +} + +mgmt_post() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X POST "$base/api/management/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +mgmt_put() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X PUT "$base/api/management/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +catalog_post() { + local base=$1 token=$2 path=$3 body=$4 + curl -sf -X POST "$base/api/catalog/v1$path" \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d "$body" +} + +wait_ready "$SOURCE" +wait_ready "$TARGET" + +SRC_TOKEN=$(get_token "$SOURCE") +echo "[init] SOURCE admin token acquired" + +# ── SOURCE: catalog (FILE storage, local to the polaris-source container) ──── + +mgmt_post "$SOURCE" "$SRC_TOKEN" "/catalogs" '{ + "catalog": { + "name": "'"$CATALOG"'", + "type": "INTERNAL", + "properties": {"default-base-location": "file:///tmp/polaris/'"$CATALOG"'/"}, + "storageConfigInfo": { + "storageType": "FILE", + "allowedLocations": ["file:///tmp/polaris/'"$CATALOG"'"] + } + } +}' +echo "[init] SOURCE: catalog created" + +# ── SOURCE: principal ───────────────────────────────────────────────────────── + +PRINCIPAL_RESP=$(mgmt_post "$SOURCE" "$SRC_TOKEN" "/principals" '{ + "principal": {"name": "test-user", "type": "user"}, + "credentialRotationRequired": false +}') + +USER_CLIENT_ID=$(echo "$PRINCIPAL_RESP" | jq -r '.credentials.clientId') +USER_SECRET=$(echo "$PRINCIPAL_RESP" | jq -r '.credentials.clientSecret') +echo "[init] SOURCE: principal test-user created (clientId=$USER_CLIENT_ID)" + +# ── SOURCE: principal role ──────────────────────────────────────────────────── + +mgmt_post "$SOURCE" "$SRC_TOKEN" "/principal-roles" '{"principalRole": {"name": "analyst-role"}}' Review Comment: How about using the `setup` feature of Polaris CLI? https://polaris.apache.org/blog/2026/03/29/introducing-the-setup-command-in-apache-polaris/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
