MonkeyCanCode commented on code in PR #3482: URL: https://github.com/apache/polaris/pull/3482#discussion_r2708386267
########## getting-started/rustfs/README.md: ########## @@ -0,0 +1,95 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Getting Started with Apache Polaris and RustFS + +## Overview + +This example uses RustFS as a storage provider with Polaris. + +Spark is used as a query engine. This example assumes a local Spark installation. +See the [Spark Notebooks Example](../spark/README.md) for a more advanced Spark setup. + +## Starting the Example + +1. Build the Polaris server image if it's not already present locally: + + ```shell + ./gradlew \ + :polaris-server:assemble \ + :polaris-server:quarkusAppPartsBuild --rerun \ + -Dquarkus.container-image.build=true + ``` + +2. Start the docker compose group by running the following command from the root of the repository: + + ```shell + docker compose -f getting-started/rustfs/docker-compose.yml up Review Comment: interesting as it works for me while I was writing this one last night, let me see why that is the case: ``` ➜ polaris git:(rustfs) docker compose -f getting-started/rustfs/docker-compose.yml up WARN[0000] No services to build [+] up 5/5 ✔ Network rustfs_default Created 0.0s ✔ Container rustfs-rustfs-1 Created 0.0s ✔ Container rustfs-setup_bucket-1 Created 0.0s ✔ Container rustfs-polaris-1 Created 0.0s ✔ Container rustfs-polaris-setup-1 Created 0.0s Attaching to polaris-1, polaris-setup-1, rustfs-1, setup_bucket-1 Container rustfs-rustfs-1 Waiting Container rustfs-rustfs-1 Waiting rustfs-1 | Initializing data directories: /data rustfs-1 | OBS log directory not configured and logs outputs to stdout rustfs-1 | !!!WARNING: Using default RUSTFS_ACCESS_KEY or RUSTFS_SECRET_KEY. Override them in production! rustfs-1 | Starting: /usr/bin/rustfs /data rustfs-1 | RustFS Http API: http://172.18.0.2:9000 http://127.0.0.1:9000 rustfs-1 | RustFS Start Time: 2026-01-20 13:32:29 rustfs-1 | {"timestamp":"2026-01-20T13:32:29.720648925Z","level":"WARN","fields":{"message":"Detected default credentials 'rustfsadmin:rustfsadmin', we recommend that you change these values with 'RUSTFS_ACCESS_KEY' and 'RUSTFS_SECRET_KEY' environment variables"},"target":"rustfs::server::http","filename":"rustfs/src/server/http.rs","line_number":171,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | Console WebUI Start Time: 2026-01-20 13:32:29 rustfs-1 | Console WebUI available at: http://172.18.0.2:9001/rustfs/console/index.html rustfs-1 | Console WebUI (localhost): http://127.0.0.1:9001/rustfs/console/index.html rustfs-1 | {"timestamp":"2026-01-20T13:32:29.755828217Z","level":"WARN","fields":{"message":"config not found, start to init"},"target":"rustfs_ecstore::tier::tier","filename":"crates/ecstore/src/tier/tier.rs","line_number":490,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | {"timestamp":"2026-01-20T13:32:29.757488175Z","level":"WARN","fields":{"message":"Configuration not found (Read the main configuration): Start initializing new configuration"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":137,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | {"timestamp":"2026-01-20T13:32:29.759169967Z","level":"WARN","fields":{"message":"Configuration initialization complete (Read the main configuration)"},"target":"rustfs_ecstore::config::com","filename":"crates/ecstore/src/config/com.rs","line_number":139,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | {"timestamp":"2026-01-20T13:32:29.778327675Z","level":"WARN","fields":{"message":"Global region is not set; attempting notification configuration for all buckets with an empty region."},"target":"rustfs::init","filename":"rustfs/src/init.rs","line_number":101,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | {"timestamp":"2026-01-20T13:32:29.779977717Z","level":"WARN","fields":{"message":"main checkpoint file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-7a0b5a9f3ada404397601c2ff017fdca.json\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":154,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | {"timestamp":"2026-01-20T13:32:29.779994508Z","level":"WARN","fields":{"message":"backup file is corrupted or not exists: Not found: checkpoint file not exists: \"/tmp/rustfs_scanner/scanner_checkpoint_scanner-node-7a0b5a9f3ada404397601c2ff017fdca.backup\""},"target":"rustfs_ahm::scanner::checkpoint","filename":"crates/ahm/src/scanner/checkpoint.rs","line_number":172,"threadName":"main","threadId":"ThreadId(1)"} rustfs-1 | RustFS server started successfully at [::]:9000, current time: 2026-01-20 13:32:29.780014092 UTC Container rustfs-rustfs-1 Healthy Container rustfs-rustfs-1 Healthy setup_bucket-1 | Creating RustFS bucket... Container rustfs-polaris-1 Waiting polaris-1 | INFO exec -a "java" java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/quarkus-run.jar polaris-1 | INFO running in /deployments setup_bucket-1 | Added `pol` successfully. setup_bucket-1 | Bucket created successfully `pol/bucket123`. setup_bucket-1 | [2026-01-20 13:32:35 UTC] 0B bucket123/ setup_bucket-1 | Bucket setup complete. polaris-1 | Listening for transport dt_socket at address: 5005 setup_bucket-1 exited with code 0 polaris-1 | polaris-1 | @@ polaris-1 | @@ polaris-1 | @@@@ polaris-1 | @@@@@ @@ @@@ @@@@@@@@ @@@ @@@@@@@@ @@@ @@@ @@@@@@@@ polaris-1 | @@@@@@@@ @@@@ @@@@@ @@@ @@@ @@@@@ @@@@ @@@@ @@@ @@@ @@@ polaris-1 | @@@@@@@@ @@@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@@ @@@@@@@@@@ @@@@@@@ polaris-1 | @@@@@@@@@ @@@@@@@ @@@ @@@ @@@@@@@@ @@@ @@@ @@@@ @@@@@@@@@@ @@@@@@@ polaris-1 | @@@@@@@@@@@@@@@@@ @@@@@@@@@@ @@@ @@@@@@@@@@ @@@@ @@@@ @@@ @@@ @@@ polaris-1 | @@@@@@@@@@@@@@ @@@ @@@@ @@@ @@@ @@@@ @@@@@@@ @@@ @@@ @@@@@@@@ polaris-1 | @@@@@@@@@@@ polaris-1 | @@@@@@@@@@@@ @@@@@@@ @@@@@@@ @@@ @@@ @@@@@@@ @@@ @@@@@@ polaris-1 | @@@@@@@@@@@@@@@ @@@@@@@@@ @@@@@@@@@ @@@ @@@@@ @@@@@@@@ @@@ @@@ @@@ polaris-1 | @@@@@@@@@@@@@@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@ @@@@ polaris-1 | @@@@@@@@ @@@@@@@@ @@@@@@@@ @@@ @@@ @@@ @@@ @@@ @@@@@@@ @@@ @@@@ polaris-1 | @@@@@@ @@@@@@ @@@ @@@@@@@@@ @@@ @@@@@@@@@@ @@@ @@@ @@@ @@@ @@@ polaris-1 | @@@@@ @@@@@@ @@@ @@@@@@@ @@@@@@@ @@@ @@@@ @@@ @@@ @@@ @@@@@@ polaris-1 | @@@ @@@@ polaris-1 | @@@@ polaris-1 | @@@ polaris-1 | @@ polaris-1 | polaris-1 | Powered by Quarkus 3.29.4 polaris-1 | 2026-01-20 13:32:37,208 WARN [io.qua.config] [,] [,,,] (main) The "quarkus.log.file.enable" config property is deprecated and should not be used anymore. polaris-1 | 2026-01-20 13:32:37,208 WARN [io.qua.config] [,] [,,,] (main) The "quarkus.log.console.enable" config property is deprecated and should not be used anymore. polaris-1 | 2026-01-20 13:32:37,711 INFO [org.apa.pol.ser.con.ServiceProducers] [,] [,,,] (main) Bootstrapping realm(s) 'POLARIS', if necessary, from root credentials set provided via the environment variable POLARIS_BOOTSTRAP_CREDENTIALS or Java system property polaris.bootstrap.credentials ... polaris-1 | 2026-01-20 13:32:37,830 INFO [org.apa.pol.ser.con.ServiceProducers] [,] [,,,] (main) Realm 'POLARIS' automatically bootstrapped, credentials taken from root credentials set provided via the environment variable POLARIS_BOOTSTRAP_CREDENTIALS or Java system property polaris.bootstrap.credentials, not printed to stdout. polaris-1 | 2026-01-20 13:32:37,848 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) ⚠️ Production readiness checks failed! Check the warnings below. polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ The current metastore is intended for tests only. Offending configuration option: 'polaris.persistence.type'. polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ The realm context resolver is configured to map requests without a realm header to the default realm. Offending configuration option: 'polaris.realm-context.require-header'. polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ A public key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.public-key-file'. polaris-1 | 2026-01-20 13:32:37,849 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) - ⚠️ A private key file wasn't provided and will be generated. Offending configuration option: 'polaris.authentication.token-broker.rsa-key-pair.private-key-file'. polaris-1 | 2026-01-20 13:32:37,850 WARN [org.apa.pol.ser.con.ProductionReadinessChecks] [,] [,,,] (main) Refer to https://polaris.apache.org/in-dev/unreleased/configuring-polaris-for-production for more information. polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Apache Polaris Server (incubating) 1.3.0-incubating on JVM (powered by Quarkus 3.29.4) started in 2.406s. Listening on: http://0.0.0.0:8181. Management interface listening on http://0.0.0.0:8182. polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Profile prod activated. polaris-1 | 2026-01-20 13:32:37,924 INFO [io.quarkus] [,] [,,,] (main) Installed features: [agroal, amazon-sdk-rds, cdi, hibernate-validator, jdbc-postgresql, micrometer, narayana-jta, oidc, opentelemetry, reactive-routes, rest, rest-jackson, security, smallrye-context-propagation, smallrye-fault-tolerance, smallrye-health, vertx] Container rustfs-polaris-1 Healthy polaris-setup-1 | (1/2) Installing oniguruma (6.9.10-r0) polaris-setup-1 | (2/2) Installing jq (1.8.1-r0) polaris-setup-1 | Executing busybox-1.37.0-r30.trigger polaris-setup-1 | OK: 14.6 MiB in 28 packages polaris-1 | 2026-01-20 13:32:42,612 INFO [org.apa.pol.ser.con.PolarisIcebergObjectMapperCustomizer] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000001,POLARIS] [,,,] (executor-thread-1) Limiting request body size to 10485760 bytes polaris-1 | 2026-01-20 13:32:42,641 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000001,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - - [20/Jan/2026:13:32:42 +0000] "POST /api/catalog/v1/oauth/tokens HTTP/1.1" 200 757 polaris-setup-1 | Creating catalog... polaris-setup-1 | OK: 14.6 MiB in 28 packages polaris-setup-1 | polaris-setup-1 | Obtained access token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwb2xhcmlzIiwic3ViIjoicm9vdCIsImlhdCI6MTc2ODkxNTk2MiwiZXhwIjoxNzY4OTE5NTYyLCJqdGkiOiI3OTgyYzk2OS1mYmUxLTQyM2ItYTI3NS1iZGY1MTJjYmY3ZDMiLCJhY3RpdmUiOnRydWUsImNsaWVudF9pZCI6InJvb3QiLCJwcmluY2lwYWxJZCI6MSwic2NvcGUiOiJQUklOQ0lQQUxfUk9MRTpBTEwifQ.wJ0dW7jdKmNJ_qCkWNd2ImTJhuw66ZaHbnYJAIpoYUpra5rj3LvWnoNpbXw_c4bvxyFvcDwgdCZx8PuXOlF1DDSDaM3ZMMxBe852J_rHKjxomN-Fz_ndXLWgmjibjgO5SbW1qTvOqzOpDXcCgwtlrtZYHi2sL27W5HQb35ieySLT6GH6BhypBT1NbWXtOuITBBRJzqSYO12oOWi35SBu-aKinIeUQytkrdgx8ZtzIVOY4OsZzERfJaQm_IjBsPh4diXQr2teeGU-kSMIEgHuqUaa-ldD6_zwv1JxR45lYbooE8kcaIxWl4xDUM6SvcJ3feQQ92Z5j7UrQlfc_VlE3w polaris-setup-1 | STORAGE_LOCATION is set to 's3://bucket123' polaris-setup-1 | Using StorageType: S3 polaris-setup-1 | polaris-setup-1 | Creating a catalog named quickstart_catalog in realm POLARIS... polaris-setup-1 | { "catalog": { "name": "quickstart_catalog", "type": "INTERNAL", "readOnly": false, "properties": { "default-base-location": "s3://bucket123" }, "storageConfigInfo": {"storageType":"S3", "endpoint":"http://localhost:9000";, "endpointInternal":"http://rustfs:9000";, "pathStyleAccess":true} } } polaris-setup-1 | * Host polaris:8181 was resolved. polaris-setup-1 | * IPv6: (none) polaris-setup-1 | * IPv4: 172.18.0.4 polaris-setup-1 | * Trying 172.18.0.4:8181... polaris-setup-1 | * Established connection to polaris (172.18.0.4 port 8181) from 172.18.0.3 port 38416 polaris-setup-1 | * using HTTP/1.x polaris-setup-1 | > POST /api/management/v1/catalogs HTTP/1.1 polaris-setup-1 | > Host: polaris:8181 polaris-setup-1 | > User-Agent: curl/8.17.0 polaris-setup-1 | > Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJwb2xhcmlzIiwic3ViIjoicm9vdCIsImlhdCI6MTc2ODkxNTk2MiwiZXhwIjoxNzY4OTE5NTYyLCJqdGkiOiI3OTgyYzk2OS1mYmUxLTQyM2ItYTI3NS1iZGY1MTJjYmY3ZDMiLCJhY3RpdmUiOnRydWUsImNsaWVudF9pZCI6InJvb3QiLCJwcmluY2lwYWxJZCI6MSwic2NvcGUiOiJQUklOQ0lQQUxfUk9MRTpBTEwifQ.wJ0dW7jdKmNJ_qCkWNd2ImTJhuw66ZaHbnYJAIpoYUpra5rj3LvWnoNpbXw_c4bvxyFvcDwgdCZx8PuXOlF1DDSDaM3ZMMxBe852J_rHKjxomN-Fz_ndXLWgmjibjgO5SbW1qTvOqzOpDXcCgwtlrtZYHi2sL27W5HQb35ieySLT6GH6BhypBT1NbWXtOuITBBRJzqSYO12oOWi35SBu-aKinIeUQytkrdgx8ZtzIVOY4OsZzERfJaQm_IjBsPh4diXQr2teeGU-kSMIEgHuqUaa-ldD6_zwv1JxR45lYbooE8kcaIxWl4xDUM6SvcJ3feQQ92Z5j7UrQlfc_VlE3w polaris-setup-1 | > Accept: application/json polaris-setup-1 | > Content-Type: application/json polaris-setup-1 | > Polaris-Realm: POLARIS polaris-setup-1 | > Content-Length: 341 polaris-setup-1 | > polaris-setup-1 | } [341 bytes data] polaris-setup-1 | * upload completely sent off: 341 bytes polaris-1 | 2026-01-20 13:32:43,030 INFO [org.apa.pol.ser.adm.PolarisServiceImpl] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002,POLARIS] [,,,] (executor-thread-1) Created new catalog class PolarisCatalog { polaris-1 | class Catalog { polaris-1 | type: INTERNAL polaris-1 | name: quickstart_catalog polaris-1 | properties: class CatalogProperties { polaris-1 | {default-base-location=s3://bucket123} polaris-1 | defaultBaseLocation: s3://bucket123 polaris-1 | } polaris-1 | createTimestamp: 1768915963026 polaris-1 | lastUpdateTimestamp: 0 polaris-1 | entityVersion: 1 polaris-1 | storageConfigInfo: class AwsStorageConfigInfo { polaris-1 | class StorageConfigInfo { polaris-1 | storageType: S3 polaris-1 | allowedLocations: [s3://bucket123] polaris-1 | } polaris-1 | roleArn: null polaris-1 | externalId: null polaris-1 | userArn: null polaris-1 | region: null polaris-1 | endpoint: http://localhost:9000 polaris-1 | stsEndpoint: null polaris-1 | stsUnavailable: null polaris-1 | endpointInternal: http://rustfs:9000 polaris-1 | pathStyleAccess: true polaris-1 | } polaris-1 | } polaris-1 | } polaris-1 | 2026-01-20 13:32:43,035 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - root [20/Jan/2026:13:32:43 +0000] "POST /api/management/v1/catalogs HTTP/1.1" 201 354 polaris-setup-1 | < HTTP/1.1 201 Created polaris-setup-1 | < Content-Type: application/json;charset=UTF-8 polaris-setup-1 | < content-length: 354 polaris-setup-1 | < X-Request-ID: 86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000002 polaris-setup-1 | < polaris-setup-1 | { [354 bytes data] polaris-setup-1 | * Connection #0 to host polaris:8181 left intact polaris-setup-1 | {"type":"INTERNAL","name":"quickstart_catalog","properties":{"default-base-location":"s3://bucket123"},"createTimestamp":1768915963026,"lastUpdateTimestamp":0,"entityVersion":1,"storageConfigInfo":{"endpoint":"http://localhost:9000","endpointInternal":"http://rustfs:9000","pathStyleAccess":true,"storageType":"S3","allowedLocations":["s3://bucket123"]}} polaris-setup-1 | Done. polaris-setup-1 | Extra grants... polaris-setup-1 | % Total % Received % Xferd Average Speed Time Time Time Current polaris-setup-1 | Dload Upload Total Spent Left Speed polaris-1 | 2026-01-20 13:32:43,047 INFO [org.apa.pol.ser.adm.PolarisServiceImpl] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000003,POLARIS] [,,,] (executor-thread-1) Adding grant class AddGrantRequest { polaris-1 | grant: class CatalogGrant { polaris-1 | class GrantResource { polaris-1 | type: catalog polaris-1 | } polaris-1 | privilege: CATALOG_MANAGE_CONTENT polaris-1 | } polaris-1 | } to catalogRole catalog_admin in catalog quickstart_catalog polaris-1 | 2026-01-20 13:32:43,062 INFO [io.qua.htt.access-log] [86411873-6122-4fb5-8ee0-5ccba6056018_0000000000000000003,POLARIS] [,,,] (executor-thread-1) 172.18.0.3 - root [20/Jan/2026:13:32:43 +0000] "PUT /api/management/v1/catalogs/quickstart_catalog/catalog-roles/catalog_admin/grants HTTP/1.1" 201 - 100 56 0 0 100 56 0 2274 --:--:-- --:--:-- --:--:-- 2333 polaris-setup-1 | Done. polaris-setup-1 exited with code 0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
