joyhaldar commented on code in PR #4461: URL: https://github.com/apache/polaris/pull/4461#discussion_r3253911798
########## site/content/in-dev/unreleased/federation/bigquery-metastore-federation.md: ########## @@ -0,0 +1,137 @@ +--- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +title: BigQuery Metastore Federation +type: docs +weight: 706 +--- + +Polaris can federate catalog operations to a BigQuery Metastore catalog. This lets BigQuery Metastore +remain the source of truth for Iceberg table metadata while Polaris brokers access, policies, and +multi-engine connectivity. + +## Build-time enablement + +The BigQuery factory is packaged as an optional extension and is not baked into default server +builds. Include it when assembling the runtime or container images by setting the `NonRESTCatalogs` +Gradle property to include `BIGQUERY` (and any other non-REST backends you need): + +```bash +./gradlew :polaris-server:assemble :polaris-server:quarkusAppPartsBuild --rerun \ + -PNonRESTCatalogs=BIGQUERY -Dquarkus.container-image.build=true +``` + +`runtime/server/build.gradle.kts` wires the extension in only when this flag is present, so binaries +built without it will reject BigQuery federation requests. + +## Feature configuration + +After building Polaris with BigQuery Metastore support, enable the necessary feature flags in your +`application.properties` file (or equivalent configuration mechanism such as environment variables +or a Kubernetes ConfigMap): + +```properties +# Allows BIGQUERY connection type +polaris.features."SUPPORTED_CATALOG_CONNECTION_TYPES"=["BIGQUERY"] + +# Allows IMPLICIT authentication, needed for BigQuery Metastore federation +polaris.features."SUPPORTED_EXTERNAL_CATALOG_AUTHENTICATION_TYPES"=["IMPLICIT"] + +# Enables the federation feature itself +polaris.features."ENABLE_CATALOG_FEDERATION"=true +``` + +For Kubernetes deployments, add these properties to the ConfigMap mounted into the Polaris +container (typically at `/deployment/config/application.properties`). + +## Runtime requirements + +- **BigQuery API access:** The Polaris deployment must be able to reach the BigQuery API + (`bigquery.googleapis.com`) over HTTPS. +- **Authentication:** BigQuery Metastore federation only supports `IMPLICIT` authentication, meaning + Polaris uses Google Application Default Credentials resolved at process startup. Ensure the + Polaris process has valid ADC available before starting the server. +- **IAM:** The identity used by Polaris must have read access to BigQuery datasets in the target + project and read access to the GCS warehouse bucket (read/write if Polaris will commit table + metadata). + +## Creating a federated catalog + +Use the Management API to create an external catalog whose connection type is `BIGQUERY`. The +following request registers a catalog that proxies to BigQuery Metastore in the GCP project +`my-gcp-project`, using `analytics_dataset` as the default warehouse: + +```bash +curl -X POST https://<polaris-host>/management/v1/catalogs \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "type": "EXTERNAL", + "name": "analytics_bigquery", + "storageConfigInfo": { + "storageType": "GCS", + "allowedLocations": ["gs://analytics-bucket/warehouse/"] Review Comment: I have dropped `allowedLocations` from the example and verified locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
