Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4318006685 > Thanks for continuous working on it, @joyhaldar! I think we are getting closer. Left some comments. Could you fix the CI failures as well? > > A few followup items(not a blocker): > > 1. Document this new feature > 2. CLI changes to support BQMS. Sorry for the delay, I will follow-up on the remaining work very soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4283048960 Thanks @joyhaldar for the change! It's very useful feature. Thanks everyone for the review. > I believe there is an option to re-run failed CI jobs without closing/reopening the PR 🤔 @dimas-b, got it. Mind sharing the option? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain merged PR #4050: URL: https://github.com/apache/polaris/pull/4050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4282947092 I believe there is an option to re-run failed CI jobs without closing/reopening the PR :thinking: +1 to merging today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4282728643 Merge is blocked by Github CI. Re-triggered. I will merge it today if there is no more feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain closed pull request #4050: Add BigQuery Metastore federation support URL: https://github.com/apache/polaris/pull/4050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4275792391 > LGTM 👍 Thanks for bearing with me 🙂 No, thank you for taking the time to thoroughly review my PR and giving me so much constructive feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3103937235
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.FederatedCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
FederatedCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+Map properties = connectionConfigInfoDpo.getProperties();
+String warehouse = properties.get("warehouse");
+if (warehouse == null || warehouse.isEmpty()) {
+ throw new IllegalArgumentException("warehouse is required for BigQuery
Metastore federation");
+}
+String projectId = properties.get("gcp.bigquery.project-id");
Review Comment:
nit: make a constant of refer to a constant from Iceberg jars?
##
spec/polaris-management-service.yml:
##
@@ -965,6 +972,16 @@ components:
warehouse:
type: string
description: The warehouse location for the hive catalog.
+
+BigQueryMetastoreConnectionConfigInfo:
+ type: object
+ description: |
+Configuration necessary for connecting to a BigQuery Metastore Catalog.
Review Comment:
nit: maybe make a note that `properties` are effectively Iceberg's BigQuery
catalog properties (for clarity)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
nandorKollar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3092960299
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
Review Comment:
Oh okay, I missed that it is used there. Makes sense, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3092819165
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
Review Comment:
Actually it's public because it's used in
`ConnectionConfigInfoDpo.fromConnectionConfigInfoModelWithSecrets()` for URI
defaulting. I kept it in the BQMS DPO since it's specific to BQMS. But let me
know if you disagree, would appreciate your inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3092809811
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.FederatedCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
FederatedCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+Map properties = connectionConfigInfoDpo.getProperties();
+String warehouse = properties.get("warehouse");
+if (warehouse == null || warehouse.isEmpty()) {
+ throw new IllegalArgumentException("warehouse is required for BigQuery
Metastore federation");
+}
+if (properties.get("gcp.bigquery.project-id") == null) {
Review Comment:
I have added empty string validation for `gcp.bigquery.project-id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3092806711
##
polaris-core/src/main/java/org/apache/polaris/core/connection/hadoop/HadoopConnectionConfigInfoDpo.java:
##
@@ -51,7 +51,8 @@ public HadoopConnectionConfigInfoDpo(
@JsonProperty(value = "serviceIdentity", required = false) @Nullable
ServiceIdentityInfoDpo serviceIdentityInfo,
@JsonProperty(value = "warehouse", required = false) @Nullable String
remoteCatalogName) {
-super(ConnectionType.HADOOP.getCode(), uri, authenticationParameters,
serviceIdentityInfo);
+super(
+ConnectionType.HADOOP.getCode(), uri, authenticationParameters,
serviceIdentityInfo, null);
Review Comment:
I have added the overloaded constructor and updated the subclasses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
nandorKollar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3091869581
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
Review Comment:
nit: I think we can keep this private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
nandorKollar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3091851946
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.FederatedCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
FederatedCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+Map properties = connectionConfigInfoDpo.getProperties();
+String warehouse = properties.get("warehouse");
+if (warehouse == null || warehouse.isEmpty()) {
+ throw new IllegalArgumentException("warehouse is required for BigQuery
Metastore federation");
+}
+if (properties.get("gcp.bigquery.project-id") == null) {
Review Comment:
Is empty string accepted for `gcp.bigquery.project-id`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
nandorKollar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3091778864
##
polaris-core/src/main/java/org/apache/polaris/core/connection/hadoop/HadoopConnectionConfigInfoDpo.java:
##
@@ -51,7 +51,8 @@ public HadoopConnectionConfigInfoDpo(
@JsonProperty(value = "serviceIdentity", required = false) @Nullable
ServiceIdentityInfoDpo serviceIdentityInfo,
@JsonProperty(value = "warehouse", required = false) @Nullable String
remoteCatalogName) {
-super(ConnectionType.HADOOP.getCode(), uri, authenticationParameters,
serviceIdentityInfo);
+super(
+ConnectionType.HADOOP.getCode(), uri, authenticationParameters,
serviceIdentityInfo, null);
Review Comment:
nit: I think we can spare adding these ugly 'null' parameters in the super
calls. We can overload the public constructor of `ConnectionConfigInfoDpo` like
this:
```
public ConnectionConfigInfoDpo(
@JsonProperty(value = "connectionTypeCode", required = true) int
connectionTypeCode,
@JsonProperty(value = "uri", required = true) @Nonnull String uri,
@JsonProperty(value = "authenticationParameters", required = true)
@Nullable
AuthenticationParametersDpo authenticationParameters,
@JsonProperty(value = "serviceIdentity", required = false)
@Nullable
ServiceIdentityInfoDpo serviceIdentity,
@JsonProperty(value = "properties", required = false) @Nullable
Map properties) {
this(connectionTypeCode, uri, authenticationParameters, serviceIdentity,
true, properties);
}
public ConnectionConfigInfoDpo(
@JsonProperty(value = "connectionTypeCode", required = true) int
connectionTypeCode,
@JsonProperty(value = "uri", required = true) @Nonnull String uri,
@JsonProperty(value = "authenticationParameters", required = true)
@Nullable
AuthenticationParametersDpo authenticationParameters,
@JsonProperty(value = "serviceIdentity", required = false) @Nullable
ServiceIdentityInfoDpo serviceIdentity) {
this(connectionTypeCode, uri, authenticationParameters, serviceIdentity,
true, null);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3090924588
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
No, I am sorry, I missed it.
I tried adding the defaulting in
`fromConnectionConfigInfoModelWithSecrets()` but it doesn't work,
`PolarisServiceImpl.validateAuthenticationParameters()` runs before conversion
and throws NPE when `authenticationParameters` is null IIUC.
Fixing that would mean changing `PolarisServiceImpl too`, which seems out of
scope for this PR.
For now I'm going with @dimas-b's suggestion, users must pass
`authenticationParameters: {authenticationType: IMPLICIT}` explicitly
([commit](https://github.com/apache/polaris/pull/4050/changes/3856a7d5a0d5f3ccd8406037614013615c5a67ce)).
Let me know if my understanding is correct. Would appreciate your guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3090685696
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
I have moved `warehouse` and `gcpProjectId` to the properties map with
validation in the factory and documented required properties in the spec in
[this
commit](https://github.com/apache/polaris/pull/4050/changes/33ae93a630e15736b29c989cc3a2b98a418ff712).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087623134
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Retracting API changes is not ideal (in a follow-up).
I'd propose to remove `gcpProjectId` and `warehouse` from OpenAPI _explicit_
properties and process corresponding Iceberg properties from the generic
`properties` bag. Also, add description to OpenAPI YAML to list required
properties.
My rationale is that it is fine to accept pass-through properties destined
to become Iceberg (federated) catalog properties
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3089414051
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
@joyhaldar, sorry to miss this one, I think we will actually need this to be
resolved. I'd suggest to add it to the method
`fromConnectionConfigInfoModelWithSecrets()`, something like this
```
case BIGQUERY:
BigQueryMetastoreConnectionConfigInfo bigqueryConfigModel =
(BigQueryMetastoreConnectionConfigInfo) connectionConfigurationModel;
// Default to IMPLICIT authentication if not provided
authenticationParameters =
bigqueryConfigModel.getAuthenticationParameters() != null
?
AuthenticationParametersDpo.fromAuthenticationParametersModelWithSecrets(
bigqueryConfigModel.getAuthenticationParameters(),
secretReferences)
: new ImplicitAuthenticationParametersDpo();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087678254
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Re: Hadoop and Hive: from my POV the same argument applies to them, but I
missed out on the review of that code when it was introduced :sweat_smile:
This is why my request in this thread is non-blocking.
However, I do not really see a reason to promote an old pattern if we're in
agreement about the rationale for generic properties in this case (hence my
request for more reviewer opinions).
--
This is an au
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087678254
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Re: Hadoop and Hive: from my POV the same argument applies to them, but I
missed out on the review of that code when it was introduced :sweat_smile:
This is why my request in this thread is non-blocking.
However, I do not really see a reason to promote an old pattern if we're in
agreement about the rationale for generic properties in this case (hence my
request for more review opinions).
--
This is an auto
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087678254
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Re: Hadoop and Hive: from my POV the same argument applies to them, but I
missed out on the review of that code when I was introduced :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact I
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087623134
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Retracting API changes is not ideal (in a follow-up).
I'd propose to remove `gcpProjectId` and `warehouse` from OpenAPI _explicit_
properties and process corresponding Iceberg properties from the generic
`properties` bag. Also, add description to OpenAPI YAML to list required
properties.
My rationale is that it is fine to accept pass-through properties destined
to become Iceberg (federated) catalog properties
Re: [PR] Add BigQuery Metastore federation support [polaris]
jbonofre commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087637094
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
@dimas-b proposal works for me, as soon as clear documented. That said, I'm
also fine to keep them as it's "bigquery" specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
us
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087623134
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Retracting API changes is not ideal (in a follow-up).
I'd propose to remove `gcpProjectId` and `warehouse` from OpenAPI _explicit_
properties and process corresponding Iceberg properties from the generic
`properties` bag. Also, add description to OpenAPI YAML to list required
properties.
I wonder what other reviewers think about this.
--
This is an automated message from the Apache Git Service.
To respond
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087582707
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Can I do this in a follow-up PR?
I'm trying to figure out the right pattern here,
[Hadoop](https://github.com/apache/polaris/blob/main/extensions/federation/hadoop/src/main/java/org/apache/polaris/extensions/federation/hadoop/HadoopFederatedCatalogFactory.java)
and
[Hive](https://github.com/apache/polaris/blob/main/extensions/federation/hive/src/main/java/org/apache/polaris/extensions/federation/hive/HiveFederatedCa
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087270417
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
From my POV this creates a skew in the Polaris API, which becomes dependent
on particular Iceberg java classes.
Conceptually, this class merely passes some user-defined properties to the
Iceberg catalog impl. This is fine. However, do we have to elevate a sub-set of
those properties to explicit Polaris OpenAPI properties?.. I'm not sure.
I believe keeping the Polaris API generic and uniform for all federated
c
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3087270417
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
From my POV this creates a skew in the Polaris API, which becomes dependent
on particular Iceberg java classes.
Conceptually, this class merely passes some user-defined properties to the
Iceberg catalog impl. This is fine. However, do we have to elevate a sub-set of
those properties to explicit Polaris OpenAPI properties?.. I'm not sure.
I believe keeping the Polaris API generic and uniform for all federated
c
Re: [PR] Add BigQuery Metastore federation support [polaris]
jbonofre commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3084903677
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
I can help on the LICENSE/NOTICE. I'm fine to do it in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3084642075
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
These are required fields in the spec. `projectId` is validated as required
in
[BigQueryProperties.java](https://github.com/apache/iceberg/blob/9a939d68358de9dac2c6ba9b236b675ebe477490/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryProperties.java#L104-L106),
and `warehouse` is validated in
[BigQueryMetastoreCatalog.java](https://github.com/apache/iceberg/blob/9a939d68358de9dac2c6ba9b236b675ebe477490/bigqu
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3081047722
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "properties", required = false) @Nullable
+ Map properties,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(
+ConnectionType.BIGQUERY.getCode(),
+uri,
+authenticationParameters,
+serviceIdentity,
+properties);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+ }
+
+ public String getWarehouse() {
+return warehouse;
+ }
+
+ public String getGcpProjectId() {
+return gcpProjectId;
+ }
+
+ @Override
+ public String toString() {
+return MoreObjects.toStringHelper(this)
+.add("connectionTypeCode", getConnectionTypeCode())
+.add("uri", getUri())
+.add("warehouse", warehouse)
+.add("gcpProjectId", gcpProjectId)
+.add(
+"authenticationParameters",
+getAuthenticationParameters() != null
+? getAuthenticationParameters().toString()
+: "null")
+.toString();
+ }
+
+ @Override
+ public @Nonnull Map asIcebergCatalogProperties(
+ PolarisCredentialManager polarisCredentialManager) {
+HashMap properties = new HashMap<>();
+
+// Required properties for BigQueryMetastoreCatalog
+properties.put(GCP_BIGQUERY_PROJECT_ID, gcpProjectId);
+properties.put(CatalogProperties.WAREHOUSE_LOCATION, warehouse);
Review Comment:
Why not use the generic `getProperties()` map (line 108) for these parameter
too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3078827107 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,27 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The warehouse location for the BigQuery Metastore Catalog +gcpProjectId: + type: string + description: The GCP project ID +properties: + type: object + additionalProperties: +type: string +description: Additional catalog properties Review Comment: I have moved `properties` to parent `ConnectionConfigInfo` so all federation types can use it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3078821789 ## extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java: ## @@ -65,6 +65,8 @@ public Catalog createCatalog( catalogProperties != null ? catalogProperties : Map.of(), connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager)); +// Credentials are resolved via Google Application Default Credentials (ADC). +// GCS storage operations use the same ADC credentials. Review Comment: Updated the comment to say `by default`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3041749275 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,27 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The warehouse location for the BigQuery Metastore Catalog +gcpProjectId: + type: string + description: The GCP project ID +properties: + type: object + additionalProperties: +type: string +description: Additional catalog properties Review Comment: Yes, the more I think about it the more I like the idea of supporting general `properties` inside `ConnectionConfigInfo`. That REST API change should be able to support both BigQuery Catalog federation and #3729 (CC: @PhillHenry). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3041785575 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,27 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The warehouse location for the BigQuery Metastore Catalog +gcpProjectId: + type: string + description: The GCP project ID +properties: + type: object + additionalProperties: +type: string +description: Additional catalog properties Review Comment: Here is another use case that justifies it. A client for the remote catalog may support more properties than those defined in the spec. Without a freeform `properties` field, we would need to keep updating the spec to accommodate properties already supported by existing clients (for example `HMS` or `BQMS`), which is both unnecessary and heavy. With properties, users can simply update their federated catalog in the metastore instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3040722087 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,27 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The warehouse location for the BigQuery Metastore Catalog +gcpProjectId: + type: string + description: The GCP project ID +properties: + type: object + additionalProperties: +type: string +description: Additional catalog properties Review Comment: Thanks for adding this. Thinking more broadly, this `properties` would be useful for other federated catalog type like HMS. I'd proposed to move it up to its parent `ConnectionConfigInfo`, so that all subtypes can benefit from it. WDYT? cc @XJDKC @HonahX @singhpk234 @dimas-b -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3034300574
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
Agreed, the point is even we want a default value for better UX. This is not
the place to handle the defaulting logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3034184491
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
I believe declaring the `IMPLICIT` authentication explicitly 😉 is actually
good. It increases clarity about the intended catalog behaviour. It is also
more robust in case of future API and backend code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3034162329
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
Sounds good. Resolving this comment thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3034154418
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
Review Comment:
No worries. Let's keep existing pattern.
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -65,6 +65,8 @@ public Catalog createCatalog(
catalogProperties != null ? catalogProperties : Map.of(),
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+// Credentials are resolved via Google Application Default Credentials
(ADC).
+// GCS storage operations use the same ADC credentials.
Review Comment:
The second statement is not 100% accurate. In principle a user can override
GCS storage credentials:
https://github.com/apache/polaris/blob/a6ca7cb2a27eb685522c8dc09d15217683714fde/runtime/service/src/main/java/org/apache/polaris/service/storage/StorageConfiguration.java#L151
I'd propose adding `... by default`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3033215658
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
Done. Added a comment explaining the ADC credential behavior in [this
commit](https://github.com/apache/polaris/pull/4050/changes/cef1b0fd0976fc313118df5d96e1df87c37e89b3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3033111375
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
Reverted to optional for now. Making it unconditional was breaking CI due to
LICENSE/NOTICE requirements for the new BigQuery dependencies. I can make it
unconditional once the license entries are sorted out, wanted to unblock the PR
first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4183745192 > I have two major questions/"concerns": > > 1. As we introduce new dependencies (for BigQuery and transitive), the LICENSE/NOTICE should be updated according to the second point. > 2. The question is: do we ship bigquery metastore federation by default in the server ? If yes, the LICENSE/NOTICE of the server should be updated. If no, no need to update it. Made it optional for now to unblock CI. Will follow up on LICENSE/NOTICE updates separately if we decide to ship it by default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3030637914
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
I think we will need an `AuthenticationType` regardless of how BigQuery
picks up credentials. In this case, it should be `IMPLICIT`(I agreed with line
58 here). However, from a syntax perspective, we may not want users to specify
it explicitly. We could default it when the `AuthenticationType` is missing.
That said, I would not recommend handling this defaulting logic here. The
defaulting to `IMPLICIT` should happen elsewhere. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3029539760
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation =
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3029534236
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
Review Comment:
Thank you for the review. These fields are now part of the [properties
bag](https://github.com/apache/polaris/pull/4050#discussion_r3029516972).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3029516972 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,44 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The GCS bucket path for the catalog (e.g., gs://bucket-name) +gcpProjectId: + type: string + description: The GCP project ID +gcpLocation: + type: string + description: The GCP location (default = us) +listAllTables: + type: boolean + description: Whether to list all tables (default = true) +impersonateServiceAccount: + type: string + description: Service account email to impersonate +impersonateLifetimeSeconds: + type: integer + description: Token lifetime in seconds for impersonation (default = 3600) +impersonateScopes: + type: array + items: +type: string + description: OAuth scopes for impersonation (default = cloud-platform) +impersonateDelegates: + type: array + items: +type: string + description: Delegation chain for impersonation Review Comment: Done. Refactored to use a properties bag. Only `warehouse` and `gcpProjectId` remain as required fields, everything else goes into the map. Thank you for your help. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3025936742 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,44 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The GCS bucket path for the catalog (e.g., gs://bucket-name) +gcpProjectId: + type: string + description: The GCP project ID +gcpLocation: + type: string + description: The GCP location (default = us) +listAllTables: + type: boolean + description: Whether to list all tables (default = true) +impersonateServiceAccount: + type: string + description: Service account email to impersonate +impersonateLifetimeSeconds: + type: integer + description: Token lifetime in seconds for impersonation (default = 3600) +impersonateScopes: + type: array + items: +type: string + description: OAuth scopes for impersonation (default = cloud-platform) +impersonateDelegates: + type: array + items: +type: string + description: Delegation chain for impersonation Review Comment: You are right, I will work on addressing this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3025896082
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
Review Comment:
The property keys in Iceberg's
[BigQueryProperties](https://github.com/apache/iceberg/blob/main/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryProperties.java)
are package-private, so I had to duplicate them here. I can add a comment
referencing the Iceberg source. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3025855618
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
Review Comment:
The existing `ConnectionConfigInfoDpo` classes (Hadoop, Hive) don't use
`@PolarisImmutable`. I can try to refactor and use it if you think it's
worthwhile. Would appreciate your inputs before I proceed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3025792008
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
BigQuery always uses [Application Default
Credentials](https://docs.cloud.google.com/docs/authentication/application-default-credentials)
or the
[GOOGLE_APPLICATION_CREDENTIALS](https://docs.cloud.google.com/docs/authentication/application-default-credentials#GAC)
environment variable to automatically pick up the user's credentials or a JSON
file pointing to their Service Account. Making `authenticationParameters`
required would just force users to type IMPLICIT for no reason IMO, but please
let me know what you think, or please let me know if you think I am incorrect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
jbonofre commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3020458241
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation =
Re: [PR] Add BigQuery Metastore federation support [polaris]
XJDKC commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3019866472
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
Review Comment:
For these property keys, should we just use the iceberg sdk's properties?
https://github.com/apache/iceberg/blob/main/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryProperties.java
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+i
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3019726630
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
Thanks for the explanation, @joyhaldar. Can we comment out this "hidden"
behavior somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3019594370
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo != null
Review Comment:
Same concern here. Should we fail fast when `authenticationParametersDpo` is
null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyrain commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3019156955 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,44 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The GCS bucket path for the catalog (e.g., gs://bucket-name) +gcpProjectId: + type: string + description: The GCP project ID +gcpLocation: + type: string + description: The GCP location (default = us) +listAllTables: + type: boolean + description: Whether to list all tables (default = true) +impersonateServiceAccount: + type: string + description: Service account email to impersonate +impersonateLifetimeSeconds: + type: integer + description: Token lifetime in seconds for impersonation (default = 3600) +impersonateScopes: + type: array + items: +type: string + description: OAuth scopes for impersonation (default = cloud-platform) +impersonateDelegates: + type: array + items: +type: string + description: Delegation chain for impersonation Review Comment: How often does the BQMS config surface change? For example, when new fields are introduced or existing fields are updated. Locking every field into the spec creates a maintenance burden, since each upstream change requires a Polaris spec PR, client regeneration, and a new release. Instead, we could consider using a bag of open properties to keep the model more flexible. ``` properties: type: object additionalProperties: type: string ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3017795685
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
Review Comment:
Just wondering: Could this class be `@PolarisImmutable`? I guess it might
save a good chunk of boiler-plate code 🤔
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authen
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3015927535
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
Done. Added null check here. If `authenticationParameters` is null, BigQuery
uses ADC by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4161986291 > I’m aligned with the direction here overall. I think there are two high-priority issues worth tightening before this moves forward: > > 1. The current BIGQUERY auth contract does not look aligned yet. IIUC, the API/model path still allows `authenticationParameters` to be omitted, while the runtime only supports `IMPLICIT` auth and later dereferences that field as if it were always present. That feels like more than a local null-safety issue, the boundary contract still looks looser than the implementation. > 2. Since this adds a new federation connector across OpenAPI, model conversion, and runtime wiring, I think we should have at least one round-trip/ wiring test for `BIGQUERY` before the contract settles. > > 2 cents: this PR also seems like a useful point to clarify whether federation here is aiming for a common Polaris contract over native Iceberg catalogs, or for backend-specific adapters behind one REST surface. I think that answer affects how strict/explicit the connector contract should be. Thank you for the detailed review. I have tried to address both points: 1. Added null checks in the factory and `asConnectionConfigInfoModel` to handle omitted `authenticationParameters`, which defaults to ADC. 2. Added `testBigQueryMetastoreConnectionConfig` in `ConnectionConfigInfoDpoTest` for round-trip validation. Please let me know if you'd like any changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3015286140
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
Yes, that's my understanding also. Both BigQuery Metastore and GCS storage
will use the same ADC credentials, or impersonated SA if configured. Thank you
for clarifying the full picture. Sorry for the late reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r3015293979 ## spec/polaris-management-service.yml: ## @@ -965,6 +967,44 @@ components: warehouse: type: string description: The warehouse location for the hive catalog. + +BigQueryMetastoreConnectionConfigInfo: + type: object + description: Configuration necessary for connecting to a BigQuery Metastore Catalog + allOf: +- $ref: '#/components/schemas/ConnectionConfigInfo' + properties: +warehouse: + type: string + description: The GCS bucket path for the catalog (e.g., gs://bucket-name) +gcpProjectId: + type: string + description: The GCP project ID +gcpLocation: + type: string + description: The GCP location (default = us) +listAllTables: + type: boolean + description: Whether to list all tables (default = true) +impersonateServiceAccount: + type: string + description: Service account email to impersonate +impersonateLifetimeSeconds: + type: integer + description: Token lifetime in seconds for impersonation (default = 3600) +impersonateScopes: + type: array + items: +type: string + description: OAuth scopes for impersonation (default = cloud-platform) +impersonateDelegates: + type: array + items: +type: string + description: Delegation chain for impersonation + required: Review Comment: Thank you for the feedback. I have added null checks in both `BigQueryMetastoreFederatedCatalogFactory` and `asConnectionConfigInfoModel`. If `authenticationParameters` is null, BigQuery uses ADC by default. Please let me know if you'd like me to change the approach. Sorry for the late reply. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3015286140
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
Yes, that's my understanding also. Both BigQuery Metastore and GCS storage
will use the same ADC credentials (or impersonated SA if configured). Thank you
for clarifying the full picture. Sorry for the late reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3014902671
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation =
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3014902671
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation =
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r3014891716
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation =
Re: [PR] Add BigQuery Metastore federation support [polaris]
flyingImer commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2996622689
##
spec/polaris-management-service.yml:
##
@@ -965,6 +967,44 @@ components:
warehouse:
type: string
description: The warehouse location for the hive catalog.
+
+BigQueryMetastoreConnectionConfigInfo:
+ type: object
+ description: Configuration necessary for connecting to a BigQuery
Metastore Catalog
+ allOf:
+- $ref: '#/components/schemas/ConnectionConfigInfo'
+ properties:
+warehouse:
+ type: string
+ description: The GCS bucket path for the catalog (e.g.,
gs://bucket-name)
+gcpProjectId:
+ type: string
+ description: The GCP project ID
+gcpLocation:
+ type: string
+ description: The GCP location (default = us)
+listAllTables:
+ type: boolean
+ description: Whether to list all tables (default = true)
+impersonateServiceAccount:
+ type: string
+ description: Service account email to impersonate
+impersonateLifetimeSeconds:
+ type: integer
+ description: Token lifetime in seconds for impersonation (default =
3600)
+impersonateScopes:
+ type: array
+ items:
+type: string
+ description: OAuth scopes for impersonation (default =
cloud-platform)
+impersonateDelegates:
+ type: array
+ items:
+type: string
+ description: Delegation chain for impersonation
+ required:
Review Comment:
IIUC, `BIGQUERY` can still omit `authenticationParameters` at the API
boundary, but the current implementation only supports `IMPLICIT` auth and
later code paths assume the field is present. Would it make sense to tighten
the BIGQUERY contract here instead, so the boundary reflects what the runtime
actually supports today?
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
IIUC, This path seems to rely on `authenticationParameters` being present,
while the public BIGQUERY contract still appears to allow it to be omitted. I
wonder if that mismatch should be resolve
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2995732243
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
So, if storage is also in GCP, the BigQuery connection and storage
connections will have to use the same credentials, effectively (in current
Polaris code).
It's not a big deal ATM, I think... just trying to understand the full
picture :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2995781653
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
Note: IIRC, Hive federation is not even bundled by default:
https://github.com/apache/polaris/blob/b67e1a594f740c580fc95b9d7a6cb78810bf6aeb/runtime/server/build.gradle.kts#L42-L43
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2995769841
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
I'd say let's make new code robust immediately and fix old code later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2995862060
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ public static final String DEFAULT_URI = "https://bigquery.googleapis.com";;
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation = g
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r2995757639 ## spec/polaris-management-service.yml: ## @@ -933,6 +934,7 @@ components: ICEBERG_REST: "#/components/schemas/IcebergRestConnectionConfigInfo" HADOOP: "#/components/schemas/HadoopConnectionConfigInfo" HIVE: "#/components/schemas/HiveConnectionConfigInfo" + BIGQUERY: "#/components/schemas/BigQueryMetastoreConnectionConfigInfo" Review Comment: I opened https://lists.apache.org/thread/m05xm7szd7znrm9yos1rnld5ljx04y0h specifically for community awareness of REST API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2995732243
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
So, if storage is also in GCP, the BigQuary connection and storage
connections will have to use the same credentials, effectively (in current
Polaris code).
It's not a big deal ATM, I think... just trying to understand the full
picture :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2992625114
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation = gcpLocation;
+this.listAllTables = listAllTables;
+this.impersonateServic
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2992625114
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+import org.apache.polaris.core.credentials.connection.ConnectionCredentials;
+import org.apache.polaris.core.identity.dpo.ServiceIdentityInfoDpo;
+import org.apache.polaris.core.identity.provider.ServiceIdentityProvider;
+
+/**
+ * The internal persistence-object counterpart to {@link
+ * org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo}
defined in the API
+ * model.
+ */
+public class BigQueryMetastoreConnectionConfigInfoDpo extends
ConnectionConfigInfoDpo {
+
+ // BigQuery Metastore catalog property keys
+ private static final String GCP_BIGQUERY_PROJECT_ID =
"gcp.bigquery.project-id";
+ private static final String GCP_BIGQUERY_LOCATION = "gcp.bigquery.location";
+ private static final String GCP_BIGQUERY_LIST_ALL_TABLES =
"gcp.bigquery.list-all-tables";
+ private static final String IMPERSONATE_SERVICE_ACCOUNT =
+ "gcp.bigquery.impersonate.service-account";
+ private static final String IMPERSONATE_LIFETIME_SECONDS =
+ "gcp.bigquery.impersonate.lifetime-seconds";
+ private static final String IMPERSONATE_SCOPES =
"gcp.bigquery.impersonate.scopes";
+ private static final String IMPERSONATE_DELEGATES =
"gcp.bigquery.impersonate.delegates";
+
+ private final String warehouse;
+ private final String gcpProjectId;
+ private final String gcpLocation;
+ private final Boolean listAllTables;
+ private final String impersonateServiceAccount;
+ private final Integer impersonateLifetimeSeconds;
+ private final List impersonateScopes;
+ private final List impersonateDelegates;
+
+ public BigQueryMetastoreConnectionConfigInfoDpo(
+ @JsonProperty(value = "uri", required = true) @Nonnull String uri,
+ @JsonProperty(value = "authenticationParameters", required = false)
@Nullable
+ AuthenticationParametersDpo authenticationParameters,
+ @JsonProperty(value = "warehouse", required = true) @Nonnull String
warehouse,
+ @JsonProperty(value = "gcpProjectId", required = true) @Nonnull String
gcpProjectId,
+ @JsonProperty(value = "gcpLocation", required = false) @Nullable String
gcpLocation,
+ @JsonProperty(value = "listAllTables", required = false) @Nullable
Boolean listAllTables,
+ @JsonProperty(value = "impersonateServiceAccount", required = false)
@Nullable
+ String impersonateServiceAccount,
+ @JsonProperty(value = "impersonateLifetimeSeconds", required = false)
@Nullable
+ Integer impersonateLifetimeSeconds,
+ @JsonProperty(value = "impersonateScopes", required = false) @Nullable
+ List impersonateScopes,
+ @JsonProperty(value = "impersonateDelegates", required = false) @Nullable
+ List impersonateDelegates,
+ @JsonProperty(value = "serviceIdentity", required = false) @Nullable
+ ServiceIdentityInfoDpo serviceIdentity) {
+super(ConnectionType.BIGQUERY.getCode(), uri, authenticationParameters,
serviceIdentity);
+this.warehouse = warehouse;
+this.gcpProjectId = gcpProjectId;
+this.gcpLocation = gcpLocation;
+this.listAllTables = listAllTables;
+this.impersonateServic
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r2992615984 ## polaris-core/src/main/java/org/apache/polaris/core/connection/ConnectionConfigInfoDpo.java: ## @@ -210,6 +213,30 @@ public static ConnectionConfigInfoDpo fromConnectionConfigInfoModelWithSecrets( hiveConfigModel.getWarehouse(), null /*Service Identity Info*/); break; + case BIGQUERY: +BigQueryMetastoreConnectionConfigInfo bigqueryConfigModel = +(BigQueryMetastoreConnectionConfigInfo) connectionConfigurationModel; +authenticationParameters = + AuthenticationParametersDpo.fromAuthenticationParametersModelWithSecrets( +bigqueryConfigModel.getAuthenticationParameters(), secretReferences); +String bigqueryUri = +bigqueryConfigModel.getUri() != null Review Comment: Done. I have moved the default URI to a public constant in `BigQueryMetastoreConnectionConfigInfoDpo`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2992573311
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
Thank you for your review JB.
I copied this pattern from
[HiveFederatedCatalogFactory](https://github.com/apache/polaris/blob/49766cca72e6a21ce21cc002a571d485a4586f44/extensions/federation/hive/src/main/java/org/apache/polaris/extensions/federation/hive/HiveFederatedCatalogFactory.java#L49)
and
[HadoopFederatedCatalogFactory](https://github.com/apache/polaris/blob/49766cca72e6a21ce21cc002a571d485a4586f44/extensions/federation/hadoop/src/main/java/org/apache/polaris/extensions/federation/hadoop/HadoopFederatedCatalogFactory.java#L50)
which also don't have null checks. Should I add the null check here and open a
separate PR to fix Hive and Hadoop as well? Or keep it consistent with existing
code for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
jbonofre commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2992444216
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/**
+ * Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration.
+ */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
Review Comment:
Should we include a `null` check here ?
If `connectionConfigInfoDpo.getAuthenticationParameters()` returns `null`,
then `authenticationParametersDpo.getAuthenticationTypeCode()` will throw NPE.
Since the constructor marks `authenticationParameters` as `required = false`
I guess it can happen.
##
polaris-core/src/main/java/org/apache/polaris/core/connection/bigquery/BigQueryMetastoreConnectionConfigInfoDpo.java:
##
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.polaris.core.connection.bigquery;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.MoreObjects;
+import jakarta.annotation.Nonnull;
+import jakarta.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.CatalogProperties;
+import
org.apache.polaris.core.admin.model.BigQueryMetastoreConnectionConfigInfo;
+import org.apache.polaris.core.admin.model.ConnectionConfigInfo;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import org.apache.polaris.core.credentials.Pol
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2992352934
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2989721049
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
Yes, please go with an unconditional dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4124166219 CI failure is unrelated to this PR, MongoDB testcontainer timeout if I understand correctly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r2985519365 ## spec/polaris-management-service.yml: ## @@ -933,6 +934,7 @@ components: ICEBERG_REST: "#/components/schemas/IcebergRestConnectionConfigInfo" HADOOP: "#/components/schemas/HadoopConnectionConfigInfo" HIVE: "#/components/schemas/HiveConnectionConfigInfo" + BIGQUERY: "#/components/schemas/BigQueryMetastoreConnectionConfigInfo" Review Comment: Thank you for your review Dmitri. I have an existing dev ML thread for this. Replied with the PR link: https://lists.apache.org/thread/n3hh5s1zxn6yn7cbowfgf8p029z6m7g1 Would you like me to start a new discussion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2985546207
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
Thank you for your review Dmitri.
BigQuery Metastore uses Application Default Credentials via Iceberg's
[BigQueryProperties](https://github.com/apache/iceberg/blob/main/bigquery/src/main/java/org/apache/iceberg/gcp/bigquery/BigQueryProperties.java#L146-L175).
Credentials are resolved automatically from the environment,
`GOOGLE_APPLICATION_CREDENTIALS`, `gcloud auth`, or attached Service Account.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2985527287
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
Should I remove the conditional statement and include it by default in the
standard distribution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on code in PR #4050: URL: https://github.com/apache/polaris/pull/4050#discussion_r2985519365 ## spec/polaris-management-service.yml: ## @@ -933,6 +934,7 @@ components: ICEBERG_REST: "#/components/schemas/IcebergRestConnectionConfigInfo" HADOOP: "#/components/schemas/HadoopConnectionConfigInfo" HIVE: "#/components/schemas/HiveConnectionConfigInfo" + BIGQUERY: "#/components/schemas/BigQueryMetastoreConnectionConfigInfo" Review Comment: I have an existing dev ML thread for this. Replied with the PR link: https://lists.apache.org/thread/n3hh5s1zxn6yn7cbowfgf8p029z6m7g1 Would you like me to start a new discussion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
joyhaldar commented on PR #4050: URL: https://github.com/apache/polaris/pull/4050#issuecomment-4122887163 > Thanks for your contribution, @joyhaldar ! The PR LGTM 👍 Still, I believe we need to open a `dev` discussion for it before merging as it includes REST API changes. Thank you for your review @dimas-b. I have an existing dev ML thread for this. Replied with the PR link: https://lists.apache.org/thread/n3hh5s1zxn6yn7cbowfgf8p029z6m7g1 Would you like me to start a new discussion? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2984649679
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
From my POV it seems reasonable to include it by default in Apache Polaris.
Downstream builds can make their own decisions since the impl. is
well-isolated in terms of build modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
nandorKollar commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2984436088
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
I think - unlike with Hadoop and Hive catalog - this doesn't bring in any
new direct dependency, which we have concern with. We should be good to include
this extension in the standard distribution no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2982796252
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
How / where does it get BigQuery connection credentials?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Add BigQuery Metastore federation support [polaris]
dimas-b commented on code in PR #4050:
URL: https://github.com/apache/polaris/pull/4050#discussion_r2982783639
##
runtime/server/build.gradle.kts:
##
@@ -43,6 +43,10 @@ dependencies {
runtimeOnly(project(":polaris-extensions-federation-hive"))
}
+ if ((project.findProperty("NonRESTCatalogs") as
String?)?.contains("BIGQUERY") == true) {
+runtimeOnly(project(":polaris-extensions-federation-bigquery"))
Review Comment:
This means `BIGQUERY` will NOT be included into the standard Apache Polaris
distributions... Is that intended?
##
extensions/federation/bigquery/src/main/java/org/apache/polaris/extensions/federation/bigquery/BigQueryMetastoreFederatedCatalogFactory.java:
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.polaris.extensions.federation.bigquery;
+
+import io.smallrye.common.annotation.Identifier;
+import jakarta.enterprise.context.ApplicationScoped;
+import java.util.Map;
+import org.apache.iceberg.catalog.Catalog;
+import org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog;
+import org.apache.iceberg.rest.RESTUtil;
+import org.apache.polaris.core.catalog.ExternalCatalogFactory;
+import org.apache.polaris.core.catalog.GenericTableCatalog;
+import org.apache.polaris.core.connection.AuthenticationParametersDpo;
+import org.apache.polaris.core.connection.AuthenticationType;
+import org.apache.polaris.core.connection.ConnectionConfigInfoDpo;
+import org.apache.polaris.core.connection.ConnectionType;
+import
org.apache.polaris.core.connection.bigquery.BigQueryMetastoreConnectionConfigInfoDpo;
+import org.apache.polaris.core.credentials.PolarisCredentialManager;
+
+/** Factory class for creating a BigQuery Metastore catalog handle based on
connection configuration. */
+@ApplicationScoped
+@Identifier(ConnectionType.BIGQUERY_FACTORY_IDENTIFIER)
+public class BigQueryMetastoreFederatedCatalogFactory implements
ExternalCatalogFactory {
+
+ @Override
+ public Catalog createCatalog(
+ ConnectionConfigInfoDpo connectionConfigInfoDpo,
+ PolarisCredentialManager polarisCredentialManager,
+ Map catalogProperties) {
+
+// Currently, Polaris supports BigQuery Metastore federation only via
IMPLICIT authentication.
+// Hence, prior to initializing the configuration, ensure that the catalog
uses
+// IMPLICIT authentication.
+AuthenticationParametersDpo authenticationParametersDpo =
+connectionConfigInfoDpo.getAuthenticationParameters();
+if (authenticationParametersDpo.getAuthenticationTypeCode()
+!= AuthenticationType.IMPLICIT.getCode()) {
+ throw new IllegalStateException(
+ "BigQuery Metastore federation only supports IMPLICIT
authentication.");
+}
+
+String warehouse =
+((BigQueryMetastoreConnectionConfigInfoDpo)
connectionConfigInfoDpo).getWarehouse();
+Map mergedProperties =
+RESTUtil.merge(
+catalogProperties != null ? catalogProperties : Map.of(),
+
connectionConfigInfoDpo.asIcebergCatalogProperties(polarisCredentialManager));
+
+BigQueryMetastoreCatalog bigQueryMetastoreCatalog = new
BigQueryMetastoreCatalog();
+bigQueryMetastoreCatalog.initialize(warehouse, mergedProperties);
Review Comment:
How / where do it get BigQuery connection credentials?
##
spec/polaris-management-service.yml:
##
@@ -933,6 +934,7 @@ components:
ICEBERG_REST: "#/components/schemas/IcebergRestConnectionConfigInfo"
HADOOP: "#/components/schemas/HadoopConnectionConfigInfo"
HIVE: "#/components/schemas/HiveConnectionConfigInfo"
+ BIGQUERY:
"#/components/schemas/BigQueryMetastoreConnectionConfigInfo"
Review Comment:
Please open a `dev` ML discussion for this (it's customary for all REST API
changes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
