Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
jerryshao merged PR #4746: URL: https://github.com/apache/gravitino/pull/4746 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742924088
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/HudiCatalogBackend.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.lakehouse.hudi.backend;
+
+import java.util.Map;
+import org.apache.gravitino.catalog.lakehouse.hudi.ops.HudiCatalogBackendOps;
+
+/** Base class for Hudi catalog backends. */
+public abstract class HudiCatalogBackend {
+
+ private final Type backendType;
+
+ private final HudiCatalogBackendOps catalogOps;
+
+ public abstract void initialize(Map properties);
+
+ public enum Type {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742904065
##
gradle/libs.versions.toml:
##
@@ -91,6 +91,7 @@ node-plugin = "7.0.1"
commons-cli = "1.2"
sun-activation-version = "1.2.0"
error-prone = "3.1.0"
+thrift = "0.12.0"
Review Comment:
Because HudiHMSBackend depends on HMS, so I introduced:
```
implementation(libs.hive2.metastore) {
exclude("*")
}
```
and the runtime depends on thrift
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742904065
##
gradle/libs.versions.toml:
##
@@ -91,6 +91,7 @@ node-plugin = "7.0.1"
commons-cli = "1.2"
sun-activation-version = "1.2.0"
error-prone = "3.1.0"
+thrift = "0.12.0"
Review Comment:
Because HudiHMSBackend depends on HMS, so I introduce:
```
implementation(libs.hive2.metastore) {
exclude("*")
}
```
and the runtime depends on thrift
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
FANNG1 commented on code in PR #4746: URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742862098 ## gradle/libs.versions.toml: ## @@ -91,6 +91,7 @@ node-plugin = "7.0.1" commons-cli = "1.2" sun-activation-version = "1.2.0" error-prone = "3.1.0" +thrift = "0.12.0" Review Comment: why using `thrift`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
jerryshao commented on PR #4746: URL: https://github.com/apache/gravitino/pull/4746#issuecomment-2327220101 Generally LGTM, just two minor issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
jerryshao commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742538326
##
gradle/libs.versions.toml:
##
@@ -194,6 +195,7 @@ kafka-clients = { group = "org.apache.kafka", name =
"kafka-clients", version.re
kafka = { group = "org.apache.kafka", name = "kafka_2.12", version.ref =
"kafka" }
curator-test = { group = "org.apache.curator", name = "curator-test",
version.ref = "curator"}
cglib = { group = "cglib", name = "cglib", version.ref = "cglib"}
+thrift = { group = "org.apache.thrift", name = "libthrift", version.ref =
"thrift"}
Review Comment:
You should also update the license.bin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
jerryshao commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1742536210
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/HudiCatalogBackend.java:
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.lakehouse.hudi.backend;
+
+import java.util.Map;
+import org.apache.gravitino.catalog.lakehouse.hudi.ops.HudiCatalogBackendOps;
+
+/** Base class for Hudi catalog backends. */
+public abstract class HudiCatalogBackend {
+
+ private final Type backendType;
+
+ private final HudiCatalogBackendOps catalogOps;
+
+ public abstract void initialize(Map properties);
+
+ public enum Type {
Review Comment:
Move the definition to the top of the class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1736332415
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.Schema;
+import org.apache.gravitino.SchemaChange;
+import org.apache.gravitino.catalog.hudi.backend.HudiCatalogBackend;
+import org.apache.gravitino.catalog.hudi.ops.HudiCatalogOps;
+import org.apache.gravitino.catalog.hudi.utils.CatalogUtils;
+import org.apache.gravitino.connector.CatalogInfo;
+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.SupportsSchemas;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
+import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.NonEmptySchemaException;
+import org.apache.gravitino.exceptions.SchemaAlreadyExistsException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.distributions.Distribution;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
+import org.apache.gravitino.rel.expressions.transforms.Transform;
+import org.apache.gravitino.rel.indexes.Index;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Operations for Interacting with Hudi Catalog. */
+public class HudiCatalogOperations implements CatalogOperations,
SupportsSchemas, TableCatalog {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HudiCatalogOperations.class);
+
+ private HudiCatalogOps hudiCatalogOps;
Review Comment:
rename to `HudiCatalogBackendOps`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1736330463
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.Schema;
+import org.apache.gravitino.SchemaChange;
+import org.apache.gravitino.catalog.hudi.backend.HudiCatalogBackend;
+import org.apache.gravitino.catalog.hudi.ops.HudiCatalogOps;
+import org.apache.gravitino.catalog.hudi.utils.CatalogUtils;
+import org.apache.gravitino.connector.CatalogInfo;
+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.SupportsSchemas;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
+import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.NonEmptySchemaException;
+import org.apache.gravitino.exceptions.SchemaAlreadyExistsException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.distributions.Distribution;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
+import org.apache.gravitino.rel.expressions.transforms.Transform;
+import org.apache.gravitino.rel.indexes.Index;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Operations for Interacting with Hudi Catalog. */
+public class HudiCatalogOperations implements CatalogOperations,
SupportsSchemas, TableCatalog {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HudiCatalogOperations.class);
+
+ private HudiCatalogOps hudiCatalogOps;
+
+ /**
+ * Load the Hudi Catalog Backend and initialize the Hudi Catalog Operations.
+ *
+ * @param config The configuration of this Catalog.
+ * @param info The information of this Catalog.
+ * @param propertiesMetadata The properties metadata of this Catalog.
+ * @throws RuntimeException if failed to initialize the Hudi Catalog
Operations.
+ */
+ @Override
+ public void initialize(
+ Map config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
+ throws RuntimeException {
+HudiCatalogBackend hudiCatalogBackend =
CatalogUtils.loadHudiCatalogBackend(config);
+hudiCatalogOps = hudiCatalogBackend.catalogOps();
+ }
+
+ /**
+ * Performs `listSchemas` operation on the Hudi Catalog to test the catalog
connection.
+ *
+ * @param catalogIdent the name of the catalog.
+ * @param type the type of the catalog.
+ * @param provider the provider of the catalog.
+ * @param comment the comment of the catalog.
+ * @param properties the properties of the catalog.
+ * @throws Exception if failed to run `listSchemas` operation on the Hudi
Catalog.
+ */
+ @Override
+ public void testConnection(
+ NameIdentifier catalogIdent,
+ Catalog.Type type,
+ String provider,
+ String comment,
+ Map properties)
+ throws Exception {
+try {
+ hudiCatalogOps.listSchemas(null);
Review Comment:
When the code reaches this here, `hudiCatalogOps` has already been
initialized, but sometimes the actual connection is established when performing
operations, so here use a real operation to test the connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1736320965
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiTable.java:
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import org.apache.gravitino.connector.BaseTable;
+import org.apache.gravitino.connector.TableOperations;
+
+public abstract class HudiTable extends BaseTable {
+
+ @Override
+ public TableOperations newOps() throws UnsupportedOperationException {
+throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ public abstract static class Builder extends BaseTableBuilder,
HudiTable> {
+T backendTable;
+
+public Builder withBackendTable(T backendTable) {
+ this.backendTable = backendTable;
+ return this;
+}
+
+@Override
+protected HudiTable internalBuild() {
+ return backendTable == null ? simpleBuild() :
buildFromTable(backendTable);
+}
+
+protected abstract HudiTable simpleBuild();
Review Comment:
comments added.
> Builds a simple HudiTable without a backend table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
mchades commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1736294122
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/backend/hms/HudiHMSSchema.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi.backend.hms;
+
+import org.apache.gravitino.catalog.hudi.HudiSchema;
+import org.apache.hadoop.hive.metastore.api.Database;
+
+public class HudiHMSSchema extends HudiSchema {
Review Comment:
Since Hudi doesn't provide a unified Schema class or interface, I defined
different HudiSchema to represent the properties from different backends to
preserve their unique properties. I also defined an abstract class
`org.apache.gravitino.catalog.lakehouse.hudi.HudiSchema.Builder` for
interconverting HudiSchema and backendSchema in different HudiSchema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
SteNicholas commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1735644848
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiTable.java:
##
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import org.apache.gravitino.connector.BaseTable;
+import org.apache.gravitino.connector.TableOperations;
+
+public abstract class HudiTable extends BaseTable {
+
+ @Override
+ public TableOperations newOps() throws UnsupportedOperationException {
+throw new UnsupportedOperationException("Not implemented yet");
+ }
+
+ public abstract static class Builder extends BaseTableBuilder,
HudiTable> {
+T backendTable;
+
+public Builder withBackendTable(T backendTable) {
+ this.backendTable = backendTable;
+ return this;
+}
+
+@Override
+protected HudiTable internalBuild() {
+ return backendTable == null ? simpleBuild() :
buildFromTable(backendTable);
+}
+
+protected abstract HudiTable simpleBuild();
Review Comment:
What is this interface used for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
SteNicholas commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1735640289
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.Schema;
+import org.apache.gravitino.SchemaChange;
+import org.apache.gravitino.catalog.hudi.backend.HudiCatalogBackend;
+import org.apache.gravitino.catalog.hudi.ops.HudiCatalogOps;
+import org.apache.gravitino.catalog.hudi.utils.CatalogUtils;
+import org.apache.gravitino.connector.CatalogInfo;
+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.SupportsSchemas;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
+import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.NonEmptySchemaException;
+import org.apache.gravitino.exceptions.SchemaAlreadyExistsException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.distributions.Distribution;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
+import org.apache.gravitino.rel.expressions.transforms.Transform;
+import org.apache.gravitino.rel.indexes.Index;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Operations for Interacting with Hudi Catalog. */
+public class HudiCatalogOperations implements CatalogOperations,
SupportsSchemas, TableCatalog {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HudiCatalogOperations.class);
+
+ private HudiCatalogOps hudiCatalogOps;
+
+ /**
+ * Load the Hudi Catalog Backend and initialize the Hudi Catalog Operations.
Review Comment:
```suggestion
* Loads the Hudi Catalog Backend and initialize the Hudi Catalog
Operations.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
SteNicholas commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1735642655
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.Schema;
+import org.apache.gravitino.SchemaChange;
+import org.apache.gravitino.catalog.hudi.backend.HudiCatalogBackend;
+import org.apache.gravitino.catalog.hudi.ops.HudiCatalogOps;
+import org.apache.gravitino.catalog.hudi.utils.CatalogUtils;
+import org.apache.gravitino.connector.CatalogInfo;
+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.SupportsSchemas;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
+import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.NonEmptySchemaException;
+import org.apache.gravitino.exceptions.SchemaAlreadyExistsException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.distributions.Distribution;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
+import org.apache.gravitino.rel.expressions.transforms.Transform;
+import org.apache.gravitino.rel.indexes.Index;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Operations for Interacting with Hudi Catalog. */
+public class HudiCatalogOperations implements CatalogOperations,
SupportsSchemas, TableCatalog {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HudiCatalogOperations.class);
+
+ private HudiCatalogOps hudiCatalogOps;
+
+ /**
+ * Load the Hudi Catalog Backend and initialize the Hudi Catalog Operations.
+ *
+ * @param config The configuration of this Catalog.
+ * @param info The information of this Catalog.
+ * @param propertiesMetadata The properties metadata of this Catalog.
+ * @throws RuntimeException if failed to initialize the Hudi Catalog
Operations.
+ */
+ @Override
+ public void initialize(
+ Map config, CatalogInfo info, HasPropertyMetadata
propertiesMetadata)
+ throws RuntimeException {
+HudiCatalogBackend hudiCatalogBackend =
CatalogUtils.loadHudiCatalogBackend(config);
+hudiCatalogOps = hudiCatalogBackend.catalogOps();
+ }
+
+ /**
+ * Performs `listSchemas` operation on the Hudi Catalog to test the catalog
connection.
+ *
+ * @param catalogIdent the name of the catalog.
+ * @param type the type of the catalog.
+ * @param provider the provider of the catalog.
+ * @param comment the comment of the catalog.
+ * @param properties the properties of the catalog.
+ * @throws Exception if failed to run `listSchemas` operation on the Hudi
Catalog.
+ */
+ @Override
+ public void testConnection(
+ NameIdentifier catalogIdent,
+ Catalog.Type type,
+ String provider,
+ String comment,
+ Map properties)
+ throws Exception {
+try {
+ hudiCatalogOps.listSchemas(null);
Review Comment:
Is there other solution to implement the `testConnection` for example
whether the `hudiCatalogOps` is successfully initialized?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
SteNicholas commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1735640289
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apache.gravitino.Catalog;
+import org.apache.gravitino.NameIdentifier;
+import org.apache.gravitino.Namespace;
+import org.apache.gravitino.Schema;
+import org.apache.gravitino.SchemaChange;
+import org.apache.gravitino.catalog.hudi.backend.HudiCatalogBackend;
+import org.apache.gravitino.catalog.hudi.ops.HudiCatalogOps;
+import org.apache.gravitino.catalog.hudi.utils.CatalogUtils;
+import org.apache.gravitino.connector.CatalogInfo;
+import org.apache.gravitino.connector.CatalogOperations;
+import org.apache.gravitino.connector.HasPropertyMetadata;
+import org.apache.gravitino.connector.SupportsSchemas;
+import org.apache.gravitino.exceptions.ConnectionFailedException;
+import org.apache.gravitino.exceptions.NoSuchCatalogException;
+import org.apache.gravitino.exceptions.NoSuchSchemaException;
+import org.apache.gravitino.exceptions.NoSuchTableException;
+import org.apache.gravitino.exceptions.NonEmptySchemaException;
+import org.apache.gravitino.exceptions.SchemaAlreadyExistsException;
+import org.apache.gravitino.exceptions.TableAlreadyExistsException;
+import org.apache.gravitino.rel.Column;
+import org.apache.gravitino.rel.Table;
+import org.apache.gravitino.rel.TableCatalog;
+import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.distributions.Distribution;
+import org.apache.gravitino.rel.expressions.sorts.SortOrder;
+import org.apache.gravitino.rel.expressions.transforms.Transform;
+import org.apache.gravitino.rel.indexes.Index;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Operations for Interacting with Hudi Catalog. */
+public class HudiCatalogOperations implements CatalogOperations,
SupportsSchemas, TableCatalog {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(HudiCatalogOperations.class);
+
+ private HudiCatalogOps hudiCatalogOps;
+
+ /**
+ * Load the Hudi Catalog Backend and initialize the Hudi Catalog Operations.
Review Comment:
```suggestion
* Loads the Hudi Catalog Backend and initialize the Hudi Catalog
Operations.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] [#4679] feat(hudi-catalog): Add code skeleton for hudi catalog [gravitino]
jerryshao commented on code in PR #4746:
URL: https://github.com/apache/gravitino/pull/4746#discussion_r1735587958
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/backend/hms/HudiHMSSchema.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi.backend.hms;
+
+import org.apache.gravitino.catalog.hudi.HudiSchema;
+import org.apache.hadoop.hive.metastore.api.Database;
+
+public class HudiHMSSchema extends HudiSchema {
Review Comment:
Why do we need to define different Schema for different backend?
##
catalogs/catalog-lakehouse-hudi/build.gradle.kts:
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+description = "catalog-lakehouse-iceberg"
+
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":core"))
Review Comment:
Should you exclude some jars here?
##
catalogs/catalog-lakehouse-hudi/build.gradle.kts:
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+description = "catalog-lakehouse-iceberg"
+
+plugins {
+ `maven-publish`
+ id("java")
+ id("idea")
+}
+
+dependencies {
+ implementation(project(":api"))
+ implementation(project(":core"))
+
+ implementation(libs.guava)
+ implementation(libs.hive2.metastore) {
+exclude("*")
+ }
+ implementation("org.apache.thrift:libthrift:0.12.0")
Review Comment:
Why do you directly write the dependency here, rather than define in the
version file?
##
catalogs/catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/hudi/HudiCatalogOperations.java:
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.catalog.hudi;
+
+import java.util.Map;
+import org.apac
