meethngala commented on code in PR #3643:
URL: https://github.com/apache/gobblin/pull/3643#discussion_r1115283314
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergCatalog.java:
##########
@@ -17,10 +17,25 @@
package org.apache.gobblin.data.management.copy.iceberg;
+import org.apache.iceberg.catalog.Catalog;
+
/**
* Any catalog from which to access {@link IcebergTable}s.
*/
public interface IcebergCatalog {
IcebergTable openTable(String dbName, String tableName);
+ String getCatalogUri();
+
+ /**
+ * Adding a sub interface to help us provide an association between {@link
Catalog} and {@link IcebergCatalog}.
+ * This helps us resolve to the Catalog to its concrete implementation class
+ * Primarily needed to access `newTableOps` method which only certain {@link
Catalog} derived classes open for public access
+ */
+ interface CatalogSpecifier {
Review Comment:
I agree with using `CatalogUtils.loadCatalog()` method while creating the
Catalogs based on their impl that's provided as arg and we are also using that
in the `IcebergCatalogFactory`. We make use of `CatalogUtils` itself for
loading catalogs based on their impl provided as part of the catalog specifier
which taken as a config.
We have `IcebergCatalog` interface to provide us with methods that will
enable us to work with `IcebergTable` and translate them to gobblin constructs
for distcp. Regarding, the `IcebergMetadataWriter` yes, we will use
`CatalogUtils.loadCatalog()` provided by the Iceberg open-source project.
@ZihanLi58 can you please elaborate on what you do mean by original
implementation about specifying the IcebergCatalog. Because, originally, it was
hard-coded to a `HiveCatalog` and to avoid that and provide a pairing between
the catalogs in gobblin and iceberg world we have this specifier!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
