[
https://issues.apache.org/jira/browse/GOBBLIN-1802?focusedWorklogId=854237&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-854237
]
ASF GitHub Bot logged work on GOBBLIN-1802:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 31/Mar/23 18:04
Start Date: 31/Mar/23 18:04
Worklog Time Spent: 10m
Work Description: meethngala commented on code in PR #3663:
URL: https://github.com/apache/gobblin/pull/3663#discussion_r1154756226
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergDatasetFinder.java:
##########
@@ -98,20 +104,43 @@ public Iterator<IcebergDataset> getDatasetsIterator()
throws IOException {
return findDatasets().iterator();
}
- protected IcebergDataset createIcebergDataset(String dbName, String tblName,
IcebergCatalog icebergCatalog, Properties properties, FileSystem fs) {
- IcebergTable icebergTable = icebergCatalog.openTable(dbName, tblName);
- return new IcebergDataset(dbName, tblName, icebergTable, properties, fs);
+ /**
+ * Requires both source and destination catalogs to connect to their
respective {@link IcebergTable}
+ * Note: the destination side {@link IcebergTable} should be present before
initiating replication
Review Comment:
It was conscious decision made based on the idea that we only register table
metadata updates for existing tables. This was to keep in mind for sinks that
serve as pure secondary reads for disaster recovery purposes. I have added a
TODO in case we want to rethink on the approach and allow creating tables as
well in case they are absent
Issue Time Tracking
-------------------
Worklog Id: (was: 854237)
Time Spent: 4h (was: 3h 50m)
> Register iceberg table metadata update with destination side catalog
> --------------------------------------------------------------------
>
> Key: GOBBLIN-1802
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1802
> Project: Apache Gobblin
> Issue Type: New Feature
> Reporter: Meeth Gala
> Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
