[
https://issues.apache.org/jira/browse/GOBBLIN-1810?focusedWorklogId=856570&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-856570
]
ASF GitHub Bot logged work on GOBBLIN-1810:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 12/Apr/23 21:03
Start Date: 12/Apr/23 21:03
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on code in PR #3672:
URL: https://github.com/apache/gobblin/pull/3672#discussion_r1164651755
##########
gobblin-iceberg/src/main/java/org/apache/gobblin/iceberg/writer/IcebergMetadataWriter.java:
##########
@@ -801,12 +813,14 @@ public void flush(String dbName, String tableName) throws
IOException {
Transaction transaction = tableMetadata.transaction.get();
Map<String, String> props = tableMetadata.newProperties.or(
Maps.newHashMap(tableMetadata.lastProperties.or(getIcebergTable(tid).properties())));
- String topic = props.get(TOPIC_NAME_KEY);
+ //Set data offset range
+ setDatasetOffsetRange(tableMetadata, props);
+ String topicName = getTopicName(tid, tableMetadata);
Review Comment:
We are still fallback to the previous topic name calculated from NativeName
here in case datasetOffsetRange does not exist. Also GTE require this
datasetName to set the HDFS dataset path information
Issue Time Tracking
-------------------
Worklog Id: (was: 856570)
Time Spent: 40m (was: 0.5h)
> Support general iceberg catalog in icebergMetadataWriter
> --------------------------------------------------------
>
> Key: GOBBLIN-1810
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1810
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Now icebergMetadataWriter only supports HiveCatalog, we want to support the
> general catalog.
> Also, we want to add "avro.schema.literal" in iceberg table property
> In addition, we need to be able to change the way we calculate the Kafka
> topic name
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
