singhpk234 commented on code in PR #14525:
URL: https://github.com/apache/iceberg/pull/14525#discussion_r2501947197
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java:
##########
@@ -206,7 +206,15 @@ private void commitToTable(
String branch =
config.tableConfig(tableIdentifier.toString()).commitBranch();
+ // Control topic partition offsets may include a subset of partition ids
if there were no
+ // records for other partitions. Merge the updated topic partitions with
the last committed
+ // offsets.
Map<Integer, Long> committedOffsets = lastCommittedOffsetsForTable(table,
branch);
+ Map<Integer, Long> mergedOffsets =
+ Stream.of(committedOffsets, controlTopicOffsets)
+ .flatMap(map -> map.entrySet().stream())
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,
Long::max));
Review Comment:
IIUC this will merge both maps and in case of matching keys take the max Val
? wondering this case:
Lets say my committed offset :
[0, 100] [2, 250]
my control topic offset :
[1, 150] [2, 200]
my output [0, 100] [1,150] [2, 250], but i think we did went back as we [2,
200] in control offset ? is this check flagged already somewhere ?
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Coordinator.java:
##########
@@ -206,7 +206,15 @@ private void commitToTable(
String branch =
config.tableConfig(tableIdentifier.toString()).commitBranch();
+ // Control topic partition offsets may include a subset of partition ids
if there were no
+ // records for other partitions. Merge the updated topic partitions with
the last committed
+ // offsets.
Map<Integer, Long> committedOffsets = lastCommittedOffsetsForTable(table,
branch);
+ Map<Integer, Long> mergedOffsets =
+ Stream.of(committedOffsets, controlTopicOffsets)
+ .flatMap(map -> map.entrySet().stream())
+ .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue,
Long::max));
Review Comment:
[doubt] IIUC this will merge both maps and in case of matching keys take the
max Val ? wondering this case:
Lets say my committed offset :
[0, 100] [2, 250]
my control topic offset :
[1, 150] [2, 200]
my output [0, 100] [1,150] [2, 250], but i think we did went back as we [2,
200] in control offset ? is this check flagged already somewhere ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
