danielcweeks commented on code in PR #17552:
URL: https://github.com/apache/iceberg/pull/17552#discussion_r3938058979
##########
kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/Channel.java:
##########
@@ -142,13 +143,35 @@ protected Map<Integer, Long> controlTopicOffsets() {
return controlTopicOffsets;
}
+ /**
+ * Commit consumer offsets. Only commits offsets if it has not committed
offsets before or the
+ * value is greater than the cached offset.
+ *
+ * <p>Note: there is a risk that two parallel coordinators may overwrite
each other's offsets, to
+ * be fixed in a follow-up PR.
+ */
protected void commitConsumerOffsets() {
Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = Maps.newHashMap();
- controlTopicOffsets()
- .forEach(
- (k, v) ->
- offsetsToCommit.put(new TopicPartition(controlTopic, k), new
OffsetAndMetadata(v)));
- consumer.commitSync(offsetsToCommit);
+ controlTopicOffsets.forEach(
+ (partition, offsetToCommit) -> {
+ Long lastCommittedOffset = committedOffsets.get(partition);
+ if (lastCommittedOffset == null || offsetToCommit >
lastCommittedOffset) {
+ TopicPartition tp = new TopicPartition(controlTopic, partition);
+ offsetsToCommit.put(tp, new OffsetAndMetadata(offsetToCommit));
+ }
+ });
+ if (!offsetsToCommit.isEmpty()) {
Review Comment:
Doesn't this capture both the scenario where there is a noop and the
preventing commits as the same thing? We should avoid confusing the two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
