wangxianghu commented on a change in pull request #3175:
URL: https://github.com/apache/hudi/pull/3175#discussion_r660220267
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java
##########
@@ -327,4 +330,19 @@ public void commitOffsetToKafka(String checkpointStr) {
LOG.warn("Committing offsets to Kafka failed, this does not impact
processing of records", e);
}
}
+
+ private Map<TopicPartition, Long> getGroupOffsets(KafkaConsumer consumer,
Set<TopicPartition> topicPartitions) {
+ Map<TopicPartition, Long> fromOffsets = new HashMap<>();
+ for (TopicPartition topicPartition : topicPartitions) {
+ OffsetAndMetadata committedOffsetAndMetadata =
consumer.committed(topicPartition);
+ if (committedOffsetAndMetadata != null) {
+ fromOffsets.put(topicPartition, committedOffsetAndMetadata.offset());
+ } else {
+ LOG.warn("There are no commits associated with this consumer group,
starting to consume form latest offset");
+ fromOffsets = consumer.endOffsets(topicPartitions);
+ break;
Review comment:
This logic seems the same as `LATEST` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
