umustafi commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1370521553
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
private void consume() {
try {
Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+ // TODO: we may be committing too early and only want to commit after
process messages
if(!enableAutoCommit) {
commitOffsets();
}
while (itr.hasNext()) {
KafkaConsumerRecord record = itr.next();
int idx = record.getPartition() % numThreads;
queues[idx].put(record);
+ // Increment queue size metric
+ updateQueueSizes(idx);
Review Comment:
Alternatively, I was thinking we can have a background thread update the
size every minute (or some other interval). That will move this out of the fast
path. Alternatively we can call this method in an async thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]