phet commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1369648784


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
   private void consume() {
     try {
       Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+      // TODO: we may be committing too early and only want to commit after 
process messages
       if(!enableAutoCommit) {
         commitOffsets();
       }
       while (itr.hasNext()) {
         KafkaConsumerRecord record = itr.next();
         int idx = record.getPartition() % numThreads;
         queues[idx].put(record);
+        // Increment queue size metric
+        updateQueueSizes(idx);

Review Comment:
   this is a pretty fundamental change to put into the fast path!  we gotta be 
certain that no critical use is adversely affected by perf regression.
   
   I see `ConcurrentLinkedQueue::size` javadoc saying `O(n)`, but I can't find 
a statement on `LinkedBlockingQueue::size`...
   
   (it's possible it IS `O(1)`, but an approximation... or it may be exact, but 
at the expense of coordination overhead.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to