[ 
https://issues.apache.org/jira/browse/GOBBLIN-1934?focusedWorklogId=887043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-887043
 ]

ASF GitHub Bot logged work on GOBBLIN-1934:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 24/Oct/23 16:45
            Start Date: 24/Oct/23 16:45
    Worklog Time Spent: 10m 
      Work Description: umustafi commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1370521553


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
   private void consume() {
     try {
       Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+      // TODO: we may be committing too early and only want to commit after 
process messages
       if(!enableAutoCommit) {
         commitOffsets();
       }
       while (itr.hasNext()) {
         KafkaConsumerRecord record = itr.next();
         int idx = record.getPartition() % numThreads;
         queues[idx].put(record);
+        // Increment queue size metric
+        updateQueueSizes(idx);

Review Comment:
   Alternatively, I was thinking we can have a background thread update the 
size every minute (or some other interval). That will move this out of the fast 
path. Alternatively we can call this method in an async thread. 





Issue Time Tracking
-------------------

    Worklog Id:     (was: 887043)
    Time Spent: 1h  (was: 50m)

> Monitor High Level Consumer queue size
> --------------------------------------
>
>                 Key: GOBBLIN-1934
>                 URL: https://issues.apache.org/jira/browse/GOBBLIN-1934
>             Project: Apache Gobblin
>          Issue Type: Improvement
>          Components: gobblin-service
>            Reporter: Urmi Mustafi
>            Assignee: Abhishek Tiwari
>            Priority: Major
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> We suspect that the HighLevelConsumer class may be dropping Kafka messages if 
> the thread for a particular consumption queue is interrupted or stalled. 
> Before changing how these queues are handled, we want to verify that blocked 
> or dropped queues is the reason for missed messages by emitting metrics about 
> each queue's size. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to