[
https://issues.apache.org/jira/browse/GOBBLIN-1934?focusedWorklogId=887043&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-887043
]
ASF GitHub Bot logged work on GOBBLIN-1934:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Oct/23 16:45
Start Date: 24/Oct/23 16:45
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1370521553
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
private void consume() {
try {
Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+ // TODO: we may be committing too early and only want to commit after
process messages
if(!enableAutoCommit) {
commitOffsets();
}
while (itr.hasNext()) {
KafkaConsumerRecord record = itr.next();
int idx = record.getPartition() % numThreads;
queues[idx].put(record);
+ // Increment queue size metric
+ updateQueueSizes(idx);
Review Comment:
Alternatively, I was thinking we can have a background thread update the
size every minute (or some other interval). That will move this out of the fast
path. Alternatively we can call this method in an async thread.
Issue Time Tracking
-------------------
Worklog Id: (was: 887043)
Time Spent: 1h (was: 50m)
> Monitor High Level Consumer queue size
> --------------------------------------
>
> Key: GOBBLIN-1934
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1934
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> We suspect that the HighLevelConsumer class may be dropping Kafka messages if
> the thread for a particular consumption queue is interrupted or stalled.
> Before changing how these queues are handled, we want to verify that blocked
> or dropped queues is the reason for missed messages by emitting metrics about
> each queue's size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)