Matthias J. Sax created KAFKA-21508:
---------------------------------------

             Summary: AsyncKafkaConsumer#currentLag() blocks on the background 
thread even when the lag is known locally
                 Key: KAFKA-21508
                 URL: https://issues.apache.org/jira/browse/KAFKA-21508
             Project: Kafka
          Issue Type: Improvement
          Components: clients, consumer
            Reporter: Matthias J. Sax


AsyncKafkaConsumer#currentLag() always sends a CurrentLagEvent to the 
background thread and waits for the result 
(applicationEventHandler.addAndGet(...)). It blocks on the event future until 
the network thread handles it at the start of its next runOnce() iteration.

The background thread then only does a local read:
 * OffsetsRequestManager#currentLag() returns 
subscriptionState.partitionLag(tp, isolationLevel), which is the high watermark 
(or last stable offset) from the last fetch response minus the position.
 * Only if the lag is unknown does it also send a one-shot ListOffsets request. 
It doesn't wait for that request and returns OptionalLong.empty().

So in the common case, where the lag is known from the last fetch, the caller 
waits on a thread handoff for a value it could read itself:
 * ConsumerSubscriptionState#partitionLag() is synchronized.
 * AsyncKafkaConsumer#position() already reads subscriptions directly on the 
application thread.
 * ClassicKafkaConsumer#currentLag() reads the lag locally too.

Callers that poll the lag per partition in their processing loop pay this 
handoff on every call. Kafka Streams is one, for task idling and the reported 
task end offsets.

*Suggested* *change:* read subscriptions.partitionLag(tp, isolationLevel) on 
the application thread. Only when it's null (and partitionEndOffset is 
unknown), enqueue a fire-and-forget event so the background thread sends the 
ListOffsets request, then return OptionalLong.empty() without waiting. The 
end-offset-requested flag check and the request stay on the background thread, 
as today.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to