Matthias J. Sax created KAFKA-21508:
---------------------------------------
Summary: AsyncKafkaConsumer#currentLag() blocks on the background
thread even when the lag is known locally
Key: KAFKA-21508
URL: https://issues.apache.org/jira/browse/KAFKA-21508
Project: Kafka
Issue Type: Improvement
Components: clients, consumer
Reporter: Matthias J. Sax
AsyncKafkaConsumer#currentLag() always sends a CurrentLagEvent to the
background thread and waits for the result
(applicationEventHandler.addAndGet(...)). It blocks on the event future until
the network thread handles it at the start of its next runOnce() iteration.
The background thread then only does a local read:
* OffsetsRequestManager#currentLag() returns
subscriptionState.partitionLag(tp, isolationLevel), which is the high watermark
(or last stable offset) from the last fetch response minus the position.
* Only if the lag is unknown does it also send a one-shot ListOffsets request.
It doesn't wait for that request and returns OptionalLong.empty().
So in the common case, where the lag is known from the last fetch, the caller
waits on a thread handoff for a value it could read itself:
* ConsumerSubscriptionState#partitionLag() is synchronized.
* AsyncKafkaConsumer#position() already reads subscriptions directly on the
application thread.
* ClassicKafkaConsumer#currentLag() reads the lag locally too.
Callers that poll the lag per partition in their processing loop pay this
handoff on every call. Kafka Streams is one, for task idling and the reported
task end offsets.
*Suggested* *change:* read subscriptions.partitionLag(tp, isolationLevel) on
the application thread. Only when it's null (and partitionEndOffset is
unknown), enqueue a fire-and-forget event so the background thread sends the
ListOffsets request, then return OptionalLong.empty() without waiting. The
end-offset-requested flag check and the request stay on the background thread,
as today.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)