[
https://issues.apache.org/jira/browse/KAFKA-18216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18022535#comment-18022535
]
Kirk True commented on KAFKA-18216:
-----------------------------------
[~frankvicky]—are you actively working on this task? [~ezhou] and I will be
looking into this in the coming days. Thanks!
> High water mark or last stable offset aren't always updated after a fetch
> request is completed
> ----------------------------------------------------------------------------------------------
>
> Key: KAFKA-18216
> URL: https://issues.apache.org/jira/browse/KAFKA-18216
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Philip Nee
> Assignee: TengYao Chi
> Priority: Minor
> Labels: async-kafka-consumer-performance,
> consumer-threading-refactor, performance
> Fix For: 4.2.0
>
>
> We've noticed AsyncKafkaConsumer doesn't always update the high water
> mark/LSO followed by handling a successful fetch response. And we know
> consumer lag metrics is calculated by HWM/LSO - current fetched position. We
> are suspecting this could have a subtle effect into how consumer lag is
> recorded, which might have a slight impact into the accuracy of client
> metrics reporting.
> The consumer records consumer lag when reading the fetched record
> The consumer updates the HWM/LSO when the background thread completes the
> fetched request.
> In the original implementation, the fetcher consistently updates the HWM/LSO
> after handling the completed fetch request.
> In the new implementation, due to the async threading model, we can't
> guarantee the sequence of the event.
> This defect is affecting neither performance nor correctness and is therefore
> marked as "Minor"
>
> This can be easily reproduced using the java-produce-consumer-demo.sh
> example. Ensure to produce enough records (I use 200000000 records, less is
> fine as well). Custom logging is required.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
