Philip Nee created KAFKA-18216:
----------------------------------
Summary: High water mark or last stable offset aren't always
updated after a fetch request is completed
Key: KAFKA-18216
URL: https://issues.apache.org/jira/browse/KAFKA-18216
Project: Kafka
Issue Type: Improvement
Components: clients, consumer
Reporter: Philip Nee
We've noticed AsyncKafkaConsumer doesn't always update the high water mark/LSO
followed by handling a successful fetch response. And we know consumer lag
metrics is calculated by HWM/LSO - current fetched position. We are suspecting
this could have a subtle effect into how consumer lag is recorded, which might
have a slight impact into the accuracy of client metrics reporting.
The consumer records consumer lag when reading the fetched record
The consumer updates the HWM/LSO when the background thread completes the
fetched request.
In the original implementation, the fetcher consistently updates the HWM/LSO
after handling the completed fetch request.
In the new implementation, due to the async threading model, we can't guarantee
the sequence of the event.
This defect is affecting neither performance nor correctness and is therefore
marked as "Minor"
This can be easily reproduced using the java-produce-consumer-demo.sh example.
Ensure to produce enough records (I use 200000000 records, less is fine as
well). Custom logging is required.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
