[
https://issues.apache.org/jira/browse/KAFKA-6209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fabien Chaillou updated KAFKA-6209:
-----------------------------------
Summary: Consumer lag is inconsistent when manually committing offset for
transactionnal messages (was: Lag is inconsistent when manually committing
offset for transactionnal messages)
> Consumer lag is inconsistent when manually committing offset for
> transactionnal messages
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-6209
> URL: https://issues.apache.org/jira/browse/KAFKA-6209
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Affects Versions: 0.11.0.1
> Reporter: Fabien Chaillou
>
> Hi,
> I'm using kafka consumers paired with kafka producers to have transactionnal
> consume -> produce -> commit flow.
> I checked the kafka-consumer-groups.sh tool to make sure my consumer were
> consuming all the messages and it appears that my consumer always have a lag
> of 1.
> After more digging, I discovered that kafka uses Control records for
> transaction handling and those records are not returned to the consumer. So
> if the last record published in the topic is part of a transaction then kafka
> will add a control record after it and the client code will only be able to
> send a commit request up to the last control record and the lag will be
> computed as 1.
> It is not really a bug per se but an inconsistent behavior and I had to dig
> into the exactly-once KIP (KIP-98) and the consumer code to figure this out.
> I think the issue should at least be documented somewhere as I'm honestly not
> sure of the proper fix.
> Thanks for your feedback.
> Fabien
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
