Fabien Chaillou created KAFKA-6209:
--------------------------------------
Summary: Lag is inconsistent when manually committing offset for
transactionnal messages
Key: KAFKA-6209
URL: https://issues.apache.org/jira/browse/KAFKA-6209
Project: Kafka
Issue Type: Improvement
Components: consumer
Affects Versions: 0.11.0.1
Reporter: Fabien Chaillou
Hi,
I'm using kafka consumers paired with kafka producers to have transactionnal
consume -> produce -> commit flow.
I checked the kafka-consumer-groups.sh tool to make sure my consumer were
consuming all the messages and it appears that my consumer always have a lag of
1.
After more digging, I discovered that kafka uses Control records for
transaction handling and those records are not returned to the consumer. So if
the last record published in the topic is part of a transaction then kafka will
add a control record after it and the client code will only be able to send a
commit request up to the last control record and the lag will be computed as 1.
It is not really a bug per se but an inconsistent behavior and I add to dig
into the exactly-once KIP (KIP-98) and the consumer code to figure this out.
I think the issue should at least be documented somewhere as I'm honestly not
sure of the proper fix.
Thanks for your feedback.
Fabien
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
