[
https://issues.apache.org/jira/browse/KAFKA-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032198#comment-18032198
]
Matthias J. Sax edited comment on KAFKA-10683 at 10/22/25 5:48 PM:
-------------------------------------------------------------------
I am just getting aware of this ticket – it's behavior by design, and the
application layer need to solve it: the consumer need to keep calling `poll()`
to maintain group membership, and thus, if a batch or record really does not
include the TX marker, and current `position() == lastRecord.offset() + 1`,
this would resolve with the consecutive `poll()` call, which will advance
`position` across the TX marker. Thus, the application code must buffer the
last committed offset, and if `poll()` returns no data, compare it to
`position()`, and if `position()` advanced (compare to the last committed
offset), the app should do another offset commit. (We have the same
implementation inside Kafka Streams.)
As a matter of fact, we actually found some other issue with using `position()`
which does not return the partition-leader epoch. This was addressed via
https://issues.apache.org/jira/browse/KAFKA-17600 and applications should use
the newly added `ConsumerRecords#nextOffsets()` API, to commit offset correctly.
We also wrote a detailed blog post about this issue
[https://www.confluent.io/blog/guide-to-consumer-offsets/]
was (Author: mjsax):
I am just getting aware of this ticket – it's behavior by design, and the
application layer need to solve it: the consumer need to keep calling `poll()`
to maintain group membership, and thus, if a batch or record really does not
include the TX marker, and current `position() == lastRecord.offset() + 1`,
this would resolve with the consecutive `poll()` call, which will advance
`position` across the TX marker. Thus, the application code must buffer the
last committed offset, and if `poll()` returns no data, compare it to
`position()`, and if `position()` advanced (compare to the last committed
offset), the app should do another offset commit. (We have the same
implementation inside Kafka Streams.)
> Consumer.position() Ignores Transaction Marker with read_uncommitted
> --------------------------------------------------------------------
>
> Key: KAFKA-10683
> URL: https://issues.apache.org/jira/browse/KAFKA-10683
> Project: Kafka
> Issue Type: Bug
> Components: clients, core
> Affects Versions: 2.6.0
> Reporter: Gary Russell
> Priority: Minor
>
> The workaround for https://issues.apache.org/jira/browse/KAFKA-6607# Says:
> {quote}
> or use `consumer.position()` that takes the commit marker into account and
> would "step over it")
> {quote}
> Note that this problem occurs with all consumers, not just Streams. We have
> implemented this solution in our project (as an option for those users
> concerned about the pseudo lag).
> We have discovered that this technique will only work with
> {code}isolation.level=read_committed{code} Otherwise, the
> {code}position(){code} call does not include the marker "record".
> https://github.com/spring-projects/spring-kafka/issues/1587#issuecomment-721899560
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
