[
https://issues.apache.org/jira/browse/KAFKA-17093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864508#comment-17864508
]
Tom Kalmijn edited comment on KAFKA-17093 at 7/10/24 7:13 AM:
--------------------------------------------------------------
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
was (Author: JIRAUSER306103):
I think at least I (too) now understand why there is a read_committed flag for
seekToEnd. If not specified seekToEnd could possibly scroll passed an ongoing
uncommitted transaction and this can result in a bug where a consumer misses a
record.
*Let's conclude that seekToEnd works as designed and as documented?*
But is there any method at all that tells us the offset of the last message in
a topic/partition? I find it mind boggling if that really does not exist. I
have seen so many threads where people are throwing each other this "last
offset minus one" logic to get to this information. It is clearly something
more people need in their designs.
There is OffsetSpec for the adminClient:
{code:java}
request = Map(topicPartition -> OffsetSpec.maxTimestamp)
adminClient.listOffsets(request, options = new
ListOffsetsOptions(READ_COMMITTED)
{code}
This yields the same result as discussed above (NOT the offset of the last
message).
OffsetSpec currently supports: 'latest', 'earliest', 'forTimestamp' and
'maxTimestamp'.
If only it would include another option like 'latestCommittedMessage', that
would be very helpful.
> KafkaConsumer.seekToEnd should return LSO
> ------------------------------------------
>
> Key: KAFKA-17093
> URL: https://issues.apache.org/jira/browse/KAFKA-17093
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Affects Versions: 3.6.1
> Environment: Ubuntu, IntelliJ, Scala "org.apache.kafka" %
> "kafka-clients" % "3.6.1"
> Reporter: Tom Kalmijn
> Assignee: Andrew Schofield
> Priority: Major
> Attachments: Kafka17093-v2.java, Kafka17093-v3.java, Kafka17093.java
>
>
>
> Expected
> When using a transactional producer then the method
> KafkaConsumer.seekToEnd(...) of a consumer configured with isolation level
> "read_committed" should return the LSO.
> Observed
> The offset returned is always the actual last offset of the partition, which
> is not the LSO if the latest offsets are occupied by transaction markers.
> Also see this Slack thread:
> https://confluentcommunity.slack.com/archives/C499EFQS0/p1720088282557559
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
