[
https://issues.apache.org/jira/browse/KAFKA-12980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367556#comment-17367556
]
Chris Egerton commented on KAFKA-12980:
---------------------------------------
CC [~hachikuji]--thoughts on this?
> Allow consumers to return from poll when position advances due to aborted
> transactions
> --------------------------------------------------------------------------------------
>
> Key: KAFKA-12980
> URL: https://issues.apache.org/jira/browse/KAFKA-12980
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Chris Egerton
> Priority: Major
>
> When {{Consumer::poll}} is invoked on a topic with an open transaction, and
> then that transaction is aborted, {{poll}} does not return unless there are
> other records available in that topic after the aborted transaction.
> Instead, {{poll}} could return in this case, even when no records are
> available.
> This facilitates reads to the end of a topic where the end offsets of a topic
> are listed and then a consumer for that topic is polled until its
> [position|https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#position(org.apache.kafka.common.TopicPartition)]
> is at or beyond each of those offsets (for example, [Connect does
> this|https://github.com/apache/kafka/blob/fce771579c3e20f20949c4c7e0a5e3a16c57c7f0/connect/runtime/src/main/java/org/apache/kafka/connect/util/KafkaBasedLog.java#L322-L345]
> when reading to the end of any of its internal topics).
> We could update the existing language in the docs for {{Consumer::poll}} from
> {quote}This method returns immediately if there are records available.
> {quote}
> to
> {quote}This method returns immediately if there are records available or if
> the position advances past control records.
> {quote}
>
> A workaround for existing users who would like to see this is to use short
> poll intervals and manually check the consumer's position in between each
> poll, but this is fairly tedious and may lead to excess CPU and network
> utilization depending on the latency requirements for knowing when the end of
> the topic has been reached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
