[
https://issues.apache.org/jira/browse/KAFKA-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-6743.
------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
> ConsumerPerformance fails to consume all messages on topics with large number
> of partitions
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-6743
> URL: https://issues.apache.org/jira/browse/KAFKA-6743
> Project: Kafka
> Issue Type: Bug
> Components: core, tools
> Affects Versions: 0.11.0.2
> Reporter: Alex Dunayevsky
> Priority: Minor
> Fix For: 2.0.0
>
>
> ConsumerPerformance fails to consume all messages on topics with large number
> of partitions due to a relatively short default polling loop timeout (1000
> ms) that is not reachable and modifiable by the end user.
> Demo: Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte
> records using kafka-producer-perf-test and consume them using
> kafka-consumer-perf-test (ConsumerPerformance). You will likely notice that
> the number of records returned by the kafka-consumer-perf-test is many times
> less than expected 50 000 000. This happens due to specific
> ConsumerPerformance implementation. As the result, in some rough cases it may
> take a long enough time to process/iterate through the records polled in
> batches, thus, the time may exceed the default hardcoded polling loop timeout
> and this is probably not what we want from this utility.
> We have two options:
> 1) Increasing polling loop timeout in ConsumerPerformance implementation. It
> defaults to 1000 ms and is hardcoded, thus cannot be changed but we could
> export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a
> script level configuration and available to the end user.
> 2) Decreasing max.poll.records on a Consumer config level. This is not a fine
> option though since we do not want to touch the default settings.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
