[
https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16424114#comment-16424114
]
ASF GitHub Bot commented on KAFKA-6724:
---------------------------------------
rootex- opened a new pull request #4818: KAFKA-6724 ConsumerPerformance resets
offsets on every startup
URL: https://github.com/apache/kafka/pull/4818
## Remove consumer offset reset on startup
### ConsumerPerformance fails to consume all messages
on topics with large number of partitions due to a relatively short default
polling loop timeout (1000 ms) that is not reachable and modifiable by the end
user.
### Demo
Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte records
using kafka-producer-perf-test and consume them using kafka-consumer-perf-test
(ConsumerPerformance). You will likely notice that the number of records
returned by the kafka-consumer-perf-test is many times less than expected 50
000 000.
This happens due to specific ConsumerPerformance implementation. As the
result, in some rough cases it may take a long enough time to process/iterate
through the records polled in batches, thus, the time may exceed the default
hardcoded polling loop timeout and this is probably not what we want from this
utility.
### Possible options
1) Increasing polling loop timeout in ConsumerPerformance implementation. It
defaults to 1000 ms and is hardcoded, thus cannot be changed but we could
export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a
script level configuration and available to the end user.
2) Decreasing max.poll.records on a Consumer config level. This is not a
fine option though since we do not want to touch the default settings.
### Committer Checklist (excluded from commit message)
- [x] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ConsumerPerformance resets offsets on every startup
> ---------------------------------------------------
>
> Key: KAFKA-6724
> URL: https://issues.apache.org/jira/browse/KAFKA-6724
> Project: Kafka
> Issue Type: Bug
> Components: core, tools
> Affects Versions: 0.11.0.1
> Reporter: Alex Dunayevsky
> Priority: Minor
> Fix For: 1.2.0
>
>
> ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for
> it's group on every startup.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
