[jira] [Commented] (KAFKA-6724) ConsumerPerformance resets offsets on every startup

2018-04-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424114#comment-16424114
 ] 

ASF GitHub Bot commented on KAFKA-6724:
---

rootex- opened a new pull request #4818: KAFKA-6724 ConsumerPerformance resets 
offsets on every startup
URL: https://github.com/apache/kafka/pull/4818
 
 
   ## Remove consumer offset reset on startup
   
   ### ConsumerPerformance fails to consume all messages 
   on topics with large number of partitions due to a relatively short default 
polling loop timeout (1000 ms) that is not reachable and modifiable by the end 
user. 
   
   ### Demo
   Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte records 
using kafka-producer-perf-test and consume them using kafka-consumer-perf-test 
(ConsumerPerformance). You will likely notice that the number of records 
returned by the kafka-consumer-perf-test is many times less than expected 50 
000 000. 
   
   This happens due to specific ConsumerPerformance implementation. As the 
result, in some rough cases it may take a long enough time to process/iterate 
through the records polled in batches, thus, the time may exceed the default 
hardcoded polling loop timeout and this is probably not what we want from this 
utility. 
   
   ### Possible options
   1) Increasing polling loop timeout in ConsumerPerformance implementation. It 
defaults to 1000 ms and is hardcoded, thus cannot be changed but we could 
export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a 
script level configuration and available to the end user.
   2) Decreasing max.poll.records on a Consumer config level. This is not a 
fine option though since we do not want to touch the default settings.
   
   
   ### Committer Checklist (excluded from commit message)
   - [x] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> ConsumerPerformance resets offsets on every startup
> ---
>
> Key: KAFKA-6724
> URL: https://issues.apache.org/jira/browse/KAFKA-6724
> Project: Kafka
>  Issue Type: Bug
>  Components: core, tools
>Affects Versions: 0.11.0.1
>Reporter: Alex Dunayevsky
>Priority: Minor
> Fix For: 1.2.0
>
>
> ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for 
> it's group on every startup. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6724) ConsumerPerformance resets offsets on every startup

2018-03-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417807#comment-16417807
 ] 

ASF GitHub Bot commented on KAFKA-6724:
---

hachikuji closed pull request #4787: KAFKA-6724 ConsumerPerformance resets 
offsets on every startup
URL: https://github.com/apache/kafka/pull/4787
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/core/src/main/scala/kafka/tools/ConsumerPerformance.scala 
b/core/src/main/scala/kafka/tools/ConsumerPerformance.scala
index a3e60e652c4..7e0dbcbe064 100644
--- a/core/src/main/scala/kafka/tools/ConsumerPerformance.scala
+++ b/core/src/main/scala/kafka/tools/ConsumerPerformance.scala
@@ -157,8 +157,6 @@ object ConsumerPerformance extends LazyLogging {
   def onPartitionsRevoked(partitions: util.Collection[TopicPartition]) {
 joinStart = System.currentTimeMillis
   }})
-consumer.poll(0)
-consumer.seekToBeginning(Collections.emptyList())
 
 // Now start the benchmark
 val startMs = System.currentTimeMillis


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> ConsumerPerformance resets offsets on every startup
> ---
>
> Key: KAFKA-6724
> URL: https://issues.apache.org/jira/browse/KAFKA-6724
> Project: Kafka
>  Issue Type: Bug
>  Components: core, tools
>Affects Versions: 0.11.0.1
>Reporter: Alex Dunayevsky
>Priority: Minor
> Fix For: 1.2.0
>
>
> ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for 
> it's group on every startup. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6724) ConsumerPerformance resets offsets on every startup

2018-03-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417369#comment-16417369
 ] 

ASF GitHub Bot commented on KAFKA-6724:
---

rootex- opened a new pull request #4787: KAFKA-6724 ConsumerPerformance resets 
offsets on every startup
URL: https://github.com/apache/kafka/pull/4787
 
 
   Remove consumer offset reset on startup
   
   *More detailed description of your change,
   if necessary. The PR title and PR message become
   the squashed commit message, so use a separate
   comment to ping reviewers.*
   
   *Summary of testing strategy (including rationale)
   for the feature or bug fix. Unit and/or integration
   tests are expected for any behaviour change and
   system tests should be considered for larger changes.*
   
   ### Committer Checklist (excluded from commit message)
   - [ x ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> ConsumerPerformance resets offsets on every startup
> ---
>
> Key: KAFKA-6724
> URL: https://issues.apache.org/jira/browse/KAFKA-6724
> Project: Kafka
>  Issue Type: Bug
>  Components: core, tools
>Affects Versions: 0.11.0.1
>Reporter: Alex Dunayevsky
>Priority: Minor
>
> ConsumerPerformance used in kafka-consumer-perf-test.sh resets offsets for 
> it's group on every startup. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)