Re: Failure to reset consumer offsets for specific topics

2017-10-26 Thread Dan Markhasin
<yuzhih...@gmail.com> wrote: > Can you update librdkafka and try again ? > Thanks > Original message From: Dan Markhasin <minimi...@gmail.com> > Date: 10/25/17 11:26 PM (GMT-08:00) To: users@kafka.apache.org Subject: > Re: Failure to reset consumer

Re: Failure to reset consumer offsets for specific topics

2017-10-26 Thread Ted Yu
Can you update librdkafka and try again ? Thanks Original message From: Dan Markhasin <minimi...@gmail.com> Date: 10/25/17 11:26 PM (GMT-08:00) To: users@kafka.apache.org Subject: Re: Failure to reset consumer offsets for specific topics For beats (the topic where time

Re: Failure to reset consumer offsets for specific topics

2017-10-26 Thread Dan Markhasin
For beats (the topic where timestamps are OK) the producers are Metricbeat / Logstash which I assume use a relatively recent producer version. For the topics with missing timestamps the producers are based on librdkafka though I'm not sure which version exactly - I wouldn't be surprised if it's

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Elyahou Ittah
Which driver is used yo produce these messages ? On Thu, Oct 26, 2017 at 8:14 AM, Dan Markhasin wrote: > Furthermore, when looking at messages produced to the data1_log topic with > print.timestamp=true they all have CreateTime:-1 whereas messages produced > to the beats

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Dan Markhasin
Furthermore, when looking at messages produced to the data1_log topic with print.timestamp=true they all have CreateTime:-1 whereas messages produced to the beats topic have valid timestamps. The producers that are sending to data1_log are older than the producers that are sending to beats - if

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Dan Markhasin
After a bit more checking it seems that Kafka isn't writing timestamps at all in the .timeindex file for the topics where offset rewind is not working. The following output is from * a different 0.11.0.0 * which also has a topic called data1_log (this cluster has not experienced any issues

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Kelly Shkuratoff
Any idea why I started receiving mails from this list as of 2:43 am today? I didn't make any changes or subscribe to anything. I even clicked unsubscribe earlier today and am still receiving mails. Maybe there's a misconfiguration in your email list? Kelly On Wed, Oct 25, 2017 at 1:44 PM, Hans

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Elyahou Ittah
Which driver is used yo produce these messages ? On Oct 26, 2017 07:11, "Dan Markhasin" wrote: > No, that flag doesn't affect which offsets are returned, only executes the > action (and resets the consumer to latest offset when used, regardless of > datetime value I

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Dan Markhasin
No, that flag doesn't affect which offsets are returned, only executes the action (and resets the consumer to latest offset when used, regardless of datetime value I provide). On 25 October 2017 at 23:44, Hans Jespersen wrote: > I think you are just missing the —execute flag.

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
Clarification: my most recent reply was w.r.t. the strange situation Dan described, not the offset resetting. On Wed, Oct 25, 2017 at 1:24 PM, Ted Yu wrote: > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > > On Wed, Oct 25,

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Hans Jespersen
I think you are just missing the —execute flag. -hans > On Oct 25, 2017, at 1:24 PM, Ted Yu wrote: > > I wonder if you have hit KAFKA-5600. > > Is it possible that you try out 0.11.0.1 ? > > Thanks > >> On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
I wonder if you have hit KAFKA-5600. Is it possible that you try out 0.11.0.1 ? Thanks On Wed, Oct 25, 2017 at 1:15 PM, Dan Markhasin wrote: > I am using 0.11.0.0. > > There is no difference configuration-wise - both have 10 partitions and 2 > replicas. There are no

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Dan Markhasin
I am using 0.11.0.0. There is no difference configuration-wise - both have 10 partitions and 2 replicas. There are no errors in the logs, but looking in the data folder it seems like Kafka is not updating the timeindex file for data1_log - notice how the timeindex file for the current log segment

Re: Failure to reset consumer offsets for specific topics

2017-10-25 Thread Ted Yu
Do you mind providing a bit more information ? Release of Kafka you use Any difference between data1_log and the other, normal topic ? Probably check the broker log where data1_log is hosted - see if there is some clue. Thanks On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin