I am using 0.11.0.0.
There is no difference configuration-wise - both have 10 partitions and 2
replicas. There are no errors in the logs, but looking in the data folder
it seems like Kafka is not updating the timeindex file for data1_log -
notice how the timeindex file for the current log segment is not being
updated.
bash-4.2$ pwd
/kafka/data/data1_log-1
bash-4.2$ ls -ltr | tail
-rw-rw-r-- 1 ibiuser it 1073731573 Oct 25 01:21 00000000000337554984.log
-rw-rw-r-- 1 ibiuser it 943616 Oct 25 01:21 00000000000337554984.index
-rw-rw-r-- 1 ibiuser it 1073734199 Oct 25 13:38 00000000000339816017.log
-rw-rw-r-- 1 ibiuser it 10485756 Oct 25 13:38
00000000000341934289.timeindex
-rw-rw-r-- 1 ibiuser it 10 Oct 25 13:38
00000000000341934289.snapshot
-rw-rw-r-- 1 ibiuser it 0 Oct 25 13:38
00000000000339816017.timeindex
-rw-rw-r-- 1 ibiuser it 566712 Oct 25 13:38 00000000000339816017.index
-rw-rw-r-- 1 ibiuser it 17 Oct 25 20:23 leader-epoch-checkpoint
-rw-rw-r-- 1 ibiuser it 10485760 Oct 25 23:03 00000000000341934289.index
-rw-rw-r-- 1 ibiuser it 461590419 Oct 25 23:04 00000000000341934289.log
For comparison, the beats topic:
bash-4.2$ cd ../beats-1
bash-4.2$ ls -ltr
total 3212088
-rw-rw-r-- 1 ibiuser it 17 Oct 25 00:23 leader-epoch-checkpoint
-rw-rw-r-- 1 ibiuser it 10 Oct 25 20:04
00000000000188672034.snapshot
-rw-rw-r-- 1 ibiuser it 2773008 Oct 25 20:04
00000000000185224087.timeindex
-rw-rw-r-- 1 ibiuser it 1073741779 Oct 25 20:04 00000000000185224087.log
-rw-rw-r-- 1 ibiuser it 1967440 Oct 25 20:04 00000000000185224087.index
-rw-rw-r-- 1 ibiuser it 10485760 Oct 25 23:03 00000000000188672034.index
-rw-rw-r-- 1 ibiuser it 10485756 Oct 25 23:04
00000000000188672034.timeindex
-rw-rw-r-- 1 ibiuser it 50166645 Oct 25 23:04 00000000000188672034.log
To give some context to why I'm even trying to reset the offsets, we had
encountered a strange situation earlier today:
1) One of the brokers had a hardware failure, and had to be rebuilt from
scratch (data partition was gone)
2) When it went down, we noticed a spike in lag in one particular consumer
group - it seems to have reset its offset to an earlier point in time (but
not the earliest offset of the topic); I have read other messages on this
mailing list of users who experienced the same behavior with 0.11.0.0
3) The broker was reinstalled and rejoined the cluster with the same
broker.id (but with no data on it) - it rebalanced and eventually all
replicas became synced and the cluster was functioning normally.
4) I then decided to bounce the same broker again to see if I can reproduce
the issue I saw in #2 - and as soon as the broker was restarted, the exact
same consumer group had its offset reset again and was lagging with
millions of records behind the current offset.
5) I then tried to manually reset the consumer group's offset to a few
minutes before I restarted the broker, only to discover this strange
behavior where no matter which datetime value I provided, it kept resetting
to the latest offset.
On 25 October 2017 at 22:48, Ted Yu <yuzhih...@gmail.com> wrote:
> Do you mind providing a bit more information ?
>
> Release of Kafka you use
>
> Any difference between data1_log and the other, normal topic ?
>
> Probably check the broker log where data1_log is hosted - see if there is
> some clue.
>
> Thanks
>
> On Wed, Oct 25, 2017 at 12:11 PM, Dan Markhasin <minimi...@gmail.com>
> wrote:
>
> > I'm trying to use the kafka-consumer-groups.sh tool in order to rewind a
> > consumer group's offset, however it seems to be returning the latest
> offset
> > regarding of the requested offset.
> >
> > You can see in the below example that two consecutive commands to reset
> the
> > offset to a specific point in time return different (increasing) offsets,
> > which are actually the latest offsets for the topic.
> >
> > - The consumer group ("test_consumer") is a console consumer that was
> > started with --from-beginning and terminated after a few seconds, just
> > enough for it to commit its offsets.
> > - The topic data1_log is very busy with thousands of incoming messages
> per
> > second
> > - The datetime value provided is approx. 5 hours earlier than the current
> > UTC time
> >
> > [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> > --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> > --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> > Note: This will only show information about consumers that use the Java
> > consumer API (non-ZooKeeper-based consumers).
> >
> >
> > TOPIC PARTITION NEW-OFFSET
> > data1_log 8 301485420
> > data1_log 1 342788637
> > data1_log 7 287621428
> > data1_log 3 268612266
> > data1_log 0 201860717
> > data1_log 9 202749553
> > data1_log 4 188974032
> > data1_log 6 234308481
> > data1_log 2 263507741
> > data1_log 5 232707238
> >
> > [admin@broker01] ~> /kafka/latest/bin/kafka-consumer-groups.sh
> > --bootstrap-server localhost:9092 --reset-offsets --group test_consumer
> > --topic data1_log --to-datetime '2017-10-25T13:40:00.000'
> > Note: This will only show information about consumers that use the Java
> > consumer API (non-ZooKeeper-based consumers).
> >
> >
> > TOPIC PARTITION NEW-OFFSET
> > data1_log 8 301485491
> > data1_log 1 342788779
> > data1_log 7 287621534
> > data1_log 3 268612364
> > data1_log 0 201860796
> > data1_log 9 202749620
> > data1_log 4 188974068
> > data1_log 6 234308564
> > data1_log 2 263507823
> > data1_log 5 232707293
> >
> > This issue seems to be topic-specific - there is a different topic (also
> > very active) where the same command consistently returns the correct
> > offsets fixed in the time for the requested datetime.
> >
> > What could be the issue here?
> >
> > Thanks,
> > Dan
> >
>
