[
https://issues.apache.org/jira/browse/KAFKA-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sharon Lucas updated KAFKA-6572:
--------------------------------
Affects Version/s: 1.0.0
Description:
Using Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh
reset-offsets option to reset offsets to a specific date/time in our production
environment.
We first tried to use the kafka-consumer-groups.sh command with the
--reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in
our staging environment and it worked correctly. Running the following command
changed it to start processing logs from February 12, 2018 (4 days ago) for a
topic that had a large lag. We did a dry run to verify before running with the
--execute option.
{code:java}
root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest
--to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic
staging-mon01-rg-elasticsearch --execute{code}
We stopped the kafka mirrors that process this topic before resetting the
offsets and started the kafka mirrors after rsetting the offsets. We verified
that it correctly started processing logs from February 12, 2018.
Then we tried resetting offsets in a production environment for a topic that
had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it
did not work as expected. We stopped the kafka mirrors that process this topic
before resetting the offsets and did a dry run to see what the new offsets
would be:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.
^@^@^@^@
TOPIC PARTITION NEW-OFFSET
prod_in-ams03-geo-ca_access 52 52084147
prod_in-ams03-geo-ca_access 106 52154199
prod_in-ams03-geo-ca_access 75 52148673
prod_in-ams03-geo-ca_access 61 52130753
prod_in-ams03-geo-ca_access 49 52151667
prod_in-ams03-geo-ca_access 48 52145233
prod_in-ams03-geo-ca_access 27 52092805
prod_in-ams03-geo-ca_access 26 52139644
prod_in-ams03-geo-ca_access 65 52157504
prod_in-ams03-geo-ca_access 105 52166289
prod_in-ams03-geo-ca_access 38 52160464
prod_in-ams03-geo-ca_access 22 52093451
prod_in-ams03-geo-ca_access 4 52151660
prod_in-ams03-geo-ca_access 90 52160296
prod_in-ams03-geo-ca_access 25 52161691
prod_in-ams03-geo-ca_access 13 52145828
prod_in-ams03-geo-ca_access 56 52162867
prod_in-ams03-geo-ca_access 42 52072094
prod_in-ams03-geo-ca_access 7 52069496
prod_in-ams03-geo-ca_access 117 52087078
prod_in-ams03-geo-ca_access 32 52073732
prod_in-ams03-geo-ca_access 102 52082022
prod_in-ams03-geo-ca_access 76 52141018
prod_in-ams03-geo-ca_access 83 52154542
prod_in-ams03-geo-ca_access 72 52095051
prod_in-ams03-geo-ca_access 85 52149907
prod_in-ams03-geo-ca_access 119 52134435
prod_in-ams03-geo-ca_access 113 52159340
prod_in-ams03-geo-ca_access 55 52146597
prod_in-ams03-geo-ca_access 18 52149079
prod_in-ams03-geo-ca_access 35 52149058
prod_in-ams03-geo-ca_access 99 52143277
prod_in-ams03-geo-ca_access 41 52158872
prod_in-ams03-geo-ca_access 112 52083901
prod_in-ams03-geo-ca_access 34 52137932
prod_in-ams03-geo-ca_access 89 52158135
prod_in-ams03-geo-ca_access 40 52144442
prod_in-ams03-geo-ca_access 53 52138400
prod_in-ams03-geo-ca_access 19 52144966
prod_in-ams03-geo-ca_access 44 52166404
prod_in-ams03-geo-ca_access 31 52155685
prod_in-ams03-geo-ca_access 10 52152151
prod_in-ams03-geo-ca_access 98 52145378
prod_in-ams03-geo-ca_access 69 52153436
prod_in-ams03-geo-ca_access 92 52093455
prod_in-ams03-geo-ca_access 5 52156448
prod_in-ams03-geo-ca_access 17 52063697
prod_in-ams03-geo-ca_access 24 52146410
prod_in-ams03-geo-ca_access 28 52156955
prod_in-ams03-geo-ca_access 82 52081194
prod_in-ams03-geo-ca_access 101 52153592
prod_in-ams03-geo-ca_access 58 52153765
prod_in-ams03-geo-ca_access 94 52162827
prod_in-ams03-geo-ca_access 116 52155307
prod_in-ams03-geo-ca_access 77 52072906
prod_in-ams03-geo-ca_access 68 52150283
prod_in-ams03-geo-ca_access 64 52155580
prod_in-ams03-geo-ca_access 109 52159915
prod_in-ams03-geo-ca_access 111 52153547
prod_in-ams03-geo-ca_access 2 52080434
prod_in-ams03-geo-ca_access 74 52148060
prod_in-ams03-geo-ca_access 47 52090187
prod_in-ams03-geo-ca_access 59 52145161
prod_in-ams03-geo-ca_access 1 52151228
prod_in-ams03-geo-ca_access 104 52159932
prod_in-ams03-geo-ca_access 14 52158366
prod_in-ams03-geo-ca_access 86 52153030
prod_in-ams03-geo-ca_access 81 52155507
prod_in-ams03-geo-ca_access 11 52158931
prod_in-ams03-geo-ca_access 100 52154646
prod_in-ams03-geo-ca_access 23 52169713
prod_in-ams03-geo-ca_access 45 52151320
prod_in-ams03-geo-ca_access 6 52157025
prod_in-ams03-geo-ca_access 87 52088913
prod_in-ams03-geo-ca_access 107 52086878
prod_in-ams03-geo-ca_access 16 52146481
prod_in-ams03-geo-ca_access 20 52148093
prod_in-ams03-geo-ca_access 103 52152437
prod_in-ams03-geo-ca_access 9 52158998
prod_in-ams03-geo-ca_access 29 52151112
prod_in-ams03-geo-ca_access 54 52149226
prod_in-ams03-geo-ca_access 36 52139520
prod_in-ams03-geo-ca_access 0 52152902
prod_in-ams03-geo-ca_access 84 52156057
prod_in-ams03-geo-ca_access 78 52155766
prod_in-ams03-geo-ca_access 118 52147043
prod_in-ams03-geo-ca_access 62 52087864
prod_in-ams03-geo-ca_access 108 52154615
prod_in-ams03-geo-ca_access 88 52147651
prod_in-ams03-geo-ca_access 39 52145255
prod_in-ams03-geo-ca_access 115 52166466
prod_in-ams03-geo-ca_access 51 52150136
prod_in-ams03-geo-ca_access 3 52147544
prod_in-ams03-geo-ca_access 73 52157150
prod_in-ams03-geo-ca_access 66 52158808
prod_in-ams03-geo-ca_access 67 52075616
prod_in-ams03-geo-ca_access 97 52086900
prod_in-ams03-geo-ca_access 15 52153225
prod_in-ams03-geo-ca_access 57 52097899
prod_in-ams03-geo-ca_access 8 52159097
prod_in-ams03-geo-ca_access 114 52149412
prod_in-ams03-geo-ca_access 37 52088584
prod_in-ams03-geo-ca_access 30 52153155
prod_in-ams03-geo-ca_access 79 52143773
prod_in-ams03-geo-ca_access 12 52089423
prod_in-ams03-geo-ca_access 21 52148187
prod_in-ams03-geo-ca_access 50 52150414
prod_in-ams03-geo-ca_access 60 52144624
prod_in-ams03-geo-ca_access 33 52149807
prod_in-ams03-geo-ca_access 46 52165008
prod_in-ams03-geo-ca_access 110 52153643
prod_in-ams03-geo-ca_access 95 52154736
prod_in-ams03-geo-ca_access 71 52150981
prod_in-ams03-geo-ca_access 96 52145091
prod_in-ams03-geo-ca_access 93 52149375
prod_in-ams03-geo-ca_access 91 52160207
prod_in-ams03-geo-ca_access 43 52151689
prod_in-ams03-geo-ca_access 63 52164909
prod_in-ams03-geo-ca_access 80 52151810
prod_in-ams03-geo-ca_access 70 52154522{code}
Then we ran the same command with the --execute option to the command to reset
the offsets:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access --execute{code}
And started the kafka mirrors for this topic.
However, it didn't work as expected for this topic in our production
environment.
The total log lag before resetting the offsets was 1,691,782,349:
{code:java}
root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d
%H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --describe --group
mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~
"prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }}
END{print "lag:", end_offset-current_offset;}'; sleep 15; done
2018-02-14 17:31:01 lag: 1691782349{code}
After resetting the offsets, the log lag dropped drastically to 24,330:
{code:java}
2018-02-14 17:31:17 lag: 24330{code}
This was too much of a drop in the log lag which showed it did not really reset
the offsets to start processing logs starting at Feb 10th (4 days ago). We
should be seeing a lag of around 37,170,000 if it worked correctly. Instead,
it seemed to reset the offsets to start processing logs starting today.
We also tried resetting the offsets to start processing logs starting at
February 8th (after stopping the mirrors) and it also did not work correctly.
Again, it seemed like it did a seek to the current date rather than resetting
offsets to the date we specified.
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.
^@^@^@
TOPIC PARTITION NEW-OFFSET
prod_in-ams03-geo-ca_access 52 52087106
prod_in-ams03-geo-ca_access 106 52156813
prod_in-ams03-geo-ca_access 75 52151755
prod_in-ams03-geo-ca_access 61 52133608
prod_in-ams03-geo-ca_access 49 52154385
prod_in-ams03-geo-ca_access 48 52148247
prod_in-ams03-geo-ca_access 27 52095874
prod_in-ams03-geo-ca_access 26 52142331
prod_in-ams03-geo-ca_access 65 52160181
prod_in-ams03-geo-ca_access 105 52169166
prod_in-ams03-geo-ca_access 38 52163198
prod_in-ams03-geo-ca_access 22 52096231
prod_in-ams03-geo-ca_access 4 52154512
prod_in-ams03-geo-ca_access 90 52163215
prod_in-ams03-geo-ca_access 25 52164527
prod_in-ams03-geo-ca_access 13 52148553
prod_in-ams03-geo-ca_access 56 52165558
prod_in-ams03-geo-ca_access 42 52075104
prod_in-ams03-geo-ca_access 7 52072333
prod_in-ams03-geo-ca_access 117 52090029
prod_in-ams03-geo-ca_access 32 52076469
prod_in-ams03-geo-ca_access 102 52085066
prod_in-ams03-geo-ca_access 76 52143820
prod_in-ams03-geo-ca_access 83 52157217
prod_in-ams03-geo-ca_access 72 52098060
prod_in-ams03-geo-ca_access 85 52152612
prod_in-ams03-geo-ca_access 119 52137225
prod_in-ams03-geo-ca_access 113 52162012
prod_in-ams03-geo-ca_access 55 52149399
prod_in-ams03-geo-ca_access 18 52151998
prod_in-ams03-geo-ca_access 35 52151756
prod_in-ams03-geo-ca_access 99 52146244
prod_in-ams03-geo-ca_access 41 52161567
prod_in-ams03-geo-ca_access 112 52086712
prod_in-ams03-geo-ca_access 34 52140841
prod_in-ams03-geo-ca_access 89 52160809
prod_in-ams03-geo-ca_access 40 52147177
prod_in-ams03-geo-ca_access 53 52141117
prod_in-ams03-geo-ca_access 19 52147752
prod_in-ams03-geo-ca_access 44 52169047
prod_in-ams03-geo-ca_access 31 52158455
prod_in-ams03-geo-ca_access 10 52155007
prod_in-ams03-geo-ca_access 98 52148061
prod_in-ams03-geo-ca_access 69 52156414
prod_in-ams03-geo-ca_access 92 52096106
prod_in-ams03-geo-ca_access 5 52159142
prod_in-ams03-geo-ca_access 17 52066397
prod_in-ams03-geo-ca_access 24 52149433
prod_in-ams03-geo-ca_access 28 52159909
prod_in-ams03-geo-ca_access 82 52083925
prod_in-ams03-geo-ca_access 101 52156350
prod_in-ams03-geo-ca_access 58 52156500
prod_in-ams03-geo-ca_access 94 52165527
prod_in-ams03-geo-ca_access 116 52158005
prod_in-ams03-geo-ca_access 77 52075690
prod_in-ams03-geo-ca_access 68 52152975
prod_in-ams03-geo-ca_access 64 52158277
prod_in-ams03-geo-ca_access 109 52162696
prod_in-ams03-geo-ca_access 111 52156535
prod_in-ams03-geo-ca_access 2 52083131
prod_in-ams03-geo-ca_access 74 52150817
prod_in-ams03-geo-ca_access 47 52092867
prod_in-ams03-geo-ca_access 59 52147851
prod_in-ams03-geo-ca_access 1 52154047
prod_in-ams03-geo-ca_access 104 52162658
prod_in-ams03-geo-ca_access 14 52161087
prod_in-ams03-geo-ca_access 86 52155828
prod_in-ams03-geo-ca_access 81 52158537
prod_in-ams03-geo-ca_access 11 52161597
prod_in-ams03-geo-ca_access 100 52157311
prod_in-ams03-geo-ca_access 23 52172470
prod_in-ams03-geo-ca_access 45 52154257
prod_in-ams03-geo-ca_access 6 52160103
prod_in-ams03-geo-ca_access 87 52091756
prod_in-ams03-geo-ca_access 107 52089608
prod_in-ams03-geo-ca_access 16 52149265
prod_in-ams03-geo-ca_access 20 52150732
prod_in-ams03-geo-ca_access 103 52155276
prod_in-ams03-geo-ca_access 9 52161987
prod_in-ams03-geo-ca_access 29 52153796
prod_in-ams03-geo-ca_access 54 52152245
prod_in-ams03-geo-ca_access 36 52142445
prod_in-ams03-geo-ca_access 0 52155910
prod_in-ams03-geo-ca_access 84 52159052
prod_in-ams03-geo-ca_access 78 52158632
prod_in-ams03-geo-ca_access 118 52149746
prod_in-ams03-geo-ca_access 62 52090560
prod_in-ams03-geo-ca_access 108 52157511
prod_in-ams03-geo-ca_access 88 52150632
prod_in-ams03-geo-ca_access 39 52148150
prod_in-ams03-geo-ca_access 115 52169203
prod_in-ams03-geo-ca_access 51 52153200
prod_in-ams03-geo-ca_access 3 52150514
prod_in-ams03-geo-ca_access 73 52159766
prod_in-ams03-geo-ca_access 66 52161783
prod_in-ams03-geo-ca_access 67 52078371
prod_in-ams03-geo-ca_access 97 52089805
prod_in-ams03-geo-ca_access 15 52156319
prod_in-ams03-geo-ca_access 57 52100913
prod_in-ams03-geo-ca_access 8 52161732
prod_in-ams03-geo-ca_access 114 52152390
prod_in-ams03-geo-ca_access 37 52091579
prod_in-ams03-geo-ca_access 30 52156230
prod_in-ams03-geo-ca_access 79 52146601
prod_in-ams03-geo-ca_access 12 52092444
prod_in-ams03-geo-ca_access 21 52151103
prod_in-ams03-geo-ca_access 50 52153126
prod_in-ams03-geo-ca_access 60 52147546
prod_in-ams03-geo-ca_access 33 52152862
prod_in-ams03-geo-ca_access 46 52167625
prod_in-ams03-geo-ca_access 110 52156369
prod_in-ams03-geo-ca_access 95 52157493
prod_in-ams03-geo-ca_access 71 52153659
prod_in-ams03-geo-ca_access 96 52148024
prod_in-ams03-geo-ca_access 93 52152429
prod_in-ams03-geo-ca_access 91 52162944
prod_in-ams03-geo-ca_access 43 52154651
prod_in-ams03-geo-ca_access 63 52167783
prod_in-ams03-geo-ca_access 80 52154510
prod_in-ams03-geo-ca_access 70 52157348{code}
If there is an invalid committed offset, could that maybe have caused it to not
work correctly? Even so, this should not be the result.
We really want to be able to use this tool to reset offsets to a specified date.
was:
We're seeing a problem using the kafka-consumer-groups.sh reset-offsets option
to reset offsets to a specific date/time in our production environment.
We first tried to use the kafka-consumer-groups.sh command with the
--reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 in
our staging environment and it worked correctly. Running the following command
changed it to start processing logs from February 12, 2018 (4 days ago) for a
topic that had a large lag. We did a dry run to verify before running with the
--execute option.
{code:java}
root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest
--to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic
staging-mon01-rg-elasticsearch --execute{code}
We stopped the kafka mirrors that process this topic before resetting the
offsets and started the kafka mirrors after rsetting the offsets. We verified
that it correctly started processing logs from February 12, 2018.
Then we tried resetting offsets in a production environment for a topic that
had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and it
did not work as expected. We stopped the kafka mirrors that process this topic
before resetting the offsets and did a dry run to see what the new offsets
would be:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.
^@^@^@^@
TOPIC PARTITION NEW-OFFSET
prod_in-ams03-geo-ca_access 52 52084147
prod_in-ams03-geo-ca_access 106 52154199
prod_in-ams03-geo-ca_access 75 52148673
prod_in-ams03-geo-ca_access 61 52130753
prod_in-ams03-geo-ca_access 49 52151667
prod_in-ams03-geo-ca_access 48 52145233
prod_in-ams03-geo-ca_access 27 52092805
prod_in-ams03-geo-ca_access 26 52139644
prod_in-ams03-geo-ca_access 65 52157504
prod_in-ams03-geo-ca_access 105 52166289
prod_in-ams03-geo-ca_access 38 52160464
prod_in-ams03-geo-ca_access 22 52093451
prod_in-ams03-geo-ca_access 4 52151660
prod_in-ams03-geo-ca_access 90 52160296
prod_in-ams03-geo-ca_access 25 52161691
prod_in-ams03-geo-ca_access 13 52145828
prod_in-ams03-geo-ca_access 56 52162867
prod_in-ams03-geo-ca_access 42 52072094
prod_in-ams03-geo-ca_access 7 52069496
prod_in-ams03-geo-ca_access 117 52087078
prod_in-ams03-geo-ca_access 32 52073732
prod_in-ams03-geo-ca_access 102 52082022
prod_in-ams03-geo-ca_access 76 52141018
prod_in-ams03-geo-ca_access 83 52154542
prod_in-ams03-geo-ca_access 72 52095051
prod_in-ams03-geo-ca_access 85 52149907
prod_in-ams03-geo-ca_access 119 52134435
prod_in-ams03-geo-ca_access 113 52159340
prod_in-ams03-geo-ca_access 55 52146597
prod_in-ams03-geo-ca_access 18 52149079
prod_in-ams03-geo-ca_access 35 52149058
prod_in-ams03-geo-ca_access 99 52143277
prod_in-ams03-geo-ca_access 41 52158872
prod_in-ams03-geo-ca_access 112 52083901
prod_in-ams03-geo-ca_access 34 52137932
prod_in-ams03-geo-ca_access 89 52158135
prod_in-ams03-geo-ca_access 40 52144442
prod_in-ams03-geo-ca_access 53 52138400
prod_in-ams03-geo-ca_access 19 52144966
prod_in-ams03-geo-ca_access 44 52166404
prod_in-ams03-geo-ca_access 31 52155685
prod_in-ams03-geo-ca_access 10 52152151
prod_in-ams03-geo-ca_access 98 52145378
prod_in-ams03-geo-ca_access 69 52153436
prod_in-ams03-geo-ca_access 92 52093455
prod_in-ams03-geo-ca_access 5 52156448
prod_in-ams03-geo-ca_access 17 52063697
prod_in-ams03-geo-ca_access 24 52146410
prod_in-ams03-geo-ca_access 28 52156955
prod_in-ams03-geo-ca_access 82 52081194
prod_in-ams03-geo-ca_access 101 52153592
prod_in-ams03-geo-ca_access 58 52153765
prod_in-ams03-geo-ca_access 94 52162827
prod_in-ams03-geo-ca_access 116 52155307
prod_in-ams03-geo-ca_access 77 52072906
prod_in-ams03-geo-ca_access 68 52150283
prod_in-ams03-geo-ca_access 64 52155580
prod_in-ams03-geo-ca_access 109 52159915
prod_in-ams03-geo-ca_access 111 52153547
prod_in-ams03-geo-ca_access 2 52080434
prod_in-ams03-geo-ca_access 74 52148060
prod_in-ams03-geo-ca_access 47 52090187
prod_in-ams03-geo-ca_access 59 52145161
prod_in-ams03-geo-ca_access 1 52151228
prod_in-ams03-geo-ca_access 104 52159932
prod_in-ams03-geo-ca_access 14 52158366
prod_in-ams03-geo-ca_access 86 52153030
prod_in-ams03-geo-ca_access 81 52155507
prod_in-ams03-geo-ca_access 11 52158931
prod_in-ams03-geo-ca_access 100 52154646
prod_in-ams03-geo-ca_access 23 52169713
prod_in-ams03-geo-ca_access 45 52151320
prod_in-ams03-geo-ca_access 6 52157025
prod_in-ams03-geo-ca_access 87 52088913
prod_in-ams03-geo-ca_access 107 52086878
prod_in-ams03-geo-ca_access 16 52146481
prod_in-ams03-geo-ca_access 20 52148093
prod_in-ams03-geo-ca_access 103 52152437
prod_in-ams03-geo-ca_access 9 52158998
prod_in-ams03-geo-ca_access 29 52151112
prod_in-ams03-geo-ca_access 54 52149226
prod_in-ams03-geo-ca_access 36 52139520
prod_in-ams03-geo-ca_access 0 52152902
prod_in-ams03-geo-ca_access 84 52156057
prod_in-ams03-geo-ca_access 78 52155766
prod_in-ams03-geo-ca_access 118 52147043
prod_in-ams03-geo-ca_access 62 52087864
prod_in-ams03-geo-ca_access 108 52154615
prod_in-ams03-geo-ca_access 88 52147651
prod_in-ams03-geo-ca_access 39 52145255
prod_in-ams03-geo-ca_access 115 52166466
prod_in-ams03-geo-ca_access 51 52150136
prod_in-ams03-geo-ca_access 3 52147544
prod_in-ams03-geo-ca_access 73 52157150
prod_in-ams03-geo-ca_access 66 52158808
prod_in-ams03-geo-ca_access 67 52075616
prod_in-ams03-geo-ca_access 97 52086900
prod_in-ams03-geo-ca_access 15 52153225
prod_in-ams03-geo-ca_access 57 52097899
prod_in-ams03-geo-ca_access 8 52159097
prod_in-ams03-geo-ca_access 114 52149412
prod_in-ams03-geo-ca_access 37 52088584
prod_in-ams03-geo-ca_access 30 52153155
prod_in-ams03-geo-ca_access 79 52143773
prod_in-ams03-geo-ca_access 12 52089423
prod_in-ams03-geo-ca_access 21 52148187
prod_in-ams03-geo-ca_access 50 52150414
prod_in-ams03-geo-ca_access 60 52144624
prod_in-ams03-geo-ca_access 33 52149807
prod_in-ams03-geo-ca_access 46 52165008
prod_in-ams03-geo-ca_access 110 52153643
prod_in-ams03-geo-ca_access 95 52154736
prod_in-ams03-geo-ca_access 71 52150981
prod_in-ams03-geo-ca_access 96 52145091
prod_in-ams03-geo-ca_access 93 52149375
prod_in-ams03-geo-ca_access 91 52160207
prod_in-ams03-geo-ca_access 43 52151689
prod_in-ams03-geo-ca_access 63 52164909
prod_in-ams03-geo-ca_access 80 52151810
prod_in-ams03-geo-ca_access 70 52154522{code}
Then we ran the same command with the --execute option to the command to reset
the offsets:
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-10T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access --execute{code}
And started the kafka mirrors for this topic.
However, it didn't work as expected for this topic in our production
environment.
The total log lag before resetting the offsets was 1,691,782,349:
{code:java}
root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d
%H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --describe --group
mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~
"prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }}
END{print "lag:", end_offset-current_offset;}'; sleep 15; done
2018-02-14 17:31:01 lag: 1691782349{code}
After resetting the offsets, the log lag dropped drastically to 24,330:
{code:java}
2018-02-14 17:31:17 lag: 24330{code}
This was too much of a drop in the log lag which showed it did not really reset
the offsets to start processing logs starting at Feb 10th (4 days ago). We
should be seeing a lag of around 37,170,000 if it worked correctly. Instead,
it seemed to reset the offsets to start processing logs starting today.
We also tried resetting the offsets to start processing logs starting at
February 8th (after stopping the mirrors) and it also did not work correctly.
Again, it seemed like it did a seek to the current date rather than resetting
offsets to the date we specified.
{code:java}
root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
--bootstrap-server NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest
--to-datetime 2018-02-08T00:00:00.000 --reset-offsets --topic
prod_in-ams03-geo-ca_access
Note: This will not show information about old Zookeeper-based consumers.
^@^@^@
TOPIC PARTITION NEW-OFFSET
prod_in-ams03-geo-ca_access 52 52087106
prod_in-ams03-geo-ca_access 106 52156813
prod_in-ams03-geo-ca_access 75 52151755
prod_in-ams03-geo-ca_access 61 52133608
prod_in-ams03-geo-ca_access 49 52154385
prod_in-ams03-geo-ca_access 48 52148247
prod_in-ams03-geo-ca_access 27 52095874
prod_in-ams03-geo-ca_access 26 52142331
prod_in-ams03-geo-ca_access 65 52160181
prod_in-ams03-geo-ca_access 105 52169166
prod_in-ams03-geo-ca_access 38 52163198
prod_in-ams03-geo-ca_access 22 52096231
prod_in-ams03-geo-ca_access 4 52154512
prod_in-ams03-geo-ca_access 90 52163215
prod_in-ams03-geo-ca_access 25 52164527
prod_in-ams03-geo-ca_access 13 52148553
prod_in-ams03-geo-ca_access 56 52165558
prod_in-ams03-geo-ca_access 42 52075104
prod_in-ams03-geo-ca_access 7 52072333
prod_in-ams03-geo-ca_access 117 52090029
prod_in-ams03-geo-ca_access 32 52076469
prod_in-ams03-geo-ca_access 102 52085066
prod_in-ams03-geo-ca_access 76 52143820
prod_in-ams03-geo-ca_access 83 52157217
prod_in-ams03-geo-ca_access 72 52098060
prod_in-ams03-geo-ca_access 85 52152612
prod_in-ams03-geo-ca_access 119 52137225
prod_in-ams03-geo-ca_access 113 52162012
prod_in-ams03-geo-ca_access 55 52149399
prod_in-ams03-geo-ca_access 18 52151998
prod_in-ams03-geo-ca_access 35 52151756
prod_in-ams03-geo-ca_access 99 52146244
prod_in-ams03-geo-ca_access 41 52161567
prod_in-ams03-geo-ca_access 112 52086712
prod_in-ams03-geo-ca_access 34 52140841
prod_in-ams03-geo-ca_access 89 52160809
prod_in-ams03-geo-ca_access 40 52147177
prod_in-ams03-geo-ca_access 53 52141117
prod_in-ams03-geo-ca_access 19 52147752
prod_in-ams03-geo-ca_access 44 52169047
prod_in-ams03-geo-ca_access 31 52158455
prod_in-ams03-geo-ca_access 10 52155007
prod_in-ams03-geo-ca_access 98 52148061
prod_in-ams03-geo-ca_access 69 52156414
prod_in-ams03-geo-ca_access 92 52096106
prod_in-ams03-geo-ca_access 5 52159142
prod_in-ams03-geo-ca_access 17 52066397
prod_in-ams03-geo-ca_access 24 52149433
prod_in-ams03-geo-ca_access 28 52159909
prod_in-ams03-geo-ca_access 82 52083925
prod_in-ams03-geo-ca_access 101 52156350
prod_in-ams03-geo-ca_access 58 52156500
prod_in-ams03-geo-ca_access 94 52165527
prod_in-ams03-geo-ca_access 116 52158005
prod_in-ams03-geo-ca_access 77 52075690
prod_in-ams03-geo-ca_access 68 52152975
prod_in-ams03-geo-ca_access 64 52158277
prod_in-ams03-geo-ca_access 109 52162696
prod_in-ams03-geo-ca_access 111 52156535
prod_in-ams03-geo-ca_access 2 52083131
prod_in-ams03-geo-ca_access 74 52150817
prod_in-ams03-geo-ca_access 47 52092867
prod_in-ams03-geo-ca_access 59 52147851
prod_in-ams03-geo-ca_access 1 52154047
prod_in-ams03-geo-ca_access 104 52162658
prod_in-ams03-geo-ca_access 14 52161087
prod_in-ams03-geo-ca_access 86 52155828
prod_in-ams03-geo-ca_access 81 52158537
prod_in-ams03-geo-ca_access 11 52161597
prod_in-ams03-geo-ca_access 100 52157311
prod_in-ams03-geo-ca_access 23 52172470
prod_in-ams03-geo-ca_access 45 52154257
prod_in-ams03-geo-ca_access 6 52160103
prod_in-ams03-geo-ca_access 87 52091756
prod_in-ams03-geo-ca_access 107 52089608
prod_in-ams03-geo-ca_access 16 52149265
prod_in-ams03-geo-ca_access 20 52150732
prod_in-ams03-geo-ca_access 103 52155276
prod_in-ams03-geo-ca_access 9 52161987
prod_in-ams03-geo-ca_access 29 52153796
prod_in-ams03-geo-ca_access 54 52152245
prod_in-ams03-geo-ca_access 36 52142445
prod_in-ams03-geo-ca_access 0 52155910
prod_in-ams03-geo-ca_access 84 52159052
prod_in-ams03-geo-ca_access 78 52158632
prod_in-ams03-geo-ca_access 118 52149746
prod_in-ams03-geo-ca_access 62 52090560
prod_in-ams03-geo-ca_access 108 52157511
prod_in-ams03-geo-ca_access 88 52150632
prod_in-ams03-geo-ca_access 39 52148150
prod_in-ams03-geo-ca_access 115 52169203
prod_in-ams03-geo-ca_access 51 52153200
prod_in-ams03-geo-ca_access 3 52150514
prod_in-ams03-geo-ca_access 73 52159766
prod_in-ams03-geo-ca_access 66 52161783
prod_in-ams03-geo-ca_access 67 52078371
prod_in-ams03-geo-ca_access 97 52089805
prod_in-ams03-geo-ca_access 15 52156319
prod_in-ams03-geo-ca_access 57 52100913
prod_in-ams03-geo-ca_access 8 52161732
prod_in-ams03-geo-ca_access 114 52152390
prod_in-ams03-geo-ca_access 37 52091579
prod_in-ams03-geo-ca_access 30 52156230
prod_in-ams03-geo-ca_access 79 52146601
prod_in-ams03-geo-ca_access 12 52092444
prod_in-ams03-geo-ca_access 21 52151103
prod_in-ams03-geo-ca_access 50 52153126
prod_in-ams03-geo-ca_access 60 52147546
prod_in-ams03-geo-ca_access 33 52152862
prod_in-ams03-geo-ca_access 46 52167625
prod_in-ams03-geo-ca_access 110 52156369
prod_in-ams03-geo-ca_access 95 52157493
prod_in-ams03-geo-ca_access 71 52153659
prod_in-ams03-geo-ca_access 96 52148024
prod_in-ams03-geo-ca_access 93 52152429
prod_in-ams03-geo-ca_access 91 52162944
prod_in-ams03-geo-ca_access 43 52154651
prod_in-ams03-geo-ca_access 63 52167783
prod_in-ams03-geo-ca_access 80 52154510
prod_in-ams03-geo-ca_access 70 52157348{code}
If there is an invalid committed offset, could that maybe have caused it to not
work correctly? Even so, this should not be the result.
We really want to be able to use this tool to reset offsets to a specified date.
> kafka-consumer-groups does not reset offsets to specified datetime correctly
> ----------------------------------------------------------------------------
>
> Key: KAFKA-6572
> URL: https://issues.apache.org/jira/browse/KAFKA-6572
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Sharon Lucas
> Priority: Major
>
> Using Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh
> reset-offsets option to reset offsets to a specific date/time in our
> production environment.
> We first tried to use the kafka-consumer-groups.sh command with the
> --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000
> in our staging environment and it worked correctly. Running the following
> command changed it to start processing logs from February 12, 2018 (4 days
> ago) for a topic that had a large lag. We did a dry run to verify before
> running with the --execute option.
> {code:java}
> root@ossmlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest
> --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic
> staging-mon01-rg-elasticsearch --execute{code}
> We stopped the kafka mirrors that process this topic before resetting the
> offsets and started the kafka mirrors after rsetting the offsets. We
> verified that it correctly started processing logs from February 12, 2018.
> Then we tried resetting offsets in a production environment for a topic that
> had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and
> it did not work as expected. We stopped the kafka mirrors that process this
> topic before resetting the offsets and did a dry run to see what the new
> offsets would be:
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.N.NNN.NNN:9092 --group
> mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000
> --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@^@
> TOPIC PARTITION NEW-OFFSET
> prod_in-ams03-geo-ca_access 52 52084147
> prod_in-ams03-geo-ca_access 106 52154199
> prod_in-ams03-geo-ca_access 75 52148673
> prod_in-ams03-geo-ca_access 61 52130753
> prod_in-ams03-geo-ca_access 49 52151667
> prod_in-ams03-geo-ca_access 48 52145233
> prod_in-ams03-geo-ca_access 27 52092805
> prod_in-ams03-geo-ca_access 26 52139644
> prod_in-ams03-geo-ca_access 65 52157504
> prod_in-ams03-geo-ca_access 105 52166289
> prod_in-ams03-geo-ca_access 38 52160464
> prod_in-ams03-geo-ca_access 22 52093451
> prod_in-ams03-geo-ca_access 4 52151660
> prod_in-ams03-geo-ca_access 90 52160296
> prod_in-ams03-geo-ca_access 25 52161691
> prod_in-ams03-geo-ca_access 13 52145828
> prod_in-ams03-geo-ca_access 56 52162867
> prod_in-ams03-geo-ca_access 42 52072094
> prod_in-ams03-geo-ca_access 7 52069496
> prod_in-ams03-geo-ca_access 117 52087078
> prod_in-ams03-geo-ca_access 32 52073732
> prod_in-ams03-geo-ca_access 102 52082022
> prod_in-ams03-geo-ca_access 76 52141018
> prod_in-ams03-geo-ca_access 83 52154542
> prod_in-ams03-geo-ca_access 72 52095051
> prod_in-ams03-geo-ca_access 85 52149907
> prod_in-ams03-geo-ca_access 119 52134435
> prod_in-ams03-geo-ca_access 113 52159340
> prod_in-ams03-geo-ca_access 55 52146597
> prod_in-ams03-geo-ca_access 18 52149079
> prod_in-ams03-geo-ca_access 35 52149058
> prod_in-ams03-geo-ca_access 99 52143277
> prod_in-ams03-geo-ca_access 41 52158872
> prod_in-ams03-geo-ca_access 112 52083901
> prod_in-ams03-geo-ca_access 34 52137932
> prod_in-ams03-geo-ca_access 89 52158135
> prod_in-ams03-geo-ca_access 40 52144442
> prod_in-ams03-geo-ca_access 53 52138400
> prod_in-ams03-geo-ca_access 19 52144966
> prod_in-ams03-geo-ca_access 44 52166404
> prod_in-ams03-geo-ca_access 31 52155685
> prod_in-ams03-geo-ca_access 10 52152151
> prod_in-ams03-geo-ca_access 98 52145378
> prod_in-ams03-geo-ca_access 69 52153436
> prod_in-ams03-geo-ca_access 92 52093455
> prod_in-ams03-geo-ca_access 5 52156448
> prod_in-ams03-geo-ca_access 17 52063697
> prod_in-ams03-geo-ca_access 24 52146410
> prod_in-ams03-geo-ca_access 28 52156955
> prod_in-ams03-geo-ca_access 82 52081194
> prod_in-ams03-geo-ca_access 101 52153592
> prod_in-ams03-geo-ca_access 58 52153765
> prod_in-ams03-geo-ca_access 94 52162827
> prod_in-ams03-geo-ca_access 116 52155307
> prod_in-ams03-geo-ca_access 77 52072906
> prod_in-ams03-geo-ca_access 68 52150283
> prod_in-ams03-geo-ca_access 64 52155580
> prod_in-ams03-geo-ca_access 109 52159915
> prod_in-ams03-geo-ca_access 111 52153547
> prod_in-ams03-geo-ca_access 2 52080434
> prod_in-ams03-geo-ca_access 74 52148060
> prod_in-ams03-geo-ca_access 47 52090187
> prod_in-ams03-geo-ca_access 59 52145161
> prod_in-ams03-geo-ca_access 1 52151228
> prod_in-ams03-geo-ca_access 104 52159932
> prod_in-ams03-geo-ca_access 14 52158366
> prod_in-ams03-geo-ca_access 86 52153030
> prod_in-ams03-geo-ca_access 81 52155507
> prod_in-ams03-geo-ca_access 11 52158931
> prod_in-ams03-geo-ca_access 100 52154646
> prod_in-ams03-geo-ca_access 23 52169713
> prod_in-ams03-geo-ca_access 45 52151320
> prod_in-ams03-geo-ca_access 6 52157025
> prod_in-ams03-geo-ca_access 87 52088913
> prod_in-ams03-geo-ca_access 107 52086878
> prod_in-ams03-geo-ca_access 16 52146481
> prod_in-ams03-geo-ca_access 20 52148093
> prod_in-ams03-geo-ca_access 103 52152437
> prod_in-ams03-geo-ca_access 9 52158998
> prod_in-ams03-geo-ca_access 29 52151112
> prod_in-ams03-geo-ca_access 54 52149226
> prod_in-ams03-geo-ca_access 36 52139520
> prod_in-ams03-geo-ca_access 0 52152902
> prod_in-ams03-geo-ca_access 84 52156057
> prod_in-ams03-geo-ca_access 78 52155766
> prod_in-ams03-geo-ca_access 118 52147043
> prod_in-ams03-geo-ca_access 62 52087864
> prod_in-ams03-geo-ca_access 108 52154615
> prod_in-ams03-geo-ca_access 88 52147651
> prod_in-ams03-geo-ca_access 39 52145255
> prod_in-ams03-geo-ca_access 115 52166466
> prod_in-ams03-geo-ca_access 51 52150136
> prod_in-ams03-geo-ca_access 3 52147544
> prod_in-ams03-geo-ca_access 73 52157150
> prod_in-ams03-geo-ca_access 66 52158808
> prod_in-ams03-geo-ca_access 67 52075616
> prod_in-ams03-geo-ca_access 97 52086900
> prod_in-ams03-geo-ca_access 15 52153225
> prod_in-ams03-geo-ca_access 57 52097899
> prod_in-ams03-geo-ca_access 8 52159097
> prod_in-ams03-geo-ca_access 114 52149412
> prod_in-ams03-geo-ca_access 37 52088584
> prod_in-ams03-geo-ca_access 30 52153155
> prod_in-ams03-geo-ca_access 79 52143773
> prod_in-ams03-geo-ca_access 12 52089423
> prod_in-ams03-geo-ca_access 21 52148187
> prod_in-ams03-geo-ca_access 50 52150414
> prod_in-ams03-geo-ca_access 60 52144624
> prod_in-ams03-geo-ca_access 33 52149807
> prod_in-ams03-geo-ca_access 46 52165008
> prod_in-ams03-geo-ca_access 110 52153643
> prod_in-ams03-geo-ca_access 95 52154736
> prod_in-ams03-geo-ca_access 71 52150981
> prod_in-ams03-geo-ca_access 96 52145091
> prod_in-ams03-geo-ca_access 93 52149375
> prod_in-ams03-geo-ca_access 91 52160207
> prod_in-ams03-geo-ca_access 43 52151689
> prod_in-ams03-geo-ca_access 63 52164909
> prod_in-ams03-geo-ca_access 80 52151810
> prod_in-ams03-geo-ca_access 70 52154522{code}
> Then we ran the same command with the --execute option to the command to
> reset the offsets:
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.N.NNN.NNN:9092 --group
> mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-10T00:00:00.000
> --reset-offsets --topic prod_in-ams03-geo-ca_access --execute{code}
> And started the kafka mirrors for this topic.
> However, it didn't work as expected for this topic in our production
> environment.
> The total log lag before resetting the offsets was 1,691,782,349:
> {code:java}
> root@ossmlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d
> %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.N.NNN.NNN:9092 --describe --group
> mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~
> "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }}
> END{print "lag:", end_offset-current_offset;}'; sleep 15; done
> 2018-02-14 17:31:01 lag: 1691782349{code}
> After resetting the offsets, the log lag dropped drastically to 24,330:
> {code:java}
> 2018-02-14 17:31:17 lag: 24330{code}
> This was too much of a drop in the log lag which showed it did not really
> reset the offsets to start processing logs starting at Feb 10th (4 days ago).
> We should be seeing a lag of around 37,170,000 if it worked correctly.
> Instead, it seemed to reset the offsets to start processing logs starting
> today.
> We also tried resetting the offsets to start processing logs starting at
> February 8th (after stopping the mirrors) and it also did not work correctly.
> Again, it seemed like it did a seek to the current date rather than
> resetting offsets to the date we specified.
> {code:java}
> root@ossmlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh
> --bootstrap-server NN.N.NNN.NNN:9092 --group
> mirror-consumer-ams03-geo-earliest --to-datetime 2018-02-08T00:00:00.000
> --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@
> TOPIC PARTITION NEW-OFFSET
> prod_in-ams03-geo-ca_access 52 52087106
> prod_in-ams03-geo-ca_access 106 52156813
> prod_in-ams03-geo-ca_access 75 52151755
> prod_in-ams03-geo-ca_access 61 52133608
> prod_in-ams03-geo-ca_access 49 52154385
> prod_in-ams03-geo-ca_access 48 52148247
> prod_in-ams03-geo-ca_access 27 52095874
> prod_in-ams03-geo-ca_access 26 52142331
> prod_in-ams03-geo-ca_access 65 52160181
> prod_in-ams03-geo-ca_access 105 52169166
> prod_in-ams03-geo-ca_access 38 52163198
> prod_in-ams03-geo-ca_access 22 52096231
> prod_in-ams03-geo-ca_access 4 52154512
> prod_in-ams03-geo-ca_access 90 52163215
> prod_in-ams03-geo-ca_access 25 52164527
> prod_in-ams03-geo-ca_access 13 52148553
> prod_in-ams03-geo-ca_access 56 52165558
> prod_in-ams03-geo-ca_access 42 52075104
> prod_in-ams03-geo-ca_access 7 52072333
> prod_in-ams03-geo-ca_access 117 52090029
> prod_in-ams03-geo-ca_access 32 52076469
> prod_in-ams03-geo-ca_access 102 52085066
> prod_in-ams03-geo-ca_access 76 52143820
> prod_in-ams03-geo-ca_access 83 52157217
> prod_in-ams03-geo-ca_access 72 52098060
> prod_in-ams03-geo-ca_access 85 52152612
> prod_in-ams03-geo-ca_access 119 52137225
> prod_in-ams03-geo-ca_access 113 52162012
> prod_in-ams03-geo-ca_access 55 52149399
> prod_in-ams03-geo-ca_access 18 52151998
> prod_in-ams03-geo-ca_access 35 52151756
> prod_in-ams03-geo-ca_access 99 52146244
> prod_in-ams03-geo-ca_access 41 52161567
> prod_in-ams03-geo-ca_access 112 52086712
> prod_in-ams03-geo-ca_access 34 52140841
> prod_in-ams03-geo-ca_access 89 52160809
> prod_in-ams03-geo-ca_access 40 52147177
> prod_in-ams03-geo-ca_access 53 52141117
> prod_in-ams03-geo-ca_access 19 52147752
> prod_in-ams03-geo-ca_access 44 52169047
> prod_in-ams03-geo-ca_access 31 52158455
> prod_in-ams03-geo-ca_access 10 52155007
> prod_in-ams03-geo-ca_access 98 52148061
> prod_in-ams03-geo-ca_access 69 52156414
> prod_in-ams03-geo-ca_access 92 52096106
> prod_in-ams03-geo-ca_access 5 52159142
> prod_in-ams03-geo-ca_access 17 52066397
> prod_in-ams03-geo-ca_access 24 52149433
> prod_in-ams03-geo-ca_access 28 52159909
> prod_in-ams03-geo-ca_access 82 52083925
> prod_in-ams03-geo-ca_access 101 52156350
> prod_in-ams03-geo-ca_access 58 52156500
> prod_in-ams03-geo-ca_access 94 52165527
> prod_in-ams03-geo-ca_access 116 52158005
> prod_in-ams03-geo-ca_access 77 52075690
> prod_in-ams03-geo-ca_access 68 52152975
> prod_in-ams03-geo-ca_access 64 52158277
> prod_in-ams03-geo-ca_access 109 52162696
> prod_in-ams03-geo-ca_access 111 52156535
> prod_in-ams03-geo-ca_access 2 52083131
> prod_in-ams03-geo-ca_access 74 52150817
> prod_in-ams03-geo-ca_access 47 52092867
> prod_in-ams03-geo-ca_access 59 52147851
> prod_in-ams03-geo-ca_access 1 52154047
> prod_in-ams03-geo-ca_access 104 52162658
> prod_in-ams03-geo-ca_access 14 52161087
> prod_in-ams03-geo-ca_access 86 52155828
> prod_in-ams03-geo-ca_access 81 52158537
> prod_in-ams03-geo-ca_access 11 52161597
> prod_in-ams03-geo-ca_access 100 52157311
> prod_in-ams03-geo-ca_access 23 52172470
> prod_in-ams03-geo-ca_access 45 52154257
> prod_in-ams03-geo-ca_access 6 52160103
> prod_in-ams03-geo-ca_access 87 52091756
> prod_in-ams03-geo-ca_access 107 52089608
> prod_in-ams03-geo-ca_access 16 52149265
> prod_in-ams03-geo-ca_access 20 52150732
> prod_in-ams03-geo-ca_access 103 52155276
> prod_in-ams03-geo-ca_access 9 52161987
> prod_in-ams03-geo-ca_access 29 52153796
> prod_in-ams03-geo-ca_access 54 52152245
> prod_in-ams03-geo-ca_access 36 52142445
> prod_in-ams03-geo-ca_access 0 52155910
> prod_in-ams03-geo-ca_access 84 52159052
> prod_in-ams03-geo-ca_access 78 52158632
> prod_in-ams03-geo-ca_access 118 52149746
> prod_in-ams03-geo-ca_access 62 52090560
> prod_in-ams03-geo-ca_access 108 52157511
> prod_in-ams03-geo-ca_access 88 52150632
> prod_in-ams03-geo-ca_access 39 52148150
> prod_in-ams03-geo-ca_access 115 52169203
> prod_in-ams03-geo-ca_access 51 52153200
> prod_in-ams03-geo-ca_access 3 52150514
> prod_in-ams03-geo-ca_access 73 52159766
> prod_in-ams03-geo-ca_access 66 52161783
> prod_in-ams03-geo-ca_access 67 52078371
> prod_in-ams03-geo-ca_access 97 52089805
> prod_in-ams03-geo-ca_access 15 52156319
> prod_in-ams03-geo-ca_access 57 52100913
> prod_in-ams03-geo-ca_access 8 52161732
> prod_in-ams03-geo-ca_access 114 52152390
> prod_in-ams03-geo-ca_access 37 52091579
> prod_in-ams03-geo-ca_access 30 52156230
> prod_in-ams03-geo-ca_access 79 52146601
> prod_in-ams03-geo-ca_access 12 52092444
> prod_in-ams03-geo-ca_access 21 52151103
> prod_in-ams03-geo-ca_access 50 52153126
> prod_in-ams03-geo-ca_access 60 52147546
> prod_in-ams03-geo-ca_access 33 52152862
> prod_in-ams03-geo-ca_access 46 52167625
> prod_in-ams03-geo-ca_access 110 52156369
> prod_in-ams03-geo-ca_access 95 52157493
> prod_in-ams03-geo-ca_access 71 52153659
> prod_in-ams03-geo-ca_access 96 52148024
> prod_in-ams03-geo-ca_access 93 52152429
> prod_in-ams03-geo-ca_access 91 52162944
> prod_in-ams03-geo-ca_access 43 52154651
> prod_in-ams03-geo-ca_access 63 52167783
> prod_in-ams03-geo-ca_access 80 52154510
> prod_in-ams03-geo-ca_access 70 52157348{code}
> If there is an invalid committed offset, could that maybe have caused it to
> not work correctly? Even so, this should not be the result.
> We really want to be able to use this tool to reset offsets to a specified
> date.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
