[ 
https://issues.apache.org/jira/browse/KAFKA-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372986#comment-16372986
 ] 

Sharon Lucas commented on KAFKA-6572:
-------------------------------------

No.  There were tons of messages whose timestamp was greater than the timestamp 
we specified.

> kafka-consumer-groups does not reset offsets to specified datetime correctly
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-6572
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6572
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 1.0.0
>            Reporter: Sharon Lucas
>            Priority: Major
>
> With Kafka 1.0.0 we are seeing a problem using the kafka-consumer-groups.sh 
> --reset-offsets option to reset offsets to a specific date/time in our 
> production environment.
> We first tried to use the kafka-consumer-groups.sh command with the 
> --reset-offsets option and with option --to-datetime 2018-02-10T00:00:00.000 
> in our staging environment and it worked correctly.  Running the following 
> command changed it to start processing logs from February 12, 2018 (4 days 
> ago) for a topic that had a large lag.  We did a dry run to verify before 
> running with the --execute option.
> {code:java}
> root@mlpstagemon0101a:/# /opt/kafka/bin/kafka-consumer-groups.sh 
> --bootstrap-server NN.NNN.NN.NN:9092 --group logstash-elasticsearch-latest 
> --to-datetime 2018-02-12T00:00:00.000-06:00 --reset-offsets --topic 
> staging-mon01-rg-elasticsearch --execute{code}
> We stopped the kafka mirrors that process this topic before resetting the 
> offsets and started the kafka mirrors after rsetting the offsets.  We 
> verified that it correctly started processing logs from February 12, 2018.
> Then we tried resetting offsets in a production environment for a topic that 
> had a very large lag using option --to-datetime 2018-02-10T00:00:00.000 and 
> it did not work as expected. We stopped the kafka mirrors that process this 
> topic before resetting the offsets and did a dry run to see what the new 
> offsets would be:
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 
> 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@^@
> TOPIC                          PARTITION  NEW-OFFSET
> prod_in-ams03-geo-ca_access    52         52084147
> prod_in-ams03-geo-ca_access    106        52154199
> prod_in-ams03-geo-ca_access    75         52148673
> prod_in-ams03-geo-ca_access    61         52130753
> prod_in-ams03-geo-ca_access    49         52151667
> prod_in-ams03-geo-ca_access    48         52145233
> prod_in-ams03-geo-ca_access    27         52092805
> prod_in-ams03-geo-ca_access    26         52139644
> prod_in-ams03-geo-ca_access    65         52157504
> prod_in-ams03-geo-ca_access    105        52166289
> prod_in-ams03-geo-ca_access    38         52160464
> prod_in-ams03-geo-ca_access    22         52093451
> prod_in-ams03-geo-ca_access    4          52151660
> prod_in-ams03-geo-ca_access    90         52160296
> prod_in-ams03-geo-ca_access    25         52161691
> prod_in-ams03-geo-ca_access    13         52145828
> prod_in-ams03-geo-ca_access    56         52162867
> prod_in-ams03-geo-ca_access    42         52072094
> prod_in-ams03-geo-ca_access    7          52069496
> prod_in-ams03-geo-ca_access    117        52087078
> prod_in-ams03-geo-ca_access    32         52073732
> prod_in-ams03-geo-ca_access    102        52082022
> prod_in-ams03-geo-ca_access    76         52141018
> prod_in-ams03-geo-ca_access    83         52154542
> prod_in-ams03-geo-ca_access    72         52095051
> prod_in-ams03-geo-ca_access    85         52149907
> prod_in-ams03-geo-ca_access    119        52134435
> prod_in-ams03-geo-ca_access    113        52159340
> prod_in-ams03-geo-ca_access    55         52146597
> prod_in-ams03-geo-ca_access    18         52149079
> prod_in-ams03-geo-ca_access    35         52149058
> prod_in-ams03-geo-ca_access    99         52143277
> prod_in-ams03-geo-ca_access    41         52158872
> prod_in-ams03-geo-ca_access    112        52083901
> prod_in-ams03-geo-ca_access    34         52137932
> prod_in-ams03-geo-ca_access    89         52158135
> prod_in-ams03-geo-ca_access    40         52144442
> prod_in-ams03-geo-ca_access    53         52138400
> prod_in-ams03-geo-ca_access    19         52144966
> prod_in-ams03-geo-ca_access    44         52166404
> prod_in-ams03-geo-ca_access    31         52155685
> prod_in-ams03-geo-ca_access    10         52152151
> prod_in-ams03-geo-ca_access    98         52145378
> prod_in-ams03-geo-ca_access    69         52153436
> prod_in-ams03-geo-ca_access    92         52093455
> prod_in-ams03-geo-ca_access    5          52156448
> prod_in-ams03-geo-ca_access    17         52063697
> prod_in-ams03-geo-ca_access    24         52146410
> prod_in-ams03-geo-ca_access    28         52156955
> prod_in-ams03-geo-ca_access    82         52081194
> prod_in-ams03-geo-ca_access    101        52153592
> prod_in-ams03-geo-ca_access    58         52153765
> prod_in-ams03-geo-ca_access    94         52162827
> prod_in-ams03-geo-ca_access    116        52155307
> prod_in-ams03-geo-ca_access    77         52072906
> prod_in-ams03-geo-ca_access    68         52150283
> prod_in-ams03-geo-ca_access    64         52155580
> prod_in-ams03-geo-ca_access    109        52159915
> prod_in-ams03-geo-ca_access    111        52153547
> prod_in-ams03-geo-ca_access    2          52080434
> prod_in-ams03-geo-ca_access    74         52148060
> prod_in-ams03-geo-ca_access    47         52090187
> prod_in-ams03-geo-ca_access    59         52145161
> prod_in-ams03-geo-ca_access    1          52151228
> prod_in-ams03-geo-ca_access    104        52159932
> prod_in-ams03-geo-ca_access    14         52158366
> prod_in-ams03-geo-ca_access    86         52153030
> prod_in-ams03-geo-ca_access    81         52155507
> prod_in-ams03-geo-ca_access    11         52158931
> prod_in-ams03-geo-ca_access    100        52154646
> prod_in-ams03-geo-ca_access    23         52169713
> prod_in-ams03-geo-ca_access    45         52151320
> prod_in-ams03-geo-ca_access    6          52157025
> prod_in-ams03-geo-ca_access    87         52088913
> prod_in-ams03-geo-ca_access    107        52086878
> prod_in-ams03-geo-ca_access    16         52146481
> prod_in-ams03-geo-ca_access    20         52148093
> prod_in-ams03-geo-ca_access    103        52152437
> prod_in-ams03-geo-ca_access    9          52158998
> prod_in-ams03-geo-ca_access    29         52151112
> prod_in-ams03-geo-ca_access    54         52149226
> prod_in-ams03-geo-ca_access    36         52139520
> prod_in-ams03-geo-ca_access    0          52152902
> prod_in-ams03-geo-ca_access    84         52156057
> prod_in-ams03-geo-ca_access    78         52155766
> prod_in-ams03-geo-ca_access    118        52147043
> prod_in-ams03-geo-ca_access    62         52087864
> prod_in-ams03-geo-ca_access    108        52154615
> prod_in-ams03-geo-ca_access    88         52147651
> prod_in-ams03-geo-ca_access    39         52145255
> prod_in-ams03-geo-ca_access    115        52166466
> prod_in-ams03-geo-ca_access    51         52150136
> prod_in-ams03-geo-ca_access    3          52147544
> prod_in-ams03-geo-ca_access    73         52157150
> prod_in-ams03-geo-ca_access    66         52158808
> prod_in-ams03-geo-ca_access    67         52075616
> prod_in-ams03-geo-ca_access    97         52086900
> prod_in-ams03-geo-ca_access    15         52153225
> prod_in-ams03-geo-ca_access    57         52097899
> prod_in-ams03-geo-ca_access    8          52159097
> prod_in-ams03-geo-ca_access    114        52149412
> prod_in-ams03-geo-ca_access    37         52088584
> prod_in-ams03-geo-ca_access    30         52153155
> prod_in-ams03-geo-ca_access    79         52143773
> prod_in-ams03-geo-ca_access    12         52089423
> prod_in-ams03-geo-ca_access    21         52148187
> prod_in-ams03-geo-ca_access    50         52150414
> prod_in-ams03-geo-ca_access    60         52144624
> prod_in-ams03-geo-ca_access    33         52149807
> prod_in-ams03-geo-ca_access    46         52165008
> prod_in-ams03-geo-ca_access    110        52153643
> prod_in-ams03-geo-ca_access    95         52154736
> prod_in-ams03-geo-ca_access    71         52150981
> prod_in-ams03-geo-ca_access    96         52145091
> prod_in-ams03-geo-ca_access    93         52149375
> prod_in-ams03-geo-ca_access    91         52160207
> prod_in-ams03-geo-ca_access    43         52151689
> prod_in-ams03-geo-ca_access    63         52164909
> prod_in-ams03-geo-ca_access    80         52151810
> prod_in-ams03-geo-ca_access    70         52154522{code}
> Then we ran the same command with the --execute option to the command to 
> reset the offsets:
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 
> 2018-02-10T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access 
> --execute{code}
> And started the kafka mirrors for this topic.
> However, it didn't work as expected for this topic in our production 
> environment.
>  The total log lag before resetting the offsets was 1,691,782,349:
> {code:java}
> root@mlbams0301a:/var/log/kafka# while true; do echo -n `date +'%Y-%m-%d 
> %H:%M:%S'`; echo -n " "; /opt/kafka/bin/kafka-consumer-groups.sh 
> --bootstrap-server NN.N.NNN.NNN:9092 --describe --group 
> mirror-consumer-ams03-geo-earliest 2>/dev/null | awk '{if ($1 ~ 
> "prod_in-ams03-geo-ca_access") { current_offset+=$3; end_offset+=$4; }} 
> END{print "lag:", end_offset-current_offset;}'; sleep 15; done
> 2018-02-14 17:31:01 lag: 1691782349{code}
> After resetting the offsets, the log lag dropped to 24,330:
> {code:java}
> 2018-02-14 17:31:17 lag: 24330{code}
> This was too much of a drop in the log lag.  This shows it did not really 
> reset the offsets to start processing logs starting at Feb 10th (4 days ago). 
>  We should be seeing a lag of around 37,170,000 if it worked correctly.  
> Instead, it seemed to reset the offsets to start processing logs starting 
> today (Feb 14th).
> We also tried resetting the offsets to start processing logs starting at 
> February 8th (after stopping the mirrors) and it also did not work correctly. 
>  Again, it seemed like it did a seek to the current date rather than 
> resetting offsets to the date we specified.
> {code:java}
> root@mlplon0401e:# /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server 
> NN.N.NNN.NNN:9092 --group mirror-consumer-ams03-geo-earliest --to-datetime 
> 2018-02-08T00:00:00.000 --reset-offsets --topic prod_in-ams03-geo-ca_access
> Note: This will not show information about old Zookeeper-based consumers.
> ^@^@^@
> TOPIC                          PARTITION  NEW-OFFSET
> prod_in-ams03-geo-ca_access    52         52087106
> prod_in-ams03-geo-ca_access    106        52156813
> prod_in-ams03-geo-ca_access    75         52151755
> prod_in-ams03-geo-ca_access    61         52133608
> prod_in-ams03-geo-ca_access    49         52154385
> prod_in-ams03-geo-ca_access    48         52148247
> prod_in-ams03-geo-ca_access    27         52095874
> prod_in-ams03-geo-ca_access    26         52142331
> prod_in-ams03-geo-ca_access    65         52160181
> prod_in-ams03-geo-ca_access    105        52169166
> prod_in-ams03-geo-ca_access    38         52163198
> prod_in-ams03-geo-ca_access    22         52096231
> prod_in-ams03-geo-ca_access    4          52154512
> prod_in-ams03-geo-ca_access    90         52163215
> prod_in-ams03-geo-ca_access    25         52164527
> prod_in-ams03-geo-ca_access    13         52148553
> prod_in-ams03-geo-ca_access    56         52165558
> prod_in-ams03-geo-ca_access    42         52075104
> prod_in-ams03-geo-ca_access    7          52072333
> prod_in-ams03-geo-ca_access    117        52090029
> prod_in-ams03-geo-ca_access    32         52076469
> prod_in-ams03-geo-ca_access    102        52085066
> prod_in-ams03-geo-ca_access    76         52143820
> prod_in-ams03-geo-ca_access    83         52157217
> prod_in-ams03-geo-ca_access    72         52098060
> prod_in-ams03-geo-ca_access    85         52152612
> prod_in-ams03-geo-ca_access    119        52137225
> prod_in-ams03-geo-ca_access    113        52162012
> prod_in-ams03-geo-ca_access    55         52149399
> prod_in-ams03-geo-ca_access    18         52151998
> prod_in-ams03-geo-ca_access    35         52151756
> prod_in-ams03-geo-ca_access    99         52146244
> prod_in-ams03-geo-ca_access    41         52161567
> prod_in-ams03-geo-ca_access    112        52086712
> prod_in-ams03-geo-ca_access    34         52140841
> prod_in-ams03-geo-ca_access    89         52160809
> prod_in-ams03-geo-ca_access    40         52147177
> prod_in-ams03-geo-ca_access    53         52141117
> prod_in-ams03-geo-ca_access    19         52147752
> prod_in-ams03-geo-ca_access    44         52169047
> prod_in-ams03-geo-ca_access    31         52158455
> prod_in-ams03-geo-ca_access    10         52155007
> prod_in-ams03-geo-ca_access    98         52148061
> prod_in-ams03-geo-ca_access    69         52156414
> prod_in-ams03-geo-ca_access    92         52096106
> prod_in-ams03-geo-ca_access    5          52159142
> prod_in-ams03-geo-ca_access    17         52066397
> prod_in-ams03-geo-ca_access    24         52149433
> prod_in-ams03-geo-ca_access    28         52159909
> prod_in-ams03-geo-ca_access    82         52083925
> prod_in-ams03-geo-ca_access    101        52156350
> prod_in-ams03-geo-ca_access    58         52156500
> prod_in-ams03-geo-ca_access    94         52165527
> prod_in-ams03-geo-ca_access    116        52158005
> prod_in-ams03-geo-ca_access    77         52075690
> prod_in-ams03-geo-ca_access    68         52152975
> prod_in-ams03-geo-ca_access    64         52158277
> prod_in-ams03-geo-ca_access    109        52162696
> prod_in-ams03-geo-ca_access    111        52156535
> prod_in-ams03-geo-ca_access    2          52083131
> prod_in-ams03-geo-ca_access    74         52150817
> prod_in-ams03-geo-ca_access    47         52092867
> prod_in-ams03-geo-ca_access    59         52147851
> prod_in-ams03-geo-ca_access    1          52154047
> prod_in-ams03-geo-ca_access    104        52162658
> prod_in-ams03-geo-ca_access    14         52161087
> prod_in-ams03-geo-ca_access    86         52155828
> prod_in-ams03-geo-ca_access    81         52158537
> prod_in-ams03-geo-ca_access    11         52161597
> prod_in-ams03-geo-ca_access    100        52157311
> prod_in-ams03-geo-ca_access    23         52172470
> prod_in-ams03-geo-ca_access    45         52154257
> prod_in-ams03-geo-ca_access    6          52160103
> prod_in-ams03-geo-ca_access    87         52091756
> prod_in-ams03-geo-ca_access    107        52089608
> prod_in-ams03-geo-ca_access    16         52149265
> prod_in-ams03-geo-ca_access    20         52150732
> prod_in-ams03-geo-ca_access    103        52155276
> prod_in-ams03-geo-ca_access    9          52161987
> prod_in-ams03-geo-ca_access    29         52153796
> prod_in-ams03-geo-ca_access    54         52152245
> prod_in-ams03-geo-ca_access    36         52142445
> prod_in-ams03-geo-ca_access    0          52155910
> prod_in-ams03-geo-ca_access    84         52159052
> prod_in-ams03-geo-ca_access    78         52158632
> prod_in-ams03-geo-ca_access    118        52149746
> prod_in-ams03-geo-ca_access    62         52090560
> prod_in-ams03-geo-ca_access    108        52157511
> prod_in-ams03-geo-ca_access    88         52150632
> prod_in-ams03-geo-ca_access    39         52148150
> prod_in-ams03-geo-ca_access    115        52169203
> prod_in-ams03-geo-ca_access    51         52153200
> prod_in-ams03-geo-ca_access    3          52150514
> prod_in-ams03-geo-ca_access    73         52159766
> prod_in-ams03-geo-ca_access    66         52161783
> prod_in-ams03-geo-ca_access    67         52078371
> prod_in-ams03-geo-ca_access    97         52089805
> prod_in-ams03-geo-ca_access    15         52156319
> prod_in-ams03-geo-ca_access    57         52100913
> prod_in-ams03-geo-ca_access    8          52161732
> prod_in-ams03-geo-ca_access    114        52152390
> prod_in-ams03-geo-ca_access    37         52091579
> prod_in-ams03-geo-ca_access    30         52156230
> prod_in-ams03-geo-ca_access    79         52146601
> prod_in-ams03-geo-ca_access    12         52092444
> prod_in-ams03-geo-ca_access    21         52151103
> prod_in-ams03-geo-ca_access    50         52153126
> prod_in-ams03-geo-ca_access    60         52147546
> prod_in-ams03-geo-ca_access    33         52152862
> prod_in-ams03-geo-ca_access    46         52167625
> prod_in-ams03-geo-ca_access    110        52156369
> prod_in-ams03-geo-ca_access    95         52157493
> prod_in-ams03-geo-ca_access    71         52153659
> prod_in-ams03-geo-ca_access    96         52148024
> prod_in-ams03-geo-ca_access    93         52152429
> prod_in-ams03-geo-ca_access    91         52162944
> prod_in-ams03-geo-ca_access    43         52154651
> prod_in-ams03-geo-ca_access    63         52167783
> prod_in-ams03-geo-ca_access    80         52154510
> prod_in-ams03-geo-ca_access    70         52157348{code}
> If there is an invalid committed offset, could that maybe have caused it to 
> not work correctly?  Even so, this should not be the result. 
> We really want to be able to use this tool to reset offsets to a specified 
> date.
> Let me know if you need more information.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to