"If consumer wants to consume message with offset 16, then consumer must have already fetched message with offset 15"
--> this may not be always true right? What if consumer just call seek(16) after construction and then poll without committed offset ever stored before? Admittedly it is rare but we do not programmably disallow it. Guozhang On Tue, Jan 23, 2018 at 10:42 PM, Dong Lin <lindon...@gmail.com> wrote: > Hey Guozhang, > > Thanks much for reviewing the KIP! > > In the scenario you described, let's assume that broker A has messages with > offset up to 10, and broker B has messages with offset up to 20. If > consumer wants to consume message with offset 9, it will not receive > OffsetOutOfRangeException > from broker A. > > If consumer wants to consume message with offset 16, then consumer must > have already fetched message with offset 15, which can only come from > broker B. Because consumer will fetch from broker B only if leaderEpoch >= > 2, then the current consumer leaderEpoch can not be 1 since this KIP > prevents leaderEpoch rewind. Thus we will not have > OffsetOutOfRangeException > in this case. > > Does this address your question, or maybe there is more advanced scenario > that the KIP does not handle? > > Thanks, > Dong > > On Tue, Jan 23, 2018 at 9:43 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Thanks Dong, I made a pass over the wiki and it lgtm. > > > > Just a quick question: can we completely eliminate the > > OffsetOutOfRangeException with this approach? Say if there is consecutive > > leader changes such that the cached metadata's partition epoch is 1, and > > the metadata fetch response returns with partition epoch 2 pointing to > > leader broker A, while the actual up-to-date metadata has partition > epoch 3 > > whose leader is now broker B, the metadata refresh will still succeed and > > the follow-up fetch request may still see OORE? > > > > > > Guozhang > > > > > > On Tue, Jan 23, 2018 at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hi all, > > > > > > I would like to start the voting process for KIP-232: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 232%3A+Detect+outdated+metadata+using+leaderEpoch+and+partitionEpoch > > > > > > The KIP will help fix a concurrency issue in Kafka which currently can > > > cause message loss or message duplication in consumer. > > > > > > Regards, > > > Dong > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
