Hey Guozhang, Thanks much for reviewing the KIP!
In the scenario you described, let's assume that broker A has messages with offset up to 10, and broker B has messages with offset up to 20. If consumer wants to consume message with offset 9, it will not receive OffsetOutOfRangeException from broker A. If consumer wants to consume message with offset 16, then consumer must have already fetched message with offset 15, which can only come from broker B. Because consumer will fetch from broker B only if leaderEpoch >= 2, then the current consumer leaderEpoch can not be 1 since this KIP prevents leaderEpoch rewind. Thus we will not have OffsetOutOfRangeException in this case. Does this address your question, or maybe there is more advanced scenario that the KIP does not handle? Thanks, Dong On Tue, Jan 23, 2018 at 9:43 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Thanks Dong, I made a pass over the wiki and it lgtm. > > Just a quick question: can we completely eliminate the > OffsetOutOfRangeException with this approach? Say if there is consecutive > leader changes such that the cached metadata's partition epoch is 1, and > the metadata fetch response returns with partition epoch 2 pointing to > leader broker A, while the actual up-to-date metadata has partition epoch 3 > whose leader is now broker B, the metadata refresh will still succeed and > the follow-up fetch request may still see OORE? > > > Guozhang > > > On Tue, Jan 23, 2018 at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > > > Hi all, > > > > I would like to start the voting process for KIP-232: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 232%3A+Detect+outdated+metadata+using+leaderEpoch+and+partitionEpoch > > > > The KIP will help fix a concurrency issue in Kafka which currently can > > cause message loss or message duplication in consumer. > > > > Regards, > > Dong > > > > > > -- > -- Guozhang >
