Hi Jun This should only be possible in situations where there is a crash or something happens to the underlying disks (assuming clean leader election). I've not come across others. The assumption, as I understand it, is that the underlying issue stems from KAFKA-1211 <https://issues.apache.org/jira/browse/KAFKA-1211> which is being addressed in KIP-101 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Generation+rather+than+High+Watermark+for+Truncation>. If you can reproduce in a more generally scenario we would be very interested.
All the best B On Mon, Dec 19, 2016 at 12:35 AM Jun MA <mj.saber1...@gmail.com> wrote: > Would be grateful to hear opinions from experts out there. Thanks in > advance > > > On Dec 17, 2016, at 11:06 AM, Jun MA <mj.saber1...@gmail.com> wrote: > > > > Hi, > > > > We saw the following FATAL error in 2 of our brokers (3 in total, the > active controller doesn’t have this) and they crashed in the same time. > > > > [2016-12-16 16:12:47,085] FATAL [ReplicaFetcherThread-0-3], Halting > because log truncation is not allowed for topic __consumer_offsets, Current > leader 3's latest offset 5910081 is less than replica 1's latest offset > 5910082 (kafka.server.ReplicaFetcherThread) > > > > Our solution is set topic __consumer_offsets > unclean.leader.election.enable=true temporarily, and restart brokers. In > this way we potentially lose offsets of some topics. Is there any better > solutions? > > > > I saw related tickets https://issues.apache.org/jira/browse/KAFKA-3861 < > https://issues.apache.org/jira/browse/KAFKA-3861>, > https://issues.apache.org/jira/browse/KAFKA-3410 < > https://issues.apache.org/jira/browse/KAFKA-3410> and understand why > brokers crashed. But we didn’t see any scenarios mentioned in above > tickets. Is there any other reason why this happened? There’s no broker > restart involved in our case. How can we prevent it from happening? > > > > We’re using 0.9.0 with unclean.leader.election.enable=false and > min.insync.replicas=2. > > > > Thanks, > > Jun > >
