What if it never comes back with unclean leader election disabled (but another broker does come back)?
On Wed, Nov 19, 2014 at 9:32 PM, Jun Rao <jun...@gmail.com> wrote: > In that case, we just wait until the broker in ISR is back and make it the > leader and take whatever data is has. > > Thanks, > > Jun > > On Tue, Nov 18, 2014 at 10:36 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > Ok, > > > > Makes sense. But if the node is not actually healthy (and underwent a > hard > > crash) it would likely not be able to avoid an 'unclean' restart.....what > > happens if unclean leader election is disabled, but there are no 'clean' > > partitions available? > > > > Jason > > > > On Wed, Nov 19, 2014 at 12:40 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > Yes, we will preserve the last replica in ISR. This way, we know which > > > replica has all committed messages and can wait for it to come back as > > the > > > leader, if unclean leader election is disabled. > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, Nov 17, 2014 at 11:06 AM, Jason Rosenberg <j...@squareup.com> > > > wrote: > > > > > > > We have had 2 nodes in a 4 node cluster die this weekend, sadly. > > > > Fortunately there was no critical data on these machines yet. > > > > > > > > The cluster is running 0.8.1.1, and using replication factor of 2 > for 2 > > > > topics, each with 20 partitions. > > > > > > > > For sake of discussion, assume that nodes A and B are still up, and C > > > and D > > > > are now down. > > > > > > > > As expected, partitions that had one replica on a good host (A or B) > > and > > > > one on a bad node (C or D), had their ISR shrink to just 1 node (A or > > B). > > > > > > > > Roughly 1/6 of the partitions had their 2 replicas on the 2 bad > nodes, > > C > > > > and D. For these, I was expecting the ISR to show up as empty, and > the > > > > partition unavailable. > > > > > > > > However, that's not what I'm seeing. When running TopicCommand > > > --describe, > > > > I see that the ISR still shows 1 replica, on node D (D was the second > > > node > > > > to go down). > > > > > > > > And, producers are still periodically trying to produce to node D > (but > > > > failing and retrying to one of the good nodes). > > > > > > > > So, it seems the cluster's meta data is still thinking that node D is > > up > > > > and serving the partitions that were only replicated on C and D. > > > However, > > > > for partitions that were on A and D, or B and D, D is not shown as > > being > > > in > > > > the ISR. > > > > > > > > Is this correct? Should the cluster continue showing the last node > to > > > have > > > > been alive for a partition as still in the ISR? > > > > > > > > Jason > > > > > > > > > >
