"Let's say you have 5 nodes in n1, n2, n3, n4, n5. n1 is the leader, n2 watches n1 etc.
Now I retryElection for n3 with joinAtHead=true. Both n2 and n3 are watching n1. So far, so good. My expectation is that deleting n1 would cause n3 to become leader, but it isn't at all guaranteed. I have a test case illustrating this" deleting n1 is not enough before that, you should ask n2 to rejoin election (joinAthead=false). This will ensure that n2 is at tail now. Now the order is n1,n3,n4.... now ask n1 to rejoin (not at head) and it will join back at tail and n3 will become leader On Wed, Dec 3, 2014 at 7:20 AM, Erick Erickson <erickerick...@gmail.com> wrote: > Thanks! I somewhat remember seeing that conversation but I confess I > didn't follow it that closely. > > I can't cope with looking at it any more tonight, but I'll check in > the morning. The problem I see is I don't think there's any way, once > a node is re-inserted in the queue, for another node to figure out > that it's not supposed to be the leader if it's first in line after > the nodes are sorted, but I may have missed that. > > Erick > > On Tue, Dec 2, 2014 at 5:34 PM, Jessica Mallet <mewmewb...@gmail.com> > wrote: > > This is reminiscent of my conversation with Noble on this SOLR-6095 > starting > > at this comment: > > > https://issues.apache.org/jira/browse/SOLR-6095?focusedCommentId=14032386&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14032386 > > > > Unfortunately I dropped off following it and my memory is a bit vague > right > > now. Reading from the comments, I think Noble had in mind that the > > tie-breaker can pick the wrong node (n2) to be the leader, but then the > > wrong node will then re-initiate the process to renounce leadership and > > re-join (according to > > > https://issues.apache.org/jira/browse/SOLR-6095?focusedCommentId=14032619&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14032619 > ). > > > > I then asked about when that renounce process will happen for n2 > > ( > https://issues.apache.org/jira/browse/SOLR-6095?focusedCommentId=14032659&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14032659 > ), > > and I'm not sure if that was ever specifically answered. Figuring if and > how > > that happens might be key in moving forward? > > > > Jessica > > > > On Tue, Dec 2, 2014 at 4:30 PM, Erick Erickson <erickerick...@gmail.com> > > wrote: > >> > >> I'm particularly interested in Noble and Mark's comments... > >> > >> Let's say you have 5 nodes in n1, n2, n3, n4, n5. > >> > >> n1 is the leader, n2 watches n1 etc. > >> > >> Now I retryElection for n3 with joinAtHead=true. Both n2 and n3 are > >> watching n1. So far, so good. > >> > >> My expectation is that deleting n1 would cause n3 to become leader, > >> but it isn't at all guaranteed. I have a test case illustrating this. > >> > >> Incidentally, I think I should get the same result by calling > >> retryElection on n1 with joinAtHead=false; n3 should become the > >> leader. > >> > >> I was working on SOLR-6691 and slowly going crazy since everything I > >> was trying would fail. Basically, to rebalance leaders (thanks Noble > >> for pointing out how far off I was in my original approach) it seemed > >> like it would be sufficient to > >> > >> 1> have the preferred leader retry the election at the head > >> 2> tell the old leader to retry at the tail > >> > >> I expected the old node that was watching the leader to figure out > >> that it wasn't really next in line and re-add itself to the end. > >> > >> But things went all to hell in a handbasket when I wrote a harness > >> that exercised it, and it drove me a bit nuts. Especially since it > >> would fail one way one time and another way the next. And it'd even > >> succeed upon occasion.... > >> > >> I figured out that my expectations weren't being met. Due to the way > >> leader queues are sorted, if the two sequence numbers are identical > >> then the tie-breaker does NOT pick the last node to join at head. It > >> picks the one with the lowest (highest? didn't track that down > >> entirely) session ID. Either way, sometimes it picks the node newly > >> added at the head and sometimes it picks the old one. > >> > >> If I _am_ on the right path, then I propose the following: > >> 1> I'll raise a new JIRA for leader sequence sorting and take it on. > >> I'm not quite sure how fix it, the ideas I have are fairly hacky. > >> > >> 2> I'll back out the REBALANCELEADER stuff. Currently it'll break > >> things badly and we're too close to 5.0 to try to do anything about > >> <1> IMO. this just means that I'll comment out the collections API > >> call in the code and update the ref guide. > >> > >> 3> When <1> is resolved, I'll put REBALANCELEADERs back in, but that > >> won't be before 5.1 > >> > >> Erick > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- ----------------------------------------------------- Noble Paul
