I'm particularly interested in Noble and Mark's comments... Let's say you have 5 nodes in n1, n2, n3, n4, n5.
n1 is the leader, n2 watches n1 etc. Now I retryElection for n3 with joinAtHead=true. Both n2 and n3 are watching n1. So far, so good. My expectation is that deleting n1 would cause n3 to become leader, but it isn't at all guaranteed. I have a test case illustrating this. Incidentally, I think I should get the same result by calling retryElection on n1 with joinAtHead=false; n3 should become the leader. I was working on SOLR-6691 and slowly going crazy since everything I was trying would fail. Basically, to rebalance leaders (thanks Noble for pointing out how far off I was in my original approach) it seemed like it would be sufficient to 1> have the preferred leader retry the election at the head 2> tell the old leader to retry at the tail I expected the old node that was watching the leader to figure out that it wasn't really next in line and re-add itself to the end. But things went all to hell in a handbasket when I wrote a harness that exercised it, and it drove me a bit nuts. Especially since it would fail one way one time and another way the next. And it'd even succeed upon occasion.... I figured out that my expectations weren't being met. Due to the way leader queues are sorted, if the two sequence numbers are identical then the tie-breaker does NOT pick the last node to join at head. It picks the one with the lowest (highest? didn't track that down entirely) session ID. Either way, sometimes it picks the node newly added at the head and sometimes it picks the old one. If I _am_ on the right path, then I propose the following: 1> I'll raise a new JIRA for leader sequence sorting and take it on. I'm not quite sure how fix it, the ideas I have are fairly hacky. 2> I'll back out the REBALANCELEADER stuff. Currently it'll break things badly and we're too close to 5.0 to try to do anything about <1> IMO. this just means that I'll comment out the collections API call in the code and update the ref guide. 3> When <1> is resolved, I'll put REBALANCELEADERs back in, but that won't be before 5.1 Erick --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
