gianm commented on issue #16411: URL: https://github.com/apache/druid/issues/16411#issuecomment-2103564632
We saw a double-leader situation recently when a ZK server cycled, and we suspect it has something to do with https://issues.apache.org/jira/browse/CURATOR-696. That Curator Jira suggests a bug was introduced by https://issues.apache.org/jira/browse/CURATOR-644 (PR: https://github.com/apache/curator/pull/430). It seems possible that this did introduce a bug, since that changed the logic from doing `reset()` always on reconnection (which would recreate the ephemeral znode) to doing `getChildren()`, which would look for existing ones, and then only call `reset()` if they could not be found. We updated to Curator 5.4 some time ago, in https://github.com/apache/druid/pull/13302. So if this is indeed what’s going on, it has potentially been an issue since Druid 25. What we saw specifically was this scenario: - OL 1 was leader prior to ZK connection loss - OL 1 reconnected to ZK and got a session id that we believe is a new session id (although we were not able to confirm that) - OL 1's LeaderLatch recipe checked the latch patch and saw an ephemeral znode there that it believed was its own, so it started leadership. - OL 2, 30s later, checked the latch path and saw no children at all (not even the one for OL 1). It created an ephemeral znode for itself, and started leadership. We think what happened is that both OLs established new sessions, even though the old sessions hadn’t expired yet. Because the old sessions hadn’t expired yet, the old ephemeral znodes were still there upon reconnection. The old leader, OL 1, saw both old znodes there and assumed it was still leader. But because those znodes were associated with different sessions, they went away in 30s. When OL 2 noticed that, it assumed there was no active leader, so it became one and then we had two leaders. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
