[
https://issues.apache.org/jira/browse/SOLR-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125182#comment-18125182
]
Lucas Kot-Zaniewski commented on SOLR-18301:
--------------------------------------------
[~ab] Yes, I am not in a huge rush here. The recreating test is a single-node
scenario. It requires there only be one overseer eligible node which
disconnects from zk for session timeout +- 1 tick. Note that it doesn't matter
how far along into the tick it reconnects, because being within any period of a
single tick allows the possibility for the ephemerals to survive internal
overseer generation boundary and this is the main precondition for the bug (not
how close it is to session timeout). Once this is met, the internal race begins
and is actually quite easy to hit. However, once you have more than one
overseer-eligible node this becomes a lot less likely. This is because it is an
internal (intra-core) race between the previous overseer clean-up and the new
overseer creation. So, again, if you have multiple overseer eligible nodes this
becomes rather unlikely unless the other nodes also lose their sessions and
rejoin the election for some reason (i.e. in the case of a complete zk outage).
*However*, even in a multi-node set-up if you run the overseer prioritizer
there is also the following possibility of node 1 jamming node 2's overseer
start. This is unlikely (requires a process pause or zk connection blip) but
possible:
1. The prioritizer re-joins node 2 at the head of the queue, directly behind
node 1, and sends node 1 a QUIT.
2. Node 1's loop exits, and the Overseer Exit Thread (OET) calls getData on the
leader znode: node 1's own id, version 0.
3. Node 1 stalls (i.e. a zk connection blip through which curator will keep
retrying or simply a GC pause) for longer than the session timeout. Its session
expires, and ZK removes node 1's leader znode and election node.
4. Node 2's watch fires. It's first in line, so it registers the leader znode
(version 0) and starts its overseer.
5. Node 1 resumes with a new session, and its delete(leader, 0) lands on node
2's node, removing it. (this can even happen from a curator retry which was
launched from a previous zk session)
6. Node 1 rejoins at the tail.
7. Node 2's overseer finds no leader znode and exits. Its OET sees NoNode and
returns without rejoining.
8. Node 2's election node is still at the head, so no watch fires anywhere. The
cluster is left with no leader znode and no running overseer until node 2's
session ends.
I haven't heard of anyone using the overseer prioritizer but it is worth
calling out.
[~houston] or [[email protected]] would you mind taking a look when you
get a chance? Feel free to tag anyone who knows this code better. Thanks!
> Overseer Election May Not Converge After ZK Disconnect
> ------------------------------------------------------
>
> Key: SOLR-18301
> URL: https://issues.apache.org/jira/browse/SOLR-18301
> Project: Solr
> Issue Type: Bug
> Affects Versions: 10.0
> Reporter: Lucas Kot-Zaniewski
> Assignee: Lucas Kot-Zaniewski
> Priority: Major
> Labels: pull-request-available
> Attachments: overseer-node-election-divergence.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> It seems the migration to curator changed when we run Overseer leader
> election. It used [to only run on session
> expiry|https://github.com/apache/solr/blob/c2091d0258400c9064b8c67fe0d974e367ecccfd/solr/solrj-zookeeper/src/java/org/apache/solr/common/cloud/ConnectionManager.java#L156-L200]
> (AFAICT). Now it runs on every reconnection which the current logic is not
> well-equipped to do. There are two separate issues:
> 1. OverseerElectionContext does an unsynchronized leader-node creation and
> then starts the overseer *conditionally* on it not being closed:
> {code:java}
> zkClient.makePath(leaderPath, Utils.toJSON(myProps),
> CreateMode.EPHEMERAL);
> log.info("Created overseer leader registration {} -> {}", leaderPath, id);
> /// if anything closes the overseer context while it is waiting here you
> get a zombie overseer
> synchronized (this) {
> boolean shutDown =
> overseer.getZkController().getCoreContainer().isShutDown();
> if (!this.isClosed && !shutDown) {
> overseer.start(id);
> }
> {code}
> You may wonder what would trigger this to close externally? Well there are
> actually several threads that are competing to start the overseer from a
> single node. One is the OverseerExitThread of the departing overseer but then
> there are the various callbacks registered with zookeeper that *also*
> retryElection. So when these race against one another they are liable to
> close one another and result in a stranded overseer with a leader node but no
> registered OverseerExitThread to clean it up. I have been able to recreate it
> consistently with
> [testOverseerWedgesUnderRapidZkReconnects|https://github.com/kotman12/solr/commit/9c0390bf9ac2f6370b3630a2c7c4955fc3da042c]
> There is also a tangential bug that exacerbated the first one. Overseer
> threads may actually interfere with each other even across different Solr
> nodes. The departing overseer from one node currently can delete the leader
> registration node of the next elected overseer on a different node. This can
> send the cluster into an unnecessary leader election, increasing the
> probability of hitting bug 1. Enter bug 2.
> 2. {{Overseer.ClusterStateUpdater.checkIfIamStillLeader}} version check is
> merely theatrical:
> {code:java}
> Stat stat = new Stat();
> final String path = OVERSEER_ELECT + "/leader";
> byte[] data;
> try {
> // CSU pretending to get useful stat data
> // In reality every leader node is new and
> // always has version=0
> data = zkClient.getData(path, null, stat);
> } catch (IllegalStateException | KeeperException.NoNodeException e) {
> return;
> } catch (Exception e) {
> log.warn("Error communicating with ZooKeeper", e);
> return;
> }
> try {
> Map<?, ?> m = (Map<?, ?>) Utils.fromJSON(data);
> String id = (String) m.get(ID);
> if (overseerCollectionConfigSetProcessor.getId().equals(id)) {
> try {
> overseerCollectionConfigSetProcessor.getId(),
> path,
> stat.getVersion());
> // CSU pretending to do a safe, versioned delete of the
> // overseer leader node.
> // In reality, we never call setData on this node and so
> // the version is always 0 and thus CSU is liable to delete
> // a random overseer's leader node, potentially leaving it
> // stranded as a zombie from bug 1
> zkClient.delete(path, stat.getVersion());
> {code}
> This gets triggered on every disconnection now since
> [https://github.com/apache/solr/pull/2855/changes:]
> {code:java}
> onDisconnect(SUSPENDED) → overseer.close() (ZkController:406)
> → ClusterStateUpdater.run() loop exits → finally spawns OverseerExitThread
> (Overseer:399)
> → checkIfIamStillLeader → rejoinOverseerElection →
> LeaderElector.retryElection
> → this.context.close() ← This sets OEC.isClosed=true
> (LeaderElector:377 → OverseerElectionContext:97-99)
> {code}
> I've been able to observe both of these several times in the wild already
> which sent me down this rabbit hole. I am especially confident in the
> explanation of the first defect which is the only way I can explain some of
> the behaviors I was seeing. When I saw that a particular node attached as
> overseer leader with {{n_0000000007}} but did not see {{"Overseer
> (id=...n_0000000007) starting"}} *anywhere* the only explanation is that it
> gets stuck in zombie mode on the second {{isClosed}} check. I have been able
> to verify this on multiple clouds so am confident this isn't a logging issue.
> I was initially skeptical since I did really see this happen several times
> but given the makeData call + synchronized block can potentially wait an
> "I/O-sized" amount of time I suppose it's not unlikely at all.
> Another interesting behavior is that the overseer election keeps looping,
> bumping the election nodes to sequence numbers in the *millions* while the
> actual overseer leader node is stuck on whatever generation got stuck in the
> zombie/no-man's-land state. I have attached an image showing this. The only
> thing that eventually terminates the loop of overseer election retries is a
> StackOverflowError.
> Regarding bug 2 I do wonder if we can borrow the parent-node-version-check
> pattern from ShardLeaderElectionContextBase which does this before it removes
> the shard-leader *registration* node every time cancelElection is called.
> This would appear to guarantee that we don't yank another overseer's leader
> node. The other thing I found odd is OverseerElectionContext::cancelElection
> doesn't delete the overseer's leader node even thoughit is seemingly *very*
> similar to the shard leader registration node concept (in that it is a
> single-node materialization of the election result) and that flow *does*
> delete its registration node on election cancel. I haven't figured out why
> this is.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
