[
https://issues.apache.org/jira/browse/SOLR-18558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129660#comment-18129660
]
Jerry Chung commented on SOLR-18558:
------------------------------------
Additional Details:
*Why nothing self-heals* (verified by grep across {{{}solr/core{}}}):
* {{zkController.register()}} is only invoked from core-load time
({{{}CoreContainer.java:1002{}}} background, {{:1442/:1451}} sync) and
{{ZkController.onReconnect()}} (re-registers all cores {*}only on ZK session
expiration/reconnect{*}). There is no background task that re-registers a core
that is loaded locally but {{DOWN}} in cluster-state.
* *This also explains why our RELOAD did nothing* — a core RELOAD re-opens the
index but the re-registration path does not re-invoke {{registerInZk}} for an
already-registered core. The only two things that re-trigger registration are
exactly the two we observed working: *a full JVM restart* (re-runs core load →
{{{}registerInZk{}}}) and a ZK session drop.
*Minimal fix candidates:*
# Bounded retry with backoff in {{registerInZk}} (or
{{{}ZkController.register{}}}) on
{{{}SolrException(SERVICE_UNAVAILABLE){}}}/{{{}NoNode{}}} — the znode race is
transient by nature; a few retries with escalation would have healed n91 within
seconds.
# Stop swallowing {{NoNodeException}} in
{{ShardLeaderElectionContextBase.runLeaderProcess}} — reschedule the election
({{{}LeaderElector.retryElection{}}}) instead of silently returning.
# A watchdog that detects "core loaded locally + {{DOWN}} in cluster-state for
> N minutes" and re-triggers registration (this is the safety net that would
have caught n91 at ~01:30 instead of 16:44).
> Stale leader election znode left after DELETEREPLICA blocks the remaining
> replica from becoming leader; registration fails with "Could not get leader
> props"
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18558
> URL: https://issues.apache.org/jira/browse/SOLR-18558
> Project: Solr
> Issue Type: Bug
> Components: Server, SolrCloud
> Affects Versions: 10.0
> Reporter: Jerry Chung
> Priority: Major
>
> After a DELETEREPLICA on a shard, the election znode of the deleted replica
> was left in /collections/<c>/leader_elect/<shard>/election, at the head of
> the queue. The remaining replica joined the election behind it and never ran
> its leader process. Registration of its core timed out after about 13
> minutes, and the shard stayed unloaded with no leader. Nothing logged that
> the replica was waiting on another election node.
> Environment
> - Solr 10.0.0 (SolrCloud), ZooKeeper 3.9.4
> - The shard has one replica, core_node93 (replica_n91)
> - Node was restarted at 03:11 (clean shutdown) and the core loaded at 03:13:08
> - DELETEREPLICA had earlier removed core_node94 from the same shard. [The
> node hosting core_node94 was live at the time but possibly busy. Add the time
> and the Overseer/collection-API log if available.]
> Observed
> 1. ZkController.register → getLeader (ZkController.java:1419) polls
> /collections/<c>/leaders/<shard>/leader once a second for leaderVoteWait +
> 600000 ms. At 03:26:09 it failed:
> SolrException: Could not get leader props (ZkController.getLeaderProps:1715)
> Caused by: KeeperException$NoNodeException: NoNode for
> /collections/<c>/leaders/shard22/leader
> SolrException: Error getting leader from zk for shard shard22
> (ZkController.getLeader:1661)
> ZkContainer: Exception registering core <c>_shard22_replica_n91
> 2. Election queue:
> $ solr zk ls /collections/<c>/leader_elect/shard22_v10/election
> 144115188964327726-core_node94-n_0000000001
> 216172782927479120-core_node93-n_0000000005
> core_node94 no longer exists in the clusterstate. Its node is ahead of
> core_node93.
> 3. Terms also hold stale entries for the deleted replica:
> {"core_node94":7, "core_node93":8, "core_node94_recovering":1}
> These don't block core_node93: it has the max term and no _recovering
> marker, so canBecomeLeader is true.
> 4. Between 03:13:08 and the timeout, the registration thread logged only
> ShardLeaderElectionContextBase make sure parent is created
> .../leaders/shard22_v10. There were no Enough replicas found, I may be the
> new leader, Our node is no longer in line to be leader or Will not register
> as leader lines. I found no ZK session expiry, disconnect or reconnect in the
> log from 03:11 to 03:40.
> Why the replica waits silently
> In LeaderElector.checkIfIamLeader (line 125 onward), a replica that isn't
> first in line only sets a watch on the node below it. The only log message
> there is at DEBUG ("Watching path ... to know if I could be the leader"). A
> stale head-of-queue node therefore blocks the shard with no visible warning.
> The stale node is ephemeral, so some ZK session was still holding it. [Add
> the owner of session 144115188964327726, converted to hex, from the ZK dump
> command.]
> Suspected cause (not confirmed or reproduced)
> DELETEREPLICA cleans up terms and the election node only when UNLOAD reaches
> the replica's node. unregister is the only place that does it
> (ZkController.java:1964-1981). The paths below can leave the election node
> behind, particularly if the node is slow or busy:
> - DeleteReplicaCmd.java:311-332: UNLOAD is sent only if the node is live. If
> the node is too slow, waitForCoreNodeGone times out after 30s and
> deleteCoreNode removes the replica from the clusterstate anyway. The core may
> still be loading, registering or electing on that node. A non-live node skips
> UNLOAD entirely, so terms and election nodes are left behind.
> - ShardLeaderElectionContext.runLeaderProcess (line 110): if (core == null)
> \{ return; } ("shutdown or removed") returns without calling
> cancelElection(). A node that was already joined stays in the queue until the
> JVM's session ends.
> - ZkController.unregister vs register: if unregister runs before the
> registration thread calls joinElection, it finds no context and cancels
> nothing. The registration thread then creates the election znode and context
> (ZkController.java:1743-1749), and nothing cancels them afterward.
> - ShardLeaderElectionContext.rejoinLeaderElection (line 503): it checks only
> cc.isShutDown(), not whether the context is still registered. After
> unregister has cancelled and removed the context, it can rejoin and leave an
> election node that nothing owns.
> Expected
> - After DELETEREPLICA completes, no election znode or terms entry remains for
> the deleted replica.
> - A replica waiting behind another election node logs it at WARN, naming the
> node it waits on.
> - A shard whose only live replica is behind a stale election node recovers on
> its own.
> Impact
> The shard is unloaded and has no leader, so it can't serve queries or index.
> The registration thread blocks for 13 minutes (with default leaderVoteWait),
> then fails.
> Workaround
> 1. Delete the stale election znode: solr zk rm
> /collections/<c>/leader_elect/<shard>/election/<stale-node> (add -z <zkHost>
> if needed).
> 2. Reload the core or restart the node. The registration thread has already
> given up.
> 3. Optionally remove the stale entries from terms/<shard>.
> Proposed fixes (for discussion)
> 1. In runLeaderProcess, call cancelElection() before returning when the core
> is null.
> 2. After joinElection in register, re-check that the replica still exists in
> the clusterstate. Cancel and bail out if it doesn't.
> 3. In rejoinLeaderElection, skip the rejoin if the context is no longer in
> electionContexts.
> 4. In checkIfIamLeader, log at WARN when waiting on a node whose replica
> isn't in the clusterstate.
> 5. When DELETEREPLICA targets a non-live node, remove the replica's terms
> entries.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
