[
https://issues.apache.org/jira/browse/SOLR-18569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132510#comment-18132510
]
ASF subversion and git services commented on SOLR-18569:
--------------------------------------------------------
Commit 0d6b2e32f598d772aed094038022a6a588fdfc5a in solr's branch
refs/heads/main from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=0d6b2e32f59 ]
SOLR-18569: Wait on init for ephemeral node-role cleanup (#5211)
Solr has long had code in ZkController to check whether the ephemeral
/live_node entry from a previous process is still around, and wait for ZK to
deleted it before proceeding.
But /live_node entries aren't the only ephemeral nodes that Solr expects to
create on startup and that can linger after process exit. The /node_roles/ tree
also contains ephemeral nodes. Currently, if any of these exist at
startup-time, Solr will hit a NodeExists exception from ZK and fail to start
up.
This commit changes ZkController's init code so that it now waits for these
nodes as well. In practice, this prevents Solr from potentially stalling on
start up because of an unexpected or unhandled NodeExists exception.
Co-authored-by: Nick Shanin <[email protected]>
> Solr should clean up /node_roles entries, like /live_nodes
> ----------------------------------------------------------
>
> Key: SOLR-18569
> URL: https://issues.apache.org/jira/browse/SOLR-18569
> Project: Solr
> Issue Type: Bug
> Affects Versions: 9.10.1
> Reporter: Jason Gerlowski
> Assignee: Jason Gerlowski
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> On Solr startup, ZkController.createEphemeralLiveNode creates
> {{/live_nodes/<node>}} and one ephemeral {{/node_roles/<role>/<mode>/<node>}}
> znode per node role in a single atomic ZK "multi" operation.
> On graceful shutdown, ZkController.removeEphemeralLiveNode deletes
> {{/live_nodes/<node>}}, but never does any cleanup for the corresponding
> {{/node_roles}} entries.
> These lingering entries are ephemeral and will get removed from ZK within
> {{zkClientTimeout}} millis after Solr shuts down. But they can still cause
> problems! For instance: we periodically see problems on a Solr restart
> because the new Solr node tries to create its {{/node_role}} entries but
> fails because the nodes already exist!
> {code}
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1778)
> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1650)
> at
> org.apache.solr.common.cloud.SolrZkClient.lambda$multi$12(SolrZkClient.java:792)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
> at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:792)
> at
> org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1244)
> at org.apache.solr.cloud.ZkController.init(ZkController.java:1059)
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:408)
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:139)
> at org.apache.solr.core.CoreContainer.loadInternal(CoreContainer.java:874)
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:808)
> {code}
> We should give ZkController.removeEphemeralLiveNode the same cleanup for
> /node_roles that it already has for /live_nodes, so that restarts don't hit
> this on startup.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
