Jan Høydahl created SOLR-18665:
----------------------------------
Summary: bin/solr start reports success before the node is usable
Key: SOLR-18665
URL: https://issues.apache.org/jira/browse/SOLR-18665
Project: Solr
Issue Type: Bug
Components: scripts and tools
Affects Versions: 10.1
Reporter: Jan Høydahl
h3. The readiness check is a listening socket
{{bin/solr start}} decides the node is up by polling for a TCP listener:
{code}
running=$(lsof \-t \-PniTCP:$SOLR_PORT_LISTEN \-sTCP:LISTEN || :)
{code}
Jetty 12 opens its network connectors in {{Server.doStart()}} before it starts
the handler tree, so the port is bound and listening long before the Solr
webapp context loads. Measured on an idle machine: the LISTEN socket appears
~1.5s before {{solr start}} returns, and before the node has even logged
{{STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER}}.
h3. What is not ready yet
The {{_default}} configset is uploaded by
{{ConfigSetService.createConfigSetService}} -> {{bootstrapDefaultConf()}},
called from {{CoreContainer.load()}}, which runs during context start, i.e.
after the port is listening. Embedded ZooKeeper and the cluster znodes
({{/aliases.json}}, {{/live_nodes}}) are created earlier, so ZooKeeper answers
and looks healthy while {{/configs/_default}} is still absent.
So {{solr start}} prints "Started Solr server on port P. Happy searching!" and
exits 0, and the very next ZooKeeper-addressed CLI call can fail. Measured
window after {{solr start}} returned: 0.75s to 0.99s on an idle machine, longer
on a loaded CI runner. HTTP-addressed calls are unaffected: they wait in the
accept backlog.
h3. Reproduce
Fails on roughly 2 of 5 attempts on an idle machine:
{code}
bin/solr start
bin/solr zk downconfig \-z localhost:<embeddedZkPort> \-n _default \-d /tmp/dc
{code}
{noformat}
ERROR: Error downloading files from zookeeper path /configs/_default to
/tmp/dc/conf
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /configs/_default
{noformat}
Seen in CI on PR #5323 (SOLR-18642), where the CATS suite's randomized method
order put this case first; it passed on retry. The BATS suite never exposed it
because its fixed file order runs {{downconfig}} ninth, after an {{upconfig}}
case that waits on {{/api/configsets}}.
h3. Fix
{{bin\solr.cmd}} already does the right thing: its start wait runs {{SolrCLI
status \-\-max\-wait\-secs}}, a real readiness check. Only the bash script
polls a socket, so the fix is to make it do the same.
Related wart in the same block: the wait loop runs in a subshell, and its
timeout branch {{exit}}s only that subshell, so {{bin/solr start}} returns 0
even when it never saw the port come up.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]