Jan Høydahl created SOLR-18665:
----------------------------------

             Summary: bin/solr start reports success before the node is usable
                 Key: SOLR-18665
                 URL: https://issues.apache.org/jira/browse/SOLR-18665
             Project: Solr
          Issue Type: Bug
          Components: scripts and tools
    Affects Versions: 10.1
            Reporter: Jan Høydahl


h3. The readiness check is a listening socket

{{bin/solr start}} decides the node is up by polling for a TCP listener:

{code}
running=$(lsof \-t \-PniTCP:$SOLR_PORT_LISTEN \-sTCP:LISTEN || :)
{code}

Jetty 12 opens its network connectors in {{Server.doStart()}} before it starts 
the handler tree, so the port is bound and listening long before the Solr 
webapp context loads. Measured on an idle machine: the LISTEN socket appears 
~1.5s before {{solr start}} returns, and before the node has even logged 
{{STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER}}.

h3. What is not ready yet

The {{_default}} configset is uploaded by 
{{ConfigSetService.createConfigSetService}} -> {{bootstrapDefaultConf()}}, 
called from {{CoreContainer.load()}}, which runs during context start, i.e. 
after the port is listening. Embedded ZooKeeper and the cluster znodes 
({{/aliases.json}}, {{/live_nodes}}) are created earlier, so ZooKeeper answers 
and looks healthy while {{/configs/_default}} is still absent.

So {{solr start}} prints "Started Solr server on port P. Happy searching!" and 
exits 0, and the very next ZooKeeper-addressed CLI call can fail. Measured 
window after {{solr start}} returned: 0.75s to 0.99s on an idle machine, longer 
on a loaded CI runner. HTTP-addressed calls are unaffected: they wait in the 
accept backlog.

h3. Reproduce

Fails on roughly 2 of 5 attempts on an idle machine:

{code}
bin/solr start
bin/solr zk downconfig \-z localhost:<embeddedZkPort> \-n _default \-d /tmp/dc
{code}

{noformat}
ERROR: Error downloading files from zookeeper path /configs/_default to 
/tmp/dc/conf
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: 
KeeperErrorCode = NoNode for /configs/_default
{noformat}

Seen in CI on PR #5323 (SOLR-18642), where the CATS suite's randomized method 
order put this case first; it passed on retry. The BATS suite never exposed it 
because its fixed file order runs {{downconfig}} ninth, after an {{upconfig}} 
case that waits on {{/api/configsets}}.

h3. Fix

{{bin\solr.cmd}} already does the right thing: its start wait runs {{SolrCLI 
status \-\-max\-wait\-secs}}, a real readiness check. Only the bash script 
polls a socket, so the fix is to make it do the same.

Related wart in the same block: the wait loop runs in a subshell, and its 
timeout branch {{exit}}s only that subshell, so {{bin/solr start}} returns 0 
even when it never saw the port come up.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to