[ 
https://issues.apache.org/jira/browse/SOLR-18665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SOLR-18665:
----------------------------------
    Labels: cats pull-request-available  (was: cats)

> bin/solr start reports success before the node is usable
> --------------------------------------------------------
>
>                 Key: SOLR-18665
>                 URL: https://issues.apache.org/jira/browse/SOLR-18665
>             Project: Solr
>          Issue Type: Bug
>          Components: scripts and tools
>    Affects Versions: 10.1
>            Reporter: Jan Høydahl
>            Priority: Major
>              Labels: cats, pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. The readiness check is a listening socket
> {{bin/solr start}} decides the node is up by polling for a TCP listener:
> {code}
> running=$(lsof \-t \-PniTCP:$SOLR_PORT_LISTEN \-sTCP:LISTEN || :)
> {code}
> Jetty 12 opens its network connectors in {{Server.doStart()}} before it 
> starts the handler tree, so the port is bound and listening long before the 
> Solr webapp context loads. Measured on an idle machine: the LISTEN socket 
> appears ~1.5s before {{solr start}} returns, and before the node has even 
> logged {{STARTING EMBEDDED ENSEMBLE ZOOKEEPER SERVER}}.
> h3. What is not ready yet
> The {{_default}} configset is uploaded by 
> {{ConfigSetService.createConfigSetService}} -> {{bootstrapDefaultConf()}}, 
> called from {{CoreContainer.load()}}, which runs during context start, i.e. 
> after the port is listening. Embedded ZooKeeper and the cluster znodes 
> ({{/aliases.json}}, {{/live_nodes}}) are created earlier, so ZooKeeper 
> answers and looks healthy while {{/configs/_default}} is still absent.
> So {{solr start}} prints "Started Solr server on port P. Happy searching!" 
> and exits 0, and the very next ZooKeeper-addressed CLI call can fail. 
> Measured window after {{solr start}} returned: 0.75s to 0.99s on an idle 
> machine, longer on a loaded CI runner. HTTP-addressed calls are unaffected: 
> they wait in the accept backlog.
> h3. Reproduce
> Fails on roughly 2 of 5 attempts on an idle machine:
> {code}
> bin/solr start
> bin/solr zk downconfig \-z localhost:<embeddedZkPort> \-n _default \-d /tmp/dc
> {code}
> {noformat}
> ERROR: Error downloading files from zookeeper path /configs/_default to 
> /tmp/dc/conf
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: 
> KeeperErrorCode = NoNode for /configs/_default
> {noformat}
> Seen in CI on PR #5323 (SOLR-18642), where the CATS suite's randomized method 
> order put this case first; it passed on retry. The BATS suite never exposed 
> it because its fixed file order runs {{downconfig}} ninth, after an 
> {{upconfig}} case that waits on {{/api/configsets}}.
> h3. Fix
> {{bin\solr.cmd}} already does the right thing: its start wait runs {{SolrCLI 
> status \-\-max\-wait\-secs}}, a real readiness check. Only the bash script 
> polls a socket, so the fix is to make it do the same.
> Related wart in the same block: the wait loop runs in a subshell, and its 
> timeout branch {{exit}}s only that subshell, so {{bin/solr start}} returns 0 
> even when it never saw the port come up.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to