[
https://issues.apache.org/jira/browse/SOLR-18607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18607:
----------------------------------
Labels: pull-request-available (was: )
> bin/solr cannot find a Solr process started in the foreground
> -------------------------------------------------------------
>
> Key: SOLR-18607
> URL: https://issues.apache.org/jira/browse/SOLR-18607
> Project: Solr
> Issue Type: Bug
> Components: scripts and tools
> Reporter: Jan Høydahl
> Assignee: Jan Høydahl
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Solr started in the foreground ({{{}solr start -f{}}}, i.e. every container)
> writes no PID file. That is by design: {{start_solr}} does {{{}exec java
> ...{}}}, and only the background branch writes
> {{{}$SOLR_PID_DIR/solr-<port>.pid{}}}.
> That should be fine, because both discovery mechanisms have a path that does
> not need the PID file: {{bin/solr}} falls back to {{{}ps{}}}, and
> {{SolrProcessManager}} scans live processes via {{{}ProcessHandle{}}}. But
> three commands fail to use them correctly, so a foreground Solr looks "not
> running" to its own CLI.
> h2. 1. {{solr stop -p <port>}} always fails (regression in 10.x)
> SOLR-15442 renamed {{-Djetty.port}} to {{-Dsolr.port.listen}} throughout
> {{{}bin/solr{}}}, but missed one line: the {{ps}} fallback in the
> stop/restart branch.
> {code:none}
> solr/bin/solr:1017 (start branch) awk
> "/\-Dsolr\.port\.listen=$SOLR_PORT_LISTEN/" <- converted
> solr/bin/solr:1030 (stop branch) awk "/\-Djetty\.port=$SOLR_PORT_LISTEN/"
> <- missed
> {code}
> Solr starts with {{{}-Dsolr.port.listen=<port>{}}}, so the stop branch's
> pattern matches nothing. Verified on the official images, started in the
> foreground, same command:
> {code:none}
> # solr:9.11
> $ solr stop -p 8983
> Sending stop command to Solr running on port 8983 ... waiting up to 180
> seconds to allow Jetty process 11 to stop gracefully.
> -> graceful shutdown, container exit 0
> # solr:10.1
> $ solr stop -p 8983
> No process found for Solr node running on port 8983
> -> exit 1, Solr still running
> {code}
> Patching only that line in a live {{solr:10.1}} container makes the same
> command shut Solr down gracefully with exit 0, confirming the root cause. One
> line fix; {{branch_9x}} is unaffected.
> h2. 2. {{solr stop --all}} exits 0 while stopping nothing
> With no PID files present the {{--all}} branch iterates an empty {{find}}
> result and exits 0, without stopping anything and without printing a warning
> — a silent false success, worse than the honest failure in case 1. The script
> already carries a TODO noting that {{none_stopped}} does not propagate out of
> the {{while}} subshell, which is why the "No Solr nodes found to stop."
> message cannot fire on this path.
> h2. 3. {{solr status}} without {{-p}} reports nothing
> {{StatusTool}} uses {{processMgr.scanSolrPidFiles()}} when no port is given,
> so it reports "No Solr nodes are running." in any container. With {{-p}} it
> uses {{{}processMgr.processForPort(){}}}, backed by
> {{{}SolrProcessManager{}}}'s live {{ProcessHandle}} scan filtering on
> {{{}-Dsolr.port.listen={}}}, and works correctly:
> {code:none}
> $ solr status
> No Solr nodes are running.
> $ solr status -p 8983
> Solr process 11 running on port 8983
> {...}
> {code}
> h2. Suggested fix
> * Case 1: correct the sysprop name on {{{}solr/bin/solr:1030{}}}.
> * Case 2: make the {{--all}} path report when it stopped nothing, and fall
> back to live process discovery rather than PID files alone.
> * Case 3: have {{StatusTool}} fall back to (or prefer) the live
> {{ProcessHandle}} scan when no PID files are found, instead of concluding
> nothing is running.
> More generally, {{SolrProcessManager}} already does discovery without PID
> files correctly; the shell fallbacks and the no-argument paths should lean on
> it rather than treating the PID file as the primary source of truth.
> h2. Note: writing a PID file in foreground mode would be the wrong fix
> {{exec}} preserves {{{}$${}}}, so capturing the PID is possible, but nothing
> removes the file when the process is killed. On a mounted volume
> ({{{}SOLR_PID_DIR{}}} is typically the data volume) a stale file survives
> into the next container, and {{solr_pid_by_port}} only checks that the
> recorded PID exists, not that it is Solr. Container PIDs are small and
> reused, so this turns a cosmetic problem into a startup failure. Reproduced
> by planting a stale PID file:
> {code:none}
> $ echo 1 > /var/solr/solr-8983.pid # stale, points at a PID that is alive
> $ # on container restart:
> Port 8983 is already being used by another process (pid: 1)
> Please choose a different port using the -p option.
> {code}
> h2. Downstream impact
> Found via solr-operator issue
> [871|https://github.com/apache/solr-operator/issues/871], where the pod's
> {{preStop}} hook runs {{{}solr stop -p <port>{}}}. That hook emits a
> {{FailedPreStopHook}} warning on both 9.x and 10.x, but only the 10.x cause
> is a Solr bug
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
