Jan Høydahl created SOLR-18607:
----------------------------------
Summary: bin/solr cannot find a Solr process started in the
foreground
Key: SOLR-18607
URL: https://issues.apache.org/jira/browse/SOLR-18607
Project: Solr
Issue Type: Bug
Components: scripts and tools
Reporter: Jan Høydahl
Assignee: Jan Høydahl
Solr started in the foreground ({{{}solr start -f{}}}, i.e. every container)
writes no PID file. That is by design: {{start_solr}} does {{{}exec java
...{}}}, and only the background branch writes
{{{}$SOLR_PID_DIR/solr-<port>.pid{}}}.
That should be fine, because both discovery mechanisms have a path that does
not need the PID file: {{bin/solr}} falls back to {{{}ps{}}}, and
{{SolrProcessManager}} scans live processes via {{{}ProcessHandle{}}}. But
three commands fail to use them correctly, so a foreground Solr looks "not
running" to its own CLI.
h2. 1. {{solr stop -p <port>}} always fails (regression in 10.x)
SOLR-15442 renamed {{-Djetty.port}} to {{-Dsolr.port.listen}} throughout
{{{}bin/solr{}}}, but missed one line: the {{ps}} fallback in the stop/restart
branch.
{code:none}
solr/bin/solr:1017 (start branch) awk
"/\-Dsolr\.port\.listen=$SOLR_PORT_LISTEN/" <- converted
solr/bin/solr:1030 (stop branch) awk "/\-Djetty\.port=$SOLR_PORT_LISTEN/"
<- missed
{code}
Solr starts with {{{}-Dsolr.port.listen=<port>{}}}, so the stop branch's
pattern matches nothing. Verified on the official images, started in the
foreground, same command:
{code:none}
# solr:9.11
$ solr stop -p 8983
Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds
to allow Jetty process 11 to stop gracefully.
-> graceful shutdown, container exit 0
# solr:10.1
$ solr stop -p 8983
No process found for Solr node running on port 8983
-> exit 1, Solr still running
{code}
Patching only that line in a live {{solr:10.1}} container makes the same
command shut Solr down gracefully with exit 0, confirming the root cause. One
line fix; {{branch_9x}} is unaffected.
h2. 2. {{solr stop --all}} exits 0 while stopping nothing
With no PID files present the {{--all}} branch iterates an empty {{find}}
result and exits 0, without stopping anything and without printing a warning —
a silent false success, worse than the honest failure in case 1. The script
already carries a TODO noting that {{none_stopped}} does not propagate out of
the {{while}} subshell, which is why the "No Solr nodes found to stop." message
cannot fire on this path.
h2. 3. {{solr status}} without {{-p}} reports nothing
{{StatusTool}} uses {{processMgr.scanSolrPidFiles()}} when no port is given, so
it reports "No Solr nodes are running." in any container. With {{-p}} it uses
{{{}processMgr.processForPort(){}}}, backed by {{{}SolrProcessManager{}}}'s
live {{ProcessHandle}} scan filtering on {{{}-Dsolr.port.listen={}}}, and works
correctly:
{code:none}
$ solr status
No Solr nodes are running.
$ solr status -p 8983
Solr process 11 running on port 8983
{...}
{code}
h2. Suggested fix
* Case 1: correct the sysprop name on {{{}solr/bin/solr:1030{}}}.
* Case 2: make the {{--all}} path report when it stopped nothing, and fall
back to live process discovery rather than PID files alone.
* Case 3: have {{StatusTool}} fall back to (or prefer) the live
{{ProcessHandle}} scan when no PID files are found, instead of concluding
nothing is running.
More generally, {{SolrProcessManager}} already does discovery without PID files
correctly; the shell fallbacks and the no-argument paths should lean on it
rather than treating the PID file as the primary source of truth.
h2. Note: writing a PID file in foreground mode would be the wrong fix
{{exec}} preserves {{{}$${}}}, so capturing the PID is possible, but nothing
removes the file when the process is killed. On a mounted volume
({{{}SOLR_PID_DIR{}}} is typically the data volume) a stale file survives into
the next container, and {{solr_pid_by_port}} only checks that the recorded PID
exists, not that it is Solr. Container PIDs are small and reused, so this turns
a cosmetic problem into a startup failure. Reproduced by planting a stale PID
file:
{code:none}
$ echo 1 > /var/solr/solr-8983.pid # stale, points at a PID that is alive
$ # on container restart:
Port 8983 is already being used by another process (pid: 1)
Please choose a different port using the -p option.
{code}
h2. Downstream impact
Found via solr-operator issue
[871|https://github.com/apache/solr-operator/issues/871], where the pod's
{{preStop}} hook runs {{{}solr stop -p <port>{}}}. That hook emits a
{{FailedPreStopHook}} warning on both 9.x and 10.x, but only the 10.x cause is
a Solr bug
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
