Whenever I stop Solr with 'bin/solr stop' or 'sudo service solr stop'
Solr's cores are corrupted and the only way to remedy the situation is to
do a full uninstall and reinstall of all traces of Solr.
Please help, I can't live like this. The error is at the bottom, as it is
long.
This is a standard Ubuntu 18.04 AMI for AWS on an r5.4xlarge instance with
a 1024GB EBS drive. The OS settings we have altered to get Solr running are:
# Set file limit for Solr higher than 1024
echo "# Solr LIMIT must be higher than 1024" | sudo tee -a
/etc/security/limits.conf
echo "${USER} hard nofile 65535" | sudo tee -a /etc/security/limits.conf
echo "${USER} soft nofile 65535" | sudo tee -a /etc/security/limits.conf
echo "${USER} hard nproc 65535" | sudo tee -a /etc/security/limits.conf
echo "${USER} soft nproc 65535" | sudo tee -a /etc/security/limits.conf
# Fix available entropy
echo "HRNGDEVICE=/dev/urandom" | sudo tee -a /etc/default/rng-tools
sudo /etc/init.d/rng-tools restart
I am installing Solr like this:
sudo bin/install_solr_service.sh ~/solr-8.5.2.tgz
This is to the point that I can't even restart Solr to set SOLR_HOST for
security reasons. It is madness.
I create the cores with this command:
sudo -u solr `which solr` create_core -c ${CORE} -d ${CORE_CONFIG_DIR}/ -V
I load the cores with this command:
sudo -u solr `which post` -c ${CORE} ${DATA_PATH}
I stop Solr with this command:
sudo solr service stop
# or
sudo /opt/solr/bin/solr stop
/var/solr is owned by user solr. /opt/solr is owned by user solr.
I just find myself wondering - am I making a mistake by not running stop
solr as user solr?
Anyway, I have no idea what is wrong. Please help!
I start Solr like this the most recent time:
sudo -u solr `which solr` start
Waiting up to 180 seconds to see Solr running on port 8983 [|]
Started Solr server on port 8983 (pid=1697). Happy searching!
This happens when I stop solr no matter how I do it, via service or solr
command, as solr or root. Deleting the data directory doesn't help. No
removal of cores helps. I use the script above to remove Solr and then
reinstall it and create/load the cores again and only then does it work :(
Error at http://solr.host:8983/solr/ follows:
HTTP ERROR 404 javax.servlet.UnavailableException: Error processing
the request. CoreContainer is either not initialized or shutting
down.URI: /solr/STATUS: 404MESSAGE:
javax.servlet.UnavailableException:
Error processing the request. CoreContainer is either not initialized
or shutting down.SERVLET: default
CAUSED BY: javax.servlet.ServletException:
javax.servlet.UnavailableException: Error processing the request.
CoreContainer is either not initialized or shutting down.
CAUSED BY: javax.servlet.UnavailableException: Error processing the
request. CoreContainer is either not initialized or shutting down.
Caused by:
javax.servlet.ServletException: javax.servlet.UnavailableException:
Error processing the request. CoreContainer is either not initialized
or shutting down.
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.servlet.UnavailableException: Error processing the
request. CoreContainer is either not initialized or shutting down.
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:371)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
... 18 more
Thanks,
Russell Jurney @rjurney <http://twitter.com/rjurney>
russell.jur...@gmail.com LI <http://linkedin.com/in/russelljurney> FB
<http://facebook.com/jurney> datasyndrome.com
