Solr version: 8.2; Zoo - 3.4 I am progressively adding collection by collections with 3 replica's on each, and all of a sudden we got to see the load averages on solr nodes were bumped and also memory usage went to 65% usage on JAVA process , with that some replica's had went to "GONE" state (as per solr cloud) , until I restarted the solr service its been this issue.
Need some guidance on where to start with on finding the root cause for this little outage? Data points: At the time we saw this outage, there are 3 instances of the copy tool which is actually pulling the data from old solr (5) and getting it indexed to new solr (8.2), which we stopped as and when we saw this outage as not really sure if that's creating the issue. we have around 12 solr nodes that are mapped to diff collections, each node got 8 Cores cpu with 64GB RAM (40GB was allocated to JVM HEAP) Based on the alerts , what we observed is that the load averages on few solr nodes were very high like 36.5, 26.7, 20 (not really sure if this is what concerning here as I used to see much lesser numbers like 3's and 4's on the averages but now this went to double digits) Also observed the below errors during that time from solr logs o.a.s.s.HttpSolrCall Unable to write response, client closed connection or we are shutting down org.eclipse.jetty.io.EofException: Closed at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:620) at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55) at org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54) at java.io.OutputStream.write(OutputStream.java:116) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) at org.apache.solr.util.FastWriter.flush(FastWriter.java:140) at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:154) at org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:93) at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:73) at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65) at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:809) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:538) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
