Hi Shawn, I included the wrong file for solr4 and did not realize until you pointed out the heap size. The correct file that is setting the Java environment is "Solr 4 tomcat setenv" I have uploaded that to the shared folder along with the requested screenshots "Solr 4 top screenshot","Solr 6 top screenshot","Solr 8 top screenshot".
I have also uploaded the solr.log, solr_gc.log, and solr_slow_requests.log from a 2 hour period of time where I was running the email load test against the solr8 implementation in which the queued tasks are taking too long to complete. solr_gc.log, solr_gc.log.1, solr_gc.log.2, solr.log, solr.log.10, solr.log.6, solr.log.7, solr.log.8, solr.log.9, solr_slow_requests.log Let me know if there is any other information that I can provide that may help to work through this. *Manzama*a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0) 2 8417 2339 543 NW York Drive, Suite 100, Bend, OR 97703 LinkedIn <http://www.linkedin.com/company/manzama> | Twitter <https://twitter.com/ManzamaInc> | Facebook <http://www.facebook.com/manzamainc> | YouTube <https://www.youtube.com/channel/UCBo3QoqewyNoo7HiT_BFuRw> On Tue, Oct 15, 2019 at 2:28 AM Shawn Heisey <apa...@elyograg.org> wrote: > On 10/14/2019 1:36 PM, Russell Bahr wrote: > > Backend replacement of solr4 and hopefully Frontend replacement as well. > > solr-spec 8.1.1 > > lucene-spec 8.1.1 > > Runtime Oracle Corporation OpenJDK 64-Bit Server VM 12 12+33 > > 1 collection 6 shards 5 replicas per shard 17,919,889 current documents > (35 days worth of documents) - indexing new documents regularly throughout > the day, deleting aged out documents nightly. > > Java 12 is not recommended. It is one of the "new feature" releases > that only gets 6 months of support. We would recommend Java 8 or Java > 11. These are the versions with long term support. Probably a good > thing to be using OpenJDK, as the official Oracle Java now requires > paying for a license. > > Solr 8 ships with settings that enable the G1GC collector instead of > CMS, because CMS is deprecated and will disappear in a future Java > version. We have seen problems with this when the system is > misconfigured as far as heap size. When the system is properly sized, > G1 tends to do better than CMS, but when the heap is too large or too > small, has a tendency to amplify garbage collection problems in comparison. > > Looking at your solr.in.sh files for each version ... the Solr 4 install > appears to be setting the heap to 512 megabytes. This is definitely not > enough for millions of documents, and if this is what the heap size is > actually set to, would almost certainly run into memory errors > frequently and have absolutely terrible performance. But you are saying > that it works well, so I don't think the heap is actually set to 512 > megabytes. Maybe the bin/solr script has been modified directly to set > the memory size instead of setting it in solr.in.sh where it should be > set. > > Solr 6 has a heap size of just under 27 gigabytes. Solr 8 has a heap > size of just under 8 gigabytes. With millions of documents, it is > likely that 8GB of heap is not quite big enough. > > For each of your installations (Solr 4, Solr 6, and Solr 8) can you > provide the screenshot described at this wiki page? > > > https://cwiki.apache.org/confluence/display/solr/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue > > It would also be helpful to see the GC logs from Solr 8. We would need > at least one GC log, making sure that they cover at least a few hours, > including the timeframe when the slow indexing and slow queries were > observed. > > Thanks, > Shawn >
