1,500 threads seems extreme by any standards so there is something happening in your install. Even with appservers for web apps, typically 100 would be a fair # of threads.
On 7/28/10, Christos Constantinou <ch...@simpleweb.co.uk> wrote: > Hi, > > Solr seems to be crashing after a JVM exception that new threads cannot be > created. I am writing in hope of advice from someone that has experienced > this before. The exception that is causing the problem is: > > Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to create > new native thread > > The memory that is allocated to Solr is 3072MB, which should be enough > memory for a ~6GB data set. The documents are not big either, they have > around 10 fields of which only one stores large text ranging between 1k-50k. > > The top command at the time of the crash shows Solr using around 1500 > threads, which I assume it is not normal. Could it be that the threads are > crashing one by one and new ones are created to cope with the queries? > > In the log file, right after the the exception, there are several thousand > commits before the server stalls completely. Normally, the log file would > report 20-30 document existence queries per second, then 1 commit per 5-30 > seconds, and some more infrequent faceted document searches on the data. > However after the exception, there are only commits until the end of the log > file. > > I am wondering if anyone has experienced this before or if it is some sort > of known bug from Solr 1.4? Is there a way to increase the details of the > exception in the logfile? > > I am attaching the output of a grep Exception command on the logfile. > > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:19:32 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:20:18 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:20:48 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:22:43 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:28:50 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:33:19 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:08 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:58 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:35:59 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:44:31 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:51:49 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:55:43 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:57:47 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > > Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to create > new native thread > > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.apache.solr.common.SolrException: Error opening new searcher. > exceeded limit of maxWarmingSearchers=2, try again later. > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > java.lang.IllegalStateException: Committed > Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > SEVERE: org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > org.mortbay.jetty.EofException > Caused by: java.net.SocketException: Broken pipe > java.lang.IllegalStateException: Committed > > > Thanks > > Christos -- Sent from my mobile device
