Glad to help. Do be aware that there are several config values that influence the commit frequency, they might also be relevant.
Best Erick On Thu, Jul 29, 2010 at 5:11 AM, Christos Constantinou < ch...@simpleweb.co.uk> wrote: > Eric, > > Thank you very much for the indicators! I had a closer look at the commit > intervals and it seems that the application is gradually increasing the > commits to almost once per second after some time - something that was > hidden in the massive amount of queries in the log file. I have changed the > code to use commitWithin rather than commit and everything looks much better > now. I believe that might have solved the problem so thanks again. > > Christos > > On 29 Jul 2010, at 01:44, Erick Erickson wrote: > > > Your commits are very suspect. How often are you making changes to your > > index? > > Do you have autocommit on? Do you commit when updating each document? > > Committing > > too often and consequently firing off warmup queries is the first place > I'd > > look. But I > > agree with dc tech, 1,500 is waaaay more than I would expect. > > > > Best > > Erick > > > > > > > > On Wed, Jul 28, 2010 at 6:53 AM, Christos Constantinou < > > ch...@simpleweb.co.uk> wrote: > > > >> Hi, > >> > >> Solr seems to be crashing after a JVM exception that new threads cannot > be > >> created. I am writing in hope of advice from someone that has > experienced > >> this before. The exception that is causing the problem is: > >> > >> Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to > >> create new native thread > >> > >> The memory that is allocated to Solr is 3072MB, which should be enough > >> memory for a ~6GB data set. The documents are not big either, they have > >> around 10 fields of which only one stores large text ranging between > 1k-50k. > >> > >> The top command at the time of the crash shows Solr using around 1500 > >> threads, which I assume it is not normal. Could it be that the threads > are > >> crashing one by one and new ones are created to cope with the queries? > >> > >> In the log file, right after the the exception, there are several > thousand > >> commits before the server stalls completely. Normally, the log file > would > >> report 20-30 document existence queries per second, then 1 commit per > 5-30 > >> seconds, and some more infrequent faceted document searches on the data. > >> However after the exception, there are only commits until the end of the > log > >> file. > >> > >> I am wondering if anyone has experienced this before or if it is some > sort > >> of known bug from Solr 1.4? Is there a way to increase the details of > the > >> exception in the logfile? > >> > >> I am attaching the output of a grep Exception command on the logfile. > >> > >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:19:31 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:19:32 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:20:18 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:20:48 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:22:43 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:27:53 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:28:50 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:33:19 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:35:08 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:35:58 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:35:59 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:44:31 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:51:49 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:55:17 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:55:43 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:56:21 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:57:36 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:57:47 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 8:58:25 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:07:20 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> > >> Exception in thread "btpool0-5" java.lang.OutOfMemoryError: unable to > >> create new native thread > >> > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.apache.solr.common.SolrException: Error opening new > searcher. > >> exceeded limit of maxWarmingSearchers=2, try again later. > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> java.lang.IllegalStateException: Committed > >> Jul 28, 2010 9:57:03 AM org.apache.solr.common.SolrException log > >> SEVERE: org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> org.mortbay.jetty.EofException > >> Caused by: java.net.SocketException: Broken pipe > >> java.lang.IllegalStateException: Committed > >> > >> > >> Thanks > >> > >> Christos > >
