And see https://issues.apache.org/jira/browse/SOLR-683 as it also may be related or have helpful info...
On Apr 23, 2012, at 8:17 AM, Mark Miller wrote: > Perhaps related is > http://www.lucidimagination.com/search/document/6d0e168c82c86a38#45c945b2de6543f4 > > On Apr 23, 2012, at 5:37 AM, Trym R. Møller wrote: > >> Hi >> >> I have succeeded in reproducing the scenario with two Solr instances >> running. They cover a single collection with two slices and two replica, two >> cores in each Solr instance. I have changed the number of threads that Jetty >> is allowed to use as follows: >> <New class="org.mortbay.thread.QueuedThreadPool"> >> <Set name="minThreads">3</Set> >> <Set name="maxThreads">3</Set> >> <Set name="lowThreads">0</Set> >> </New> >> And when indexing a single document this works fine but when concurrently >> indexing 10 documents, Solr frequently hangs. >> I know that Jetty per default are allowed to use 10.000 threads, but in my >> other setup, all these 10.000 allowed thread are used on a single Solr >> instance (I have 7 Solr instances) after some days and the hanging scenario >> occurs. >> >> I'm not sure if just adjusting the allowed number of threads are the best >> solution and would like to get some input as what to expect and if there are >> other things I can adjust. >> My setup is as written before 7 Solr instances handling a single collection >> with 28 leaders and 28 replicas distributed fairly on the Solrs (8 cores on >> each Solr). >> >> Thanks for any input. >> >> Best regards Trym >> >> >> Den 19-04-2012 14:36, Yonik Seeley skrev: >>> On Thu, Apr 19, 2012 at 4:25 AM, "Trym R. Møller"<t...@sigmat.dk> wrote: >>>> Hi >>>> >>>> I am using Solr trunk and have 7 Solr instances running with 28 leaders and >>>> 28 replicas for a single collection. >>>> After indexing a while (a couple of days) the solrs start hanging and doing >>>> a thread dump on the jvm I see blocked threads like the following: >>>> Thread 2369: (state = BLOCKED) >>>> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; >>>> information may be imprecise) >>>> - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, >>>> line=158 (Compiled frame) >>>> - >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() >>>> @bci=42, line=1987 (Compiled frame) >>>> - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 >>>> (Compiled frame) >>>> - java.util.concurrent.ExecutorCompletionService.take() @bci=4, line=164 >>>> (Compiled frame) >>>> - org.apache.solr.update.SolrCmdDistributor.checkResponses(boolean) >>>> @bci=27, line=350 (Compiled frame) >>>> - org.apache.solr.update.SolrCmdDistributor.finish() @bci=18, line=98 >>>> (Compiled frame) >>>> - org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish() >>>> @bci=4, line=299 (Compiled frame) >>>> - org.apache.solr.update.processor.DistributedUpdateProcessor.finish() >>>> @bci=1, line=817 (Compiled frame) >>>> ... >>>> - org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 >>>> (Interpreted frame) >>>> >>>> I read the stack trace as my indexing client has indexed a document and >>>> this >>>> Solr is now waiting for the replica? to respond before returning an answer >>>> to the client. >>> Correct. What's the full stack trace like on both a leader and replica? >>> We need to know what the replica is blocking on. >>> >>> What version of trunk are you using? >>> >>> -Yonik >>> lucenerevolution.com - Lucene/Solr Open Source Search Conference. >>> Boston May 7-10 > > - Mark Miller > lucidimagination.com > > > > > > > > > > > - Mark Miller lucidimagination.com
