Hi
I have succeeded in reproducing the scenario with two Solr instances
running. They cover a single collection with two slices and two replica,
two cores in each Solr instance. I have changed the number of threads
that Jetty is allowed to use as follows:
<New class="org.mortbay.thread.QueuedThreadPool">
<Set name="minThreads">3</Set>
<Set name="maxThreads">3</Set>
<Set name="lowThreads">0</Set>
</New>
And when indexing a single document this works fine but when
concurrently indexing 10 documents, Solr frequently hangs.
I know that Jetty per default are allowed to use 10.000 threads, but in
my other setup, all these 10.000 allowed thread are used on a single
Solr instance (I have 7 Solr instances) after some days and the hanging
scenario occurs.
I'm not sure if just adjusting the allowed number of threads are the
best solution and would like to get some input as what to expect and if
there are other things I can adjust.
My setup is as written before 7 Solr instances handling a single
collection with 28 leaders and 28 replicas distributed fairly on the
Solrs (8 cores on each Solr).
Thanks for any input.
Best regards Trym
Den 19-04-2012 14:36, Yonik Seeley skrev:
On Thu, Apr 19, 2012 at 4:25 AM, "Trym R. Møller"<t...@sigmat.dk> wrote:
Hi
I am using Solr trunk and have 7 Solr instances running with 28 leaders and
28 replicas for a single collection.
After indexing a while (a couple of days) the solrs start hanging and doing
a thread dump on the jvm I see blocked threads like the following:
Thread 2369: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=158 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
@bci=42, line=1987 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399
(Compiled frame)
- java.util.concurrent.ExecutorCompletionService.take() @bci=4, line=164
(Compiled frame)
- org.apache.solr.update.SolrCmdDistributor.checkResponses(boolean)
@bci=27, line=350 (Compiled frame)
- org.apache.solr.update.SolrCmdDistributor.finish() @bci=18, line=98
(Compiled frame)
- org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish()
@bci=4, line=299 (Compiled frame)
- org.apache.solr.update.processor.DistributedUpdateProcessor.finish()
@bci=1, line=817 (Compiled frame)
...
- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582
(Interpreted frame)
I read the stack trace as my indexing client has indexed a document and this
Solr is now waiting for the replica? to respond before returning an answer
to the client.
Correct. What's the full stack trace like on both a leader and replica?
We need to know what the replica is blocking on.
What version of trunk are you using?
-Yonik
lucenerevolution.com - Lucene/Solr Open Source Search Conference.
Boston May 7-10
