Re: Solr thread problems

[苗海泉]

Thank you for reading my question in detail. Let me explain. With 1169
threads, our collection is only 937 in number, and it is the number of
threads in a solr node, not the total number of threads in the cluster.
With more than a thousand collections, solrCloud is in a poor condition, so
we've reduced the number of collections.

For our business, we add only unmodified systems, and we do not delete
individual documents, so the delete wait you say may not be the case
because we did not delete the action.


I tested under the solrCloud commit scheduling thread and query thread and
found that the number of commit scheduling threads and the replica'num solr
node is related to the case of automatic commit, commit scheduling thread
number = 2 * solr replica num, If there is no automatic commit, commit the
number of scheduled threads = solr replica num, I was puzzled is that the
commit scheduling thread after execution (non-automatic submission) is
still not destroyed, as the collection number increases, the commit
scheduling thread Will be more and more until solrcloud restart it.

2018-03-02 10:15 GMT+08:00 Shawn Heisey :

> On 3/1/2018 4:31 AM, 苗海泉 wrote:
>
>> My question is, what is the relationship between the number of threads in
>> the commitScheduler thread pool and what? The number of searcherExecutor
>> thread pool and the above have a relationship, why so much, thank you!
>>
>
> I don't understand this question.  Can you try again?
>
> Looking at the information you've provided ... are you doing a large
> number of simultaneous DeleteByQuery operations?That does seem like a lot
> of threads in a WAITING state.  I'm not sure what is going on, but if there
> are a lot of DeleteByQuery operations happening, it MIGHT cause that.
>
> In another discussion, you mentioned having more than one thousand
> collections.  This is going to result in Solr creating a large number of
> threads.  I'm actually surprised that you don't have far more than 1169
> threads.
>
> Every experiment I've done with thousands of collections has turned out
> badly, often before I even reach 1000 of them. SolrCloud just does not like
> dealing with it.
>
> Three years ago, I filed the following issue about it. Somebody marked it
> as fixed, though I have no idea why.  It doesn't appear to be fixed to me,
> and there were never any code changes related to the issue:
>
> https://issues.apache.org/jira/browse/SOLR-7191
>
>
>
> In that discussion about lots of collections, you asked this:
>
> Thank you for your advice on gc tools, what do you suggest to me?
>>
>
> I don't understand this question either.  I had just given you some advice
> about GC tools.  What are you asking?
>
> Thanks,
> Shawn
>
>


-- 
==
联创科技
知行如一
==

Re: Solr thread problems

[Shawn Heisey]

On 3/1/2018 4:31 AM, 苗海泉 wrote:
My question is, what is the relationship between the number of threads 
in the commitScheduler thread pool and what? The number of 
searcherExecutor thread pool and the above have a relationship, why so 
much, thank you!


I don't understand this question.  Can you try again?

Looking at the information you've provided ... are you doing a large 
number of simultaneous DeleteByQuery operations?That does seem like a 
lot of threads in a WAITING state.  I'm not sure what is going on, but 
if there are a lot of DeleteByQuery operations happening, it MIGHT cause 
that.


In another discussion, you mentioned having more than one thousand 
collections.  This is going to result in Solr creating a large number of 
threads.  I'm actually surprised that you don't have far more than 1169 
threads.


Every experiment I've done with thousands of collections has turned out 
badly, often before I even reach 1000 of them. SolrCloud just does not 
like dealing with it.


Three years ago, I filed the following issue about it. Somebody marked 
it as fixed, though I have no idea why.  It doesn't appear to be fixed 
to me, and there were never any code changes related to the issue:


https://issues.apache.org/jira/browse/SOLR-7191



In that discussion about lots of collections, you asked this:


Thank you for your advice on gc tools, what do you suggest to me?


I don't understand this question either.  I had just given you some 
advice about GC tools.  What are you asking?


Thanks,
Shawn

Solr thread problems

[苗海泉]

 I jstack through the analysis of our solr thread found that the total
number of threads 1169 of which 1037 threads are the wait state, the 1037
threads, there are two types of threads is very large, 486 threads with
same trace trace searcherExecutor-5694-thread -1,
searcherExecutor-5694-thread-1 - priority: 5 - threadId: 0x7f0f0401f000
- nativeId: 0x406a - state: WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park (Native Method)
- parking to wait for <0x7f1787000578> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject)
at java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer $
ConditionObject.await (AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take
(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor $ Worker.run
(ThreadPoolExecutor.java:617)
at java.lang.Thread.run (Thread.java:745)
Locked ownable synchronizers:
- None



546 threads with same stack trace
commitScheduler-5592-thread-1
commitScheduler-5592-thread-1 - priority: 5 - threadId: 0x7f1174e56000
- nativeId: 0x2120 - state: WAITING
stackTrace:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park (Native Method)
- parking to wait for <0x7f174d2011b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer $ ConditionObject)
at java.util.concurrent.locks.LockSupport.park (LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer $
ConditionObject.await (AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor $ DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor $ DelayedWorkQueue.take
(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask
(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor $ Worker.run
(ThreadPoolExecutor.java:617)
at java.lang.Thread.run (Thread.java:745)
Locked ownable synchronizers:
- None

--


My question is, what is the relationship between the number of threads in
the commitScheduler thread pool and what? The number of searcherExecutor
thread pool and the above have a relationship, why so much, thank you!