[
https://issues.apache.org/jira/browse/SOLR-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154804#comment-17154804
]
Ilan Ginzburg commented on SOLR-11208:
--------------------------------------
What would be the issue implementing the OP's suggestion of getting rid of the
{{SynchronousQueue}} in the {{ExecutorUtil.MDCAwareThreadPoolExecutor}} built
in {{OverseerCollectionMessageHandler}} to use a {{LinkedBlockingQueue}} (can
be unbounded or bounded) or {{ArrayBlockingQueue}} (bounded)?
If we go for bounded, we can dimension the queue to a very large size without
the penalty of having too many threads. If we go unbounded and if the queue
grows to infinity like it theoretically could, then we have other issues anyway
and the system is not functional regardless.
I'd go unbounded so one less configuration parameter to worry about. Possibly
issue warn logs if queue exceeds 20k entries (something not expected to happen
but if it does we'll know).
Given we already use multiple threads in the executor, there are no constraints
on execution order that would be relaxed by using a real queue. Currently
already a {{CollectionOperation}} submitted earlier can get executed after a
{{CollectionOperation}} submitted later. Clients must be careful not to submit
a subsequent operation before they know a previous one completed. Having a
queue is not going to change anything there.
If nobody objects, I'd replace the {{SynchronousQueue}} by an unbounded
{{LinkedBlockingQueue}} in {{OverseerCollectionMessageHandler}} (and change the
value of 5 currently passed as {{corePoolSize}} to 10 to match
{{maximumPooolSize}} given that {{maximumPooolSize}} will no longer be used).
> Usage SynchronousQueue in Executors prevent large scale operations
> ------------------------------------------------------------------
>
> Key: SOLR-11208
> URL: https://issues.apache.org/jira/browse/SOLR-11208
> Project: Solr
> Issue Type: Bug
> Affects Versions: 6.6
> Reporter: Björn Häuser
> Priority: Major
> Attachments: response.json
>
>
> I am not sure where to start with this one.
> I tried to post this already on the mailing list:
> https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201708.mbox/%3c48c49426-33a2-4d79-ae26-a4515b8f8...@gmail.com%3e
> In short: the usage of a SynchronousQueue as the workQeue prevents more tasks
> than max threads.
> For example, taken from OverseerCollectionMessageHandler:
> {code:java}
> ExecutorService tpe = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 10,
> 0L, TimeUnit.MILLISECONDS,
> new SynchronousQueue<>(),
> new
> DefaultSolrThreadFactory("OverseerCollectionMessageHandlerThreadFactory"));
> {code}
> This Executor is used when doing a REPLACENODE (= ADDREPLICA) command. When
> the node has more than 10 collections this will fail with the mentioned
> java.util.concurrent.RejectedExecutionException.
> I am also not sure how to fix this. Just replacing the queue with a different
> implementation feels wrong to me or could cause unwanted side behaviour.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
