[
https://issues.apache.org/jira/browse/LUCENE-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984827#comment-16984827
]
Atri Sharma commented on LUCENE-9066:
-------------------------------------
Here is the high level plan of attack that I am considering:
1) Introduce a heuristic in IndexSearcher#slices which accounts for the current
work queue length of the passed in Executor. A simple idea could be to only
create a new slice iff all of current conditions are met AND the executor's
work queue length is less than the number of cores.
2) Introduce an interface which allows giving real time statistics about the
current state of the environment which can be used by IndexSearcher to decide
whether to execute a slice on a new thread or to execute it on a caller thread.
Potentially, the interface can be as simple as the same heuristic as above
where IndexSearcher executes a slice on a new thread only when the queue of the
executor is smaller than number of cores * 1.5 (similar to what Elasticsearch
does, AFAIK). The interface can be used to provide a custom implementation to
IndexSearcher thus paving the path for external modules to provide statistics
into the system and allowing Lucene to be a friendly neighbor in multi tenant
systems.
Thoughts?
CC [~jpountz]
> Modal Strategy In Concurrent Query Execution
> --------------------------------------------
>
> Key: LUCENE-9066
> URL: https://issues.apache.org/jira/browse/LUCENE-9066
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Atri Sharma
> Priority: Major
>
> When executing a query concurrently today, we do not take any sort of system
> statistics into account. For e.g. if the node is under high pressure, it is
> not advisable to spawn a large number of threads for a query since they are
> most likely to be blocked waiting for CPU to be available. However, the
> converse is that for a lightly loaded cluster, the query can consume as many
> threads as required.
>
> This Jira tracks high level efforts in this direction. The first idea is to
> account the Executor's wait queue's size as a factor when allocating slices
> to a query's segments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
