Some time ago I created something similar, it's kinda a backport into Lucene 4.3:
https://github.com/apache/incubator-blur/blob/65640200a8e7dd539c1dd4d920255c717102b9b2/blur-query/src/main/java/org/apache/blur/lucene/search/CloneableCollector.java#L25 It's handles the execution of searching the segments in parallel but doesn't provide any limitations on parallelism. Aaron On Tue, Jun 28, 2016 at 6:37 AM, Ravikumar Govindarajan < [email protected]> wrote: > Aaron, > > Just an update.. > > https://issues.apache.org/jira/browse/LUCENE-5299 > > You can now use any collector & get guaranteed parallel execution. They > have also provided a "parallelism" hint that will limit the number of > search threads at request level... > > i.e., we can fix blur executor thread-count at 128 & limit "parallelism" at > a max of 4 threads per request.. > > On Fri, Feb 6, 2015 at 5:25 PM, Ravikumar Govindarajan < > [email protected]> wrote: > > > Thanks for the clarifications. > > > > Another point I thought about is the disk efficiency of a serving a > > random-IO. Many parallel threads could end-up hitting just one or two > disks > > in the cluster… > > > > Think I can skip it safely for my work-loads. > > > > -- > > Ravi > > > > On Fri, Feb 6, 2015 at 3:09 PM, Aaron McCurry <[email protected]> > wrote: > > > >> The ServiceExecutor (thread pool) put inside the IndexSearcher was an > >> attempt at making the segments search in parallel when available. > However > >> there is a limitation in Lucene that does not allow segment parallel > >> searches when you are using Collectors. > >> > >> > >> > https://github.com/apache/lucene-solr/blob/lucene_solr_4_3_0/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L595 > >> > >> We override this method to allow for Tracing: > >> > >> > >> > https://github.com/apache/incubator-blur/blob/master/blur-core/src/main/java/org/apache/blur/server/IndexSearcherCloseableBase.java#L46 > >> > >> and here: > >> > >> > >> > https://github.com/apache/incubator-blur/blob/master/blur-core/src/main/java/org/apache/blur/server/IndexSearcherCloseableSecureBase.java#L51 > >> > >> I agree that if you are already running a lot of shards per server that > if > >> we were to enhance Lucene to allow for parallel searching of segments it > >> could become counter productive. I have seen underutilized systems that > >> could take advantage of the parallel segment search, so as with any > >> feature > >> like this, it depends. :-) > >> > >> Aaron > >> > >> On Fri, Feb 6, 2015 at 2:39 AM, Ravikumar Govindarajan < > >> [email protected]> wrote: > >> > >> > Blur by default uses a SearchExecutor for IndexSearcher. I believe > >> lucene > >> > helps searching segments of a single shard in parallel. > >> > > >> > Our previous index was built on a lower version of lucene where such a > >> > feature was absent and we ran sequential search per shard only… > >> > > >> > What is the general recommendation for blur? Is it advisable to use > the > >> > SearchExecutor? What will happen when there are many parallel queries > >> for > >> > different shards. Will SearchExecutor become a bottle-neck? > >> > > >> > Any help is much appreciated... > >> > > >> > -- > >> > Ravi > >> > > >> > > > > >
