Hi, use a bounded thread pool.
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Desidero [mailto:desid...@gmail.com] > Sent: Tuesday, October 01, 2013 11:37 PM > To: java-user@lucene.apache.org > Subject: Re: Query performance in Lucene 4.x > > For anyone who was wondering, this was actually resolved in a different > thread today. I misread the information in the > IndexSearcher(IndexReader,ExecutorService) constructor documentation - I > was under the impression that it was submitting a thread for each index > shard (MultiReader wraps 20 shards, so 20 tasks) but it was really submitting > a task for each segment within each shard (20 shards * ~10 segments = ~200 > tasks) which is horrible. Since my index changes infrequently, I'm using > forceMerge(1) before sending out updated indexes to the slave servers. > Without any extra tuning (threads, # of shards, etc) I've gone from ~2900 > requests per minute to ~10k requests per minute. > > Thanks to Adrien and Mike for the clarification and Benson for bringing up > the question that led to my answer. > > I'm still pretty new to Lucene so I have a lot of poking around to do, but I'm > going to try to implement the "virtual segment" concept that Mike > mentioned. It'll be really helpful for those of us who want parallelism within > queries and don't want to forceMerge. > > > On Fri, Sep 27, 2013 at 9:55 AM, Desidero <desid...@gmail.com> wrote: > > > Erick, > > > > Thank you for responding. > > > > I ran tests using both compressed fields and uncompressed fields, and > > it was significantly slower with uncompressed fields. I looked into > > the lazy field loading per your suggestion, but we don't get any > > values from the returned Documents until the result set has been > appropriately reduced. > > Since we only store one retrievable field and we always need to get > > it, it doesn't save any time loading it lazily. > > > > I'll try running a test without loading any fields just to see how it > > affects performance and let you know how that goes. > > > > Regards, > > Matt > > > > > > On Fri, Sep 27, 2013 at 8:01 AM, Erick Erickson > <erickerick...@gmail.com>wrote: > > > >> Hmmm, since 4.1, fields have been stored compressed by default. > >> I suppose it's possible that this is a result of > >> compressing/uncompressing. > >> > >> What happens if > >> 1> you enable lazy field loading > >> 2> don't load any fields? > >> > >> FWIW, > >> Erick > >> > >> On Thu, Sep 26, 2013 at 10:55 AM, Desidero <desid...@gmail.com> > wrote: > >> > A quick update: > >> > > >> > In order to confirm that none of the standard migration changes had > >> > a negative effect on performance, I ported my Lucene 4.x version > >> > back to Lucene 3.6.2 and kept the newer API rather than using the > >> > custom ParallelMultiSearcher and other deprecated methods/classes. > >> > > >> > Performance in 3.6.2 is even faster than before (~2900 requests/min > >> with 4.x > >> > vs ~6200 requests/min with 3.6.2), so none of my code changes > >> > should be causing the difference. It seems to be something Lucene > >> > is doing under > >> the > >> > covers. > >> > > >> > Again, if there's any other information if I can provide to help > >> determine > >> > what's going on, please let me know. > >> > > >> > Thanks, > >> > Matt > >> > > >> > > >> > > >> > ------------------------------------------------------------------- > >> > -- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> > For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
