Since you are just interested in retrieving the top n hits, sounds to me that TopDocs is the way to go. It's not a drop-in replacement for Hits but the transition is pretty straightforward.
Then if MatchAllDocsQuery + filters gives you good enough performance you could stop, if it doesn't look at Boolean queries. And of course do take note of the comments about sorting. http://wiki.apache.org/lucene-java/ImproveSearchingSpeed is well worth a look. -- Ian. On Mon, Nov 30, 2009 at 4:34 PM, Michel Nadeau <aka...@gmail.com> wrote: > I'll definitely switch to a Collector. > > It's just not clear for me if I should use BooleanQueries or > MatchAllDocuments+Filters ? > > And should I write my own collector or the TopDocs one is perfect for me ? > > - Mike > aka...@gmail.com > > > On Mon, Nov 30, 2009 at 11:30 AM, Erick Erickson > <erickerick...@gmail.com>wrote: > >> The problem with hits is that a it re-executes the query >> every N documents where N is 100 (?). >> >> So, a loop like >> for (int idx : hits.length) { >> do something.... >> } >> >> Assuming my memory is right and it's every 100, your query will >> re-execute (length/100) times. Which is unfortunate. >> >> The very quick test to see where to concentrate first would be to take >> a time stamp just before you hit your loop..... >> >> This will tell you whether this loop is the culprit, but it really doesn't >> matter because you'll follow the advice from Uwe and Shai anyway <G>. >> >> Filtering and Sorting are applied to Collectors before you see them..... >> >> The other bit would be to investigate your sorting. Remember that the >> first sort or two take quite a while since the relevant caches are >> populated with first used, so second+ queries should be faster. The >> Wiki has some timing/speedup advice..... >> >> Best >> Erick >> >> >> On Mon, Nov 30, 2009 at 11:10 AM, Michel Nadeau <aka...@gmail.com> wrote: >> >> > What is the main difference between Hits and Collectors? >> > >> > - Mike >> > aka...@gmail.com >> > >> > >> > On Mon, Nov 30, 2009 at 11:03 AM, Uwe Schindler <u...@thetaphi.de> wrote: >> > >> > > And if you only have a filter and apply it to all documents, make a >> > > ConstantScoreQuery on top of the filter: >> > > >> > > Query q=new ConstantScoreQuery(cluCF); >> > > >> > > Then remove the filter from your search method call and only execute >> this >> > > query. >> > > >> > > And if you iterate over all results never-ever use Hits! (its already >> > > deprecated). Write a Collector instead (as you are not interested in >> > > scoring). >> > > >> > > And: If you replace a relational database with Lucene, be sure not to >> > think >> > > in a relational sense with foreign keys / primary keys and so on. In >> > > general >> > > you should flatten everything. >> > > >> > > Uwe >> > > >> > > ----- >> > > Uwe Schindler >> > > H.-H.-Meier-Allee 63, D-28213 Bremen >> > > http://www.thetaphi.de >> > > eMail: u...@thetaphi.de >> > > >> > > >> > > > -----Original Message----- >> > > > From: Shai Erera [mailto:ser...@gmail.com] >> > > > Sent: Monday, November 30, 2009 4:56 PM >> > > > To: java-user@lucene.apache.org >> > > > Subject: Re: Performance problems with Lucene 2.9 >> > > > >> > > > Hi >> > > > >> > > > First you can use MatchAllDocsQuery, which matches all documents. It >> > will >> > > > save a HUGE posting list (TAG:TAG), and performs much faster. For >> > example >> > > > TAG:TAG computes a score for each doc, even though you don't need it. >> > > > MatchAllDocsQuery doesn't. >> > > > >> > > > Second, move away from Hits ! :) Use Collectors instead. >> > > > >> > > > If I understand the chain of filters, do you think you can code them >> > with >> > > > a >> > > > BooleanQuery that is added BooleanClauses, each with is Term >> > > > (field:value)? >> > > > You can add clauses w/ OR, AND, NOT etc. >> > > > >> > > > Note that in Lucene 2.9, you can avoid scoring documents very easily, >> > > > which >> > > > is a performance win if you don't need scores (i.e. if you just want >> to >> > > > match everything, not caring for scores). >> > > > >> > > > Shai >> > > > >> > > > On Mon, Nov 30, 2009 at 5:47 PM, Michel Nadeau <aka...@gmail.com> >> > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > we use Lucene to store around 300 millions of records. We use the >> > index >> > > > > both >> > > > > for conventional searching, but also for all the system's data - we >> > > > > replaced >> > > > > MySQL with Lucene because it was simply not working at all with >> MySQL >> > > > due >> > > > > to >> > > > > the amount or records. Our problem is that we have HUGE performance >> > > > > problems... whenever we search, it takes forever to return results, >> > and >> > > > > Java >> > > > > uses 100% CPU/RAM. >> > > > > >> > > > > Our index fields are like this: >> > > > > >> > > > > TYPE >> > > > > PK >> > > > > FOREIGN_PK >> > > > > TAG >> > > > > ...other information depending on type... >> > > > > >> > > > > * All fields are Field.Index.UN_TOKENIZED >> > > > > * The field "TAG" always contains the value "TAG". >> > > > > >> > > > > Whenever we search in the index, our query is "TAG:TAG" to match >> all >> > > > > documents, and we do the search like this: >> > > > > >> > > > > // Search >> > > > > Hits h = searcher.search(q, cluCF, cluSort); >> > > > > >> > > > > cluCF is a ChainedFilter containing all the other filters (like >> > > > > FOREIGN_PK=12345, TYPE=a, etc.). >> > > > > >> > > > > I know that the method is probably crazy because "TAG:TAG" is >> > matching >> > > > all >> > > > > 300M documents and then it applies filters; so that's probably why >> > > every >> > > > > little query is taking 100% CPU/RAM.... but I don't know how to do >> it >> > > > > properly. >> > > > > >> > > > > Help ! Any advice is welcome. >> > > > > >> > > > > - Mike >> > > > > aka...@gmail.com >> > > > > >> > > >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> > > For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > >> > > >> > >> > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
