Eric, I was pursuing a different direction yesterday which is not fast enough. Basically I was using the highlighter to figure out if a page has a hit or not. But that is too expensive. I end up with 15 ms per page and that sums up.
I have to allow ad-hoc queries, so it sounds like the solution will not be trivial from what I understand from your reply. Just one question before I start using the SpanQuery approach: Do you have anything you can say about performance regarding your solution? How many pages per document in average you have to deal with? How performant or expensive is the approach? As usual, many thanks in advance for your input. Andreas -----Original Message----- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Wednesday, May 23, 2007 2:05 PM To: java-user@lucene.apache.org Subject: Re: How to filter fields with hits from result set Two things to watch... 1> Think about indexing the special page-end token with an increment gap of 0 (see SynonymAnalyzer in Lucene In Action). That preserves the sense of phrases across page breaks. 2> Assembling the span query is tricky. Search the mail archive for SpanQuery to see an exchange I had with the originator of this concept. Suffice it to say that converting an ad-hoc query into a set of SpanQueries is not trivial, but it certainly is do-able. But you'd have a much easier time of it if you were able to control the queries and dis-allow ad-hoc queries. It all depends upon the requirements of the application. Any time you can avoid supporting arbitrary boolean logic for the user input, your job is easier <G>.... But you should be able to run up a demo with simple queries that you control to prove out the methodology in any case..... Best Erick On 5/23/07, Andreas Guther <[EMAIL PROTECTED]> wrote: > > Eric, > > Thank you very much for your response. That sounds very interesting. > Let me do some experimenting to see if I fully understood your solution. > Otherwise I have to come back to you with more questions. > > Andreas > > > > > -----Original Message----- > From: Erick Erickson [mailto:[EMAIL PROTECTED] > Sent: Wednesday, May 23, 2007 12:00 PM > To: java-user@lucene.apache.org > Subject: Re: How to filter fields with hits from result set > > As luck would have it, I've done something very similar. What I had > to do is index a special token at the end of each page. Then I could > get the term offsets for each page.... > > Then I used one of the SpanQuery.getSpans to get all of the > offsets of the hits throughout all of the pages. > > now I have a list of all the offsets of the *last* term on each > page and a list of the offsets of the hits. From these two > lists I can know which pages have hits. > > > Best > Erick > > On 5/23/07, Andreas Guther <[EMAIL PROTECTED]> wrote: > > > > Hi, > > > > If a search returns a document that has multiple fields with the same > > name, is there a way to filter only those fields that contain hits? > > > > > > Background: > > > > I am indexing documents and we store all content in our index for > > display reasons. We want to show only those pages containing hits. > My > > first implementation was saving each page in a Lucene document. For > > performance reasons why are now looking into indexing the complete > > indexed document as a single Lucene document. > > > > Every page is added to a field in the Lucene document named > > page-content. That means I am ending with as many fields named > > page-content as the document has pages. > > > > My search now returns me a single Lucene document in contrary to my > > first approach with page per Lucene document. My problem right now > is: > > how can I limit the returned page-contents fields for pages to those > > field entries that contain hits. If I have hits on pages five pages > > from a document with 10 pages I would like to have only the pages with > > the hits, not all. > > > > Is there anything in Lucene that limits the returned fields to fields > > with hits only? > > > > Thanks in advance, > > > > Andreas > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
