Re: Mixing database and lucene searches

2004-05-12 Thread Glen Stampoultzis
I think I follow what you're saying. Thanks Phil. Regards, Glen Phil brunet [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] -- Snip -- If you can't guaranty a fixed number of Lucene results (and it is often the case !), a good way is to duplicate the last PK and so to round

Re: Mixing database and lucene searches

2004-05-11 Thread Matt Quail
Eric Jain wrote: To ask a silly question: What approach does Lucene use for ranges and sorting? A range such as '10-60' is expanded into a boolean query containing all terms that are in the index and lie within the specified range, e.g. '10 or 11 or 20 or 59'. Yes, using a range search requires

Re: Mixing database and lucene searches

2004-05-11 Thread Erik Hatcher
On May 11, 2004, at 7:41 PM, Glen Stampoultzis wrote: Is it possible to use float and date ranges in that case? Or maybe I should just read the details in the manual and stop asking stupid questions. :-) Terms in the index are *always* text. The DateField class in Lucene helps do conversions,

Re: Mixing database and lucene searches

2004-05-11 Thread Paul
Erik Hatcher wrote: On May 11, 2004, at 7:41 PM, Glen Stampoultzis wrote: Is it possible to use float and date ranges in that case? Or maybe I should just read the details in the manual and stop asking stupid questions. :-) Terms in the index are *always* text. The DateField class

Re: Mixing database and lucene searches

2004-05-11 Thread Paul
Matt Quail wrote: But presumably if you make sure that the field name is indexed in Lucene (not necessarily stored), it's far quicker to do a two-stage process, where you search Lucene for text:foo AND name:matt and then just fill-in any other metadata with a fast little DB lookup for the

Re: Mixing database and lucene searches

2004-05-11 Thread Phil brunet
] Subject: Re: Mixing database and lucene searches Date: Tue, 11 May 2004 14:39:23 +1000 Thanks for the help. I think I'll got with putting all the fields into lucene. Just one comment about your strategy for combining db and lucene searches. It seems that it would slow down significantly

Re: Mixing database and lucene searches

2004-05-11 Thread Eric Jain
If you *really* don't want to (or can't) put all the searchable fields into lucene, then you are going to need to do a lucene-db join. Here are two good reasons: 1. Range queries 2. Sorting Yes, Lucene can do both, but I find that in both cases the approach Lucene uses is not suitable for

Re: Mixing database and lucene searches

2004-05-11 Thread Erik Hatcher
On May 10, 2004, at 11:02 PM, Glen Stampoultzis wrote: Hi... I have a query screen where most of the fields search a regular database but one field searches for text in the body of the document. You could say the database holds metadata about the documents. Effectively this means I have two

Re: Mixing database and lucene searches

2004-05-11 Thread Glen Stampoultzis
Phil brunet [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] -- Snip -- If you can't guaranty a fixed number of Lucene results (and it is often the case !), a good way is to duplicate the last PK and so to round to a fixed number. Hi... I'm not sure what you mean by that last

Re: Mixing database and lucene searches

2004-05-11 Thread Glen Stampoultzis
Eric Jain [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] If you *really* don't want to (or can't) put all the searchable fields into lucene, then you are going to need to do a lucene-db join. Here are two good reasons: 1. Range queries 2. Sorting Yes, Lucene can do both, but

Re: Mixing database and lucene searches

2004-05-11 Thread Glen Stampoultzis
Eric Jain [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] If you *really* don't want to (or can't) put all the searchable fields into lucene, then you are going to need to do a lucene-db join. Here are two good reasons: 1. Range queries 2. Sorting Yes, Lucene can do both, but

Re: Mixing database and lucene searches

2004-05-11 Thread Eric Jain
To ask a silly question: What approach does Lucene use for ranges and sorting? A range such as '10-60' is expanded into a boolean query containing all terms that are in the index and lie within the specified range, e.g. '10 or 11 or 20 or 59'. For sorting, Lucene loads into memory all terms

Re: Mixing database and lucene searches

2004-05-11 Thread Gerard Sychay
Eric Jain [EMAIL PROTECTED] 05/11/04 04:47AM Hits hits = searcher.search(new TermQuery(text, foo) Set hitPKs = new Set(); for each doc in hits: hitPKs.put(doc.getField(pk)) Retrieving even one custom field for every document of a possibly large data set can end up being very

Re: Mixing database and lucene searches

2004-05-11 Thread Ype Kingma
On Tuesday 11 May 2004 17:26, Gerard Sychay wrote: Eric Jain [EMAIL PROTECTED] 05/11/04 04:47AM Hits hits = searcher.search(new TermQuery(text, foo) Set hitPKs = new Set(); for each doc in hits: hitPKs.put(doc.getField(pk)) Retrieving even one custom field for every document

Mixing database and lucene searches

2004-05-10 Thread Glen Stampoultzis
Hi... I have a query screen where most of the fields search a regular database but one field searches for text in the body of the document. You could say the database holds metadata about the documents. Effectively this means I have two separate queries going on - a lucene query and a database

Re: Mixing database and lucene searches

2004-05-10 Thread Paul
Glen Stampoultzis wrote: Hi... I have a query screen where most of the fields search a regular database but one field searches for text in the body of the document. You could say the database holds metadata about the documents. Effectively this means I have two separate queries going on - a

Re: Mixing database and lucene searches

2004-05-10 Thread Matt Quail
Glen Stampoultzis wrote: Anyone have any strategies for dealing with this? I'm wondering whether it's better to replicate searchable fields in the lucene index. This means being very careful that updates get done in two places so it is not ideal. If you *can* manage to update your index when the

Re: Mixing database and lucene searches

2004-05-10 Thread Paul
Matt Quail wrote: And join: if you want to do a query like select rows where name='matt' and text contains 'foo', then the psuedo code is: Hits hits = searcher.search(new TermQuery(text, foo) Set hitPKs = new Set(); for each doc in hits: hitPKs.put(doc.getField(pk)) rsPKs = new Set();