RE: Filtering w/ Multiple Terms
After re-reading the book (again), and the javadocs (again), it dawned on my little brain that I needed to have a doc and freq array *the size of maxDocs* for the index reader. I also needed to iterate through the docs array and call bitSet.set for each entry in docs (that was valid, of course). Everything is good now Thanks! Jerry Jalenak Senior Programmer / Analyst, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] > -Original Message- > From: Erik Hatcher [mailto:[EMAIL PROTECTED] > Sent: Monday, January 24, 2005 1:27 PM > To: Lucene Users List > Subject: Re: Filtering w/ Multiple Terms > > > As Paul suggested, output the Lucene document numbers from your Hits, > and also output which bit you're setting in your filter. Do > those sets > overlap? > > Erik > > On Jan 24, 2005, at 2:13 PM, Jerry Jalenak wrote: > > > Paul / Erik - > > > > I'm use the ParallelMultiSearcher to search three indexes > concurrently > > - > > hence the three entries into AccountFilter. If I remove the filter > > from my > > query, and simply enter the query on the command line, I > get two hits > > back. > > In other words, I can enter this: > > > > smith AND (account:0011) > > > > and get hits back. When I add the filter back in (which > should take > > care of > > the account:0011 part of the query), and enter only smith > as my query, > > I get > > 0 hits. > > > > > > > > Jerry Jalenak > > Senior Programmer / Analyst, Web Publishing > > LabOne, Inc. > > 10101 Renner Blvd. > > Lenexa, KS 66219 > > (913) 577-1496 > > > > [EMAIL PROTECTED] > > > > > >> -Original Message- > >> From: Erik Hatcher [mailto:[EMAIL PROTECTED] > >> Sent: Monday, January 24, 2005 1:07 PM > >> To: Lucene Users List > >> Subject: Re: Filtering w/ Multiple Terms > >> > >> > >> > >> On Jan 24, 2005, at 12:26 PM, Jerry Jalenak wrote: > >>> I spent some time reading the Lucene in Action book this weekend > >>> (great job, > >>> btw) > >> > >> Thanks! > >> > >>> public class AccountFilter extends Filter > >>> I see where the AccountFilter is setting the cooresponding > >> 'bits', but > >>> I end > >>> up without any 'hits': > >>> > >>> Entering AccountFilter... > >>> Entering AccountFilter... > >>> Entering AccountFilter... > >>> Setting bit on > >>> Setting bit on > >>> Setting bit on > >>> Setting bit on > >>> Setting bit on > >>> Leaving AccountFilter... > >>> Leaving AccountFilter... > >>> Leaving AccountFilter... > >>> ... Found 0 matching documents in 1000 ms > >>> > >>> Can anyone tell me what I've done wrong? > >> > >> A filter constrains which documents will be consulted during > >> a search, > >> but the Query needs to match some documents that are > turned on by the > >> filter bits. I'm guessing that your Query did not match any of the > >> documents you turned on. > >> > >>Erik > >> > >> > >> > - > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: > [EMAIL PROTECTED] > >> > >> > > > > This transmission (and any information attached to it) may be > > confidential and > > is intended solely for the use of the individual or entity > to which it > > is > > addressed. If you are not the intended recipient or the person > > responsible for > > delivering the transmission to the intended recipient, be > advised that > > you > > have received this transmission in error and that any use, > > dissemination, > > forwarding, printing, or copying of this information is strictly > > prohibited. > > If you have received this transmission in error, please immediately > > notify > > LabOne at the following email address: > > [EMAIL PROTECTED] > > > > > > > - > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Filtering w/ Multiple Terms
As Paul suggested, output the Lucene document numbers from your Hits, and also output which bit you're setting in your filter. Do those sets overlap? Erik On Jan 24, 2005, at 2:13 PM, Jerry Jalenak wrote: Paul / Erik - I'm use the ParallelMultiSearcher to search three indexes concurrently - hence the three entries into AccountFilter. If I remove the filter from my query, and simply enter the query on the command line, I get two hits back. In other words, I can enter this: smith AND (account:0011) and get hits back. When I add the filter back in (which should take care of the account:0011 part of the query), and enter only smith as my query, I get 0 hits. Jerry Jalenak Senior Programmer / Analyst, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] -Original Message- From: Erik Hatcher [mailto:[EMAIL PROTECTED] Sent: Monday, January 24, 2005 1:07 PM To: Lucene Users List Subject: Re: Filtering w/ Multiple Terms On Jan 24, 2005, at 12:26 PM, Jerry Jalenak wrote: I spent some time reading the Lucene in Action book this weekend (great job, btw) Thanks! public class AccountFilter extends Filter I see where the AccountFilter is setting the cooresponding 'bits', but I end up without any 'hits': Entering AccountFilter... Entering AccountFilter... Entering AccountFilter... Setting bit on Setting bit on Setting bit on Setting bit on Setting bit on Leaving AccountFilter... Leaving AccountFilter... Leaving AccountFilter... ... Found 0 matching documents in 1000 ms Can anyone tell me what I've done wrong? A filter constrains which documents will be consulted during a search, but the Query needs to match some documents that are turned on by the filter bits. I'm guessing that your Query did not match any of the documents you turned on. Erik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Filtering w/ Multiple Terms
Paul / Erik - I'm use the ParallelMultiSearcher to search three indexes concurrently - hence the three entries into AccountFilter. If I remove the filter from my query, and simply enter the query on the command line, I get two hits back. In other words, I can enter this: smith AND (account:0011) and get hits back. When I add the filter back in (which should take care of the account:0011 part of the query), and enter only smith as my query, I get 0 hits. Jerry Jalenak Senior Programmer / Analyst, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] > -Original Message- > From: Erik Hatcher [mailto:[EMAIL PROTECTED] > Sent: Monday, January 24, 2005 1:07 PM > To: Lucene Users List > Subject: Re: Filtering w/ Multiple Terms > > > > On Jan 24, 2005, at 12:26 PM, Jerry Jalenak wrote: > > I spent some time reading the Lucene in Action book this weekend > > (great job, > > btw) > > Thanks! > > > public class AccountFilter extends Filter > > I see where the AccountFilter is setting the cooresponding > 'bits', but > > I end > > up without any 'hits': > > > > Entering AccountFilter... > > Entering AccountFilter... > > Entering AccountFilter... > > Setting bit on > > Setting bit on > > Setting bit on > > Setting bit on > > Setting bit on > > Leaving AccountFilter... > > Leaving AccountFilter... > > Leaving AccountFilter... > > ... Found 0 matching documents in 1000 ms > > > > Can anyone tell me what I've done wrong? > > A filter constrains which documents will be consulted during > a search, > but the Query needs to match some documents that are turned on by the > filter bits. I'm guessing that your Query did not match any of the > documents you turned on. > > Erik > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Filtering w/ Multiple Terms
On Jan 24, 2005, at 12:26 PM, Jerry Jalenak wrote: I spent some time reading the Lucene in Action book this weekend (great job, btw) Thanks! public class AccountFilter extends Filter I see where the AccountFilter is setting the cooresponding 'bits', but I end up without any 'hits': Entering AccountFilter... Entering AccountFilter... Entering AccountFilter... Setting bit on Setting bit on Setting bit on Setting bit on Setting bit on Leaving AccountFilter... Leaving AccountFilter... Leaving AccountFilter... ... Found 0 matching documents in 1000 ms Can anyone tell me what I've done wrong? A filter constrains which documents will be consulted during a search, but the Query needs to match some documents that are turned on by the filter bits. I'm guessing that your Query did not match any of the documents you turned on. Erik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Filtering w/ Multiple Terms
Jerry,
On Monday 24 January 2005 18:26, Jerry Jalenak wrote:
> I spent some time reading the Lucene in Action book this weekend (great job,
> btw), and came across the section on using custom filters. Since the data
> that I need to use to filter my hit set with comes from a database, I
> thought it would be worth my effort this morning to write a custom filter
> that would handle the filtering for me. So, using the example from the book
> (page 210), I've coded an AccountFilter:
>
> public class AccountFilter extends Filter
> {
> public AccountFilter()
> {}
>
> public BitSet bits(IndexReader indexReader)
> throws IOException
> {
> System.out.println("Entering AccountFilter...");
> BitSet bitSet = new BitSet(indexReader.maxDoc());
>
> String[] reportingAccounts = new String[] {"0011", "4kfs"};
>
> int[] docs = new int[1];
> int[] freqs = new int[1];
>
> for (int i = 0; i < reportingAccounts.length; i++)
> {
> String reportingAccount = reportingAccounts[i];
> if (reportingAccount != null)
> {
> TermDocs termDocs = indexReader.termDocs(new
> Term("account", reportingAccount));
> int count = termDocs.read(docs, freqs);
> if (count == 1)
Unless "account" is a primary key fied, it's better to loop over the termdocs.
> {
> System.out.println("Setting bit
> on");
> bitSet.set(docs[0]);
> }
> }
> }
> System.out.println("Leaving AccountFilter...");
> return bitSet;
> }
> }
>
> I see where the AccountFilter is setting the cooresponding 'bits', but I end
> up without any 'hits':
>
> Entering AccountFilter...
> Entering AccountFilter...
> Entering AccountFilter...
> Setting bit on
> Setting bit on
> Setting bit on
> Setting bit on
> Setting bit on
> Leaving AccountFilter...
> Leaving AccountFilter...
> Leaving AccountFilter...
I don't see any recursion in your code, but this output
suggests nesting three deep. Something does not add up here.
> ... Found 0 matching documents in 1000 ms
>
> Can anyone tell me what I've done wrong?
Maybe all query hits were filtered out?
Could you compare the docnrs in the bits of the filter with the
unfiltered query hits docnrs?
Regards,
Paul Elschot
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Filtering w/ Multiple Terms
I spent some time reading the Lucene in Action book this weekend (great job,
btw), and came across the section on using custom filters. Since the data
that I need to use to filter my hit set with comes from a database, I
thought it would be worth my effort this morning to write a custom filter
that would handle the filtering for me. So, using the example from the book
(page 210), I've coded an AccountFilter:
public class AccountFilter extends Filter
{
public AccountFilter()
{}
public BitSet bits(IndexReader indexReader)
throws IOException
{
System.out.println("Entering AccountFilter...");
BitSet bitSet = new BitSet(indexReader.maxDoc());
String[] reportingAccounts = new String[] {"0011", "4kfs"};
int[] docs = new int[1];
int[] freqs = new int[1];
for (int i = 0; i < reportingAccounts.length; i++)
{
String reportingAccount = reportingAccounts[i];
if (reportingAccount != null)
{
TermDocs termDocs = indexReader.termDocs(new
Term("account", reportingAccount));
int count = termDocs.read(docs, freqs);
if (count == 1)
{
System.out.println("Setting bit
on");
bitSet.set(docs[0]);
}
}
}
System.out.println("Leaving AccountFilter...");
return bitSet;
}
}
I see where the AccountFilter is setting the cooresponding 'bits', but I end
up without any 'hits':
Entering AccountFilter...
Entering AccountFilter...
Entering AccountFilter...
Setting bit on
Setting bit on
Setting bit on
Setting bit on
Setting bit on
Leaving AccountFilter...
Leaving AccountFilter...
Leaving AccountFilter...
... Found 0 matching documents in 1000 ms
Can anyone tell me what I've done wrong?
Jerry Jalenak
Senior Programmer / Analyst, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS 66219
(913) 577-1496
[EMAIL PROTECTED]
> -Original Message-
> From: Otis Gospodnetic [mailto:[EMAIL PROTECTED]
> Sent: Friday, January 21, 2005 8:15 AM
> To: Lucene Users List
> Subject: RE: Filtering w/ Multiple Terms
>
>
> This:
> http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/se
> arch/BooleanQuery.TooManyClauses.html
> ?
>
> You can control that limit via
> http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/se
> arch/BooleanQuery.html#maxClauseCount
>
> Otis
>
>
> --- Jerry Jalenak <[EMAIL PROTECTED]> wrote:
>
> > OK. But isn't there a limit on the number of
> BooleanQueries that can
> > be
> > combined with AND / OR / etc?
> >
> >
> >
> > Jerry Jalenak
> > Senior Programmer / Analyst, Web Publishing
> > LabOne, Inc.
> > 10101 Renner Blvd.
> > Lenexa, KS 66219
> > (913) 577-1496
> >
> > [EMAIL PROTECTED]
> >
> >
> > > -Original Message-
> > > From: Erik Hatcher [mailto:[EMAIL PROTECTED]
> > > Sent: Thursday, January 20, 2005 5:05 PM
> > > To: Lucene Users List
> > > Subject: Re: Filtering w/ Multiple Terms
> > >
> > >
> > >
> > > On Jan 20, 2005, at 5:02 PM, Jerry Jalenak wrote:
> > >
> > > > In looking at the examples for filtering of hits, it looks
> > > like I can
> > > > only
> > > > specify a single term; i.e.
> > > >
> > > > Filter f = new QueryFilter(new TermQuery(new
> Term("acct",
> > > > "acct1")));
> > > >
> > > > I need to specify more than one term in my filter. Short of
> > using
> > > > something
> > > > like ChainFilter, how are others handling this?
> > >
> > > You can make as complex of a Query as you want for
> > > QueryFilter. If you
> > > want to filter on multiple terms, construct a BooleanQuery
> > > with nested
> > > TermQuery's, either in an AND or OR fashion.
> > >
> > > Erik
> > >
> > >
> > >
> >
> -
> > > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > For additional commands, e-mail:
> > [EMAIL PROTECTED]
> > >
>
RE: Filtering w/ Multiple Terms
This:
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/BooleanQuery.TooManyClauses.html
?
You can control that limit via
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/BooleanQuery.html#maxClauseCount
Otis
--- Jerry Jalenak <[EMAIL PROTECTED]> wrote:
> OK. But isn't there a limit on the number of BooleanQueries that can
> be
> combined with AND / OR / etc?
>
>
>
> Jerry Jalenak
> Senior Programmer / Analyst, Web Publishing
> LabOne, Inc.
> 10101 Renner Blvd.
> Lenexa, KS 66219
> (913) 577-1496
>
> [EMAIL PROTECTED]
>
>
> > -Original Message-
> > From: Erik Hatcher [mailto:[EMAIL PROTECTED]
> > Sent: Thursday, January 20, 2005 5:05 PM
> > To: Lucene Users List
> > Subject: Re: Filtering w/ Multiple Terms
> >
> >
> >
> > On Jan 20, 2005, at 5:02 PM, Jerry Jalenak wrote:
> >
> > > In looking at the examples for filtering of hits, it looks
> > like I can
> > > only
> > > specify a single term; i.e.
> > >
> > > Filter f = new QueryFilter(new TermQuery(new Term("acct",
> > > "acct1")));
> > >
> > > I need to specify more than one term in my filter. Short of
> using
> > > something
> > > like ChainFilter, how are others handling this?
> >
> > You can make as complex of a Query as you want for
> > QueryFilter. If you
> > want to filter on multiple terms, construct a BooleanQuery
> > with nested
> > TermQuery's, either in an AND or OR fashion.
> >
> > Erik
> >
> >
> >
> -
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail:
> [EMAIL PROTECTED]
> >
> >
>
> This transmission (and any information attached to it) may be
> confidential and
> is intended solely for the use of the individual or entity to which
> it is
> addressed. If you are not the intended recipient or the person
> responsible for
> delivering the transmission to the intended recipient, be advised
> that you
> have received this transmission in error and that any use,
> dissemination,
> forwarding, printing, or copying of this information is strictly
> prohibited.
> If you have received this transmission in error, please immediately
> notify
> LabOne at the following email address:
> [EMAIL PROTECTED]
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: Filtering w/ Multiple Terms
OK. But isn't there a limit on the number of BooleanQueries that can be
combined with AND / OR / etc?
Jerry Jalenak
Senior Programmer / Analyst, Web Publishing
LabOne, Inc.
10101 Renner Blvd.
Lenexa, KS 66219
(913) 577-1496
[EMAIL PROTECTED]
> -Original Message-
> From: Erik Hatcher [mailto:[EMAIL PROTECTED]
> Sent: Thursday, January 20, 2005 5:05 PM
> To: Lucene Users List
> Subject: Re: Filtering w/ Multiple Terms
>
>
>
> On Jan 20, 2005, at 5:02 PM, Jerry Jalenak wrote:
>
> > In looking at the examples for filtering of hits, it looks
> like I can
> > only
> > specify a single term; i.e.
> >
> > Filter f = new QueryFilter(new TermQuery(new Term("acct",
> > "acct1")));
> >
> > I need to specify more than one term in my filter. Short of using
> > something
> > like ChainFilter, how are others handling this?
>
> You can make as complex of a Query as you want for
> QueryFilter. If you
> want to filter on multiple terms, construct a BooleanQuery
> with nested
> TermQuery's, either in an AND or OR fashion.
>
> Erik
>
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
This transmission (and any information attached to it) may be confidential and
is intended solely for the use of the individual or entity to which it is
addressed. If you are not the intended recipient or the person responsible for
delivering the transmission to the intended recipient, be advised that you
have received this transmission in error and that any use, dissemination,
forwarding, printing, or copying of this information is strictly prohibited.
If you have received this transmission in error, please immediately notify
LabOne at the following email address: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Filtering w/ Multiple Terms
On Jan 20, 2005, at 5:02 PM, Jerry Jalenak wrote:
In looking at the examples for filtering of hits, it looks like I can
only
specify a single term; i.e.
Filter f = new QueryFilter(new TermQuery(new Term("acct",
"acct1")));
I need to specify more than one term in my filter. Short of using
something
like ChainFilter, how are others handling this?
You can make as complex of a Query as you want for QueryFilter. If you
want to filter on multiple terms, construct a BooleanQuery with nested
TermQuery's, either in an AND or OR fashion.
Erik
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Filtering w/ Multiple Terms
Maybe you should try making a BooleanQuery out of the TermQuerys and
then passing that to QueryFilter. I've never tried it, but it should
work, right?
- andy g
On Thu, 20 Jan 2005 16:02:26 -0600, Jerry Jalenak
<[EMAIL PROTECTED]> wrote:
> In looking at the examples for filtering of hits, it looks like I can only
> specify a single term; i.e.
>
> Filter f = new QueryFilter(new TermQuery(new Term("acct",
> "acct1")));
>
> I need to specify more than one term in my filter. Short of using something
> like ChainFilter, how are others handling this?
>
> Thanks!
>
> Jerry Jalenak
> Senior Programmer / Analyst, Web Publishing
> LabOne, Inc.
> 10101 Renner Blvd.
> Lenexa, KS 66219
> (913) 577-1496
>
> [EMAIL PROTECTED]
>
> This transmission (and any information attached to it) may be confidential and
> is intended solely for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient or the person responsible for
> delivering the transmission to the intended recipient, be advised that you
> have received this transmission in error and that any use, dissemination,
> forwarding, printing, or copying of this information is strictly prohibited.
> If you have received this transmission in error, please immediately notify
> LabOne at the following email address: [EMAIL PROTECTED]
>
> -
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
