On Sat, Nov 15, 2003 at 11:30:36AM -0500, Erik Hatcher wrote:
> On Friday, November 14, 2003, at 07:16 PM, Dror Matalon wrote:
> >We're seeing slow response time when we apply datefilter. A search that
> >takes 7 msec with no datefilter takes 368 msec when I filter on the
> >last
> >fifteen days, and 632 msec on the last 30 days.
> >
> >Initially we saved doing
> >document.add(Field.Keyword("dtstamp", dtstamp));
> >
> >and then change to doing
> >document.add(Field.Keyword("dtstamp",
> >DateField.dateToString(dtstamp)));
> >
> >where dtstamp is a java.util.Date
>
> Both of the above lines of code are equivalent. This is where having
> open-source is handy :)
>
> public static final Field Keyword(String name, Date value) {
> return new Field(name, DateField.dateToString(value), true, true,
> false);
> }
>
> >We search doing the following:
> >
> > days_ago_value = Long.parseLong(days); //could throw
> >NumberFormatException
> > days_ago_value = new java.util.Date().getTime() - (days_ago_value *
> >86400000L);
> > hits = indexSearcher.search(query, DateFilter.After("dtstamp",
> >days_ago_value));
>
> DateFilter itself is walking all the terms in the range you provide
> before executing the query. If you have a lot of terms in that range,
> you can see that there is obviously some cycles spinning to do the work
> needed.
I see. So for large data sets when a small number of items are returned,
it might be cheaper to just fetch the documents and see if they fall in
the date range.
>
> >Not only is the query slow, but it seems to be slower the more results
> >it returns.
> >
> >
> >Any suggestions?
>
> If this date range is pretty static, you could (in Lucene's CVS
> codebase) wrap the DateFilter with a CachingWrappingFilter. Or you
> could construct a long-lived instance of an equivalent QueryFilter and
> reuse it across multiple queries. You would likely see dramatic
> differences using either of these approaches.
>
I did some timing tests and it looks like time is spent in the actual
query rather than in creating the filter object. This seems to
contradict your suggestion here, or am I misunderstanding how filters
work?
Thanks for the feedback,
Dror
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
--
Dror Matalon
Zapatec Inc
1700 MLK Way
Berkeley, CA 94709
http://www.fastbuzz.com
http://www.zapatec.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
