It strikes me that removing just the seconds could very well reduce overhead to 1/60 of original. 30 second query turns into 500ms query. Just a swag though.
-Todd -----Original Message----- From: Alok Dhir [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 29, 2008 1:48 PM To: solr-user@lucene.apache.org Subject: Re: date range query performance Well, no - we don't care so much about the seconds, but hours & minutes are indeed crucial. --- Alok K. Dhir Symplicity Corporation www.symplicity.com (703) 351-0200 x 8080 [EMAIL PROTECTED] On Oct 29, 2008, at 4:41 PM, Chris Harris wrote: > Do you need to search down to the minutes and seconds level? If > searching by > date provides sufficient granularity, for instance, you can > normalize all > the time-of-day portions of the timestamps to midnight while > indexing. (So > index any event happening on Oct 01, 2008 as 2008-10-01T00:00:00Z.) > That > would give Solr many fewer unique timestamp values to go through. > > On Wed, Oct 29, 2008 at 1:30 PM, Alok Dhir <[EMAIL PROTECTED]> > wrote: > >> Hi -- using solr 1.3 -- roughly 11M docs on a 64 gig 8 core machine. >> >> Fairly simple schema -- no large text fields, standard request >> handler. 4 >> small facet fields. >> >> The index is an event log -- a primary search/retrieval requirement >> is date >> range queries. >> >> A simple query without a date range subquery is ridiculously fast - >> 2ms. >> The same query with a date range takes up to 30s (30,000ms). >> >> Concrete example, this query just look 18s: >> >> instance:client\-csm.symplicity.com AND dt: >> [2008-10-01T04:00:00Z TO >> 2008-10-30T03:59:59Z] AND label_facet:"Added to Position" >> >> The exact same query without the date range took 2ms. >> >> I saw a thread from Apr 2008 which explains the problem being due >> to too >> much precision on the DateField type, and the range expansion >> leading to far >> too many elements being checked. Proposed solution appears to be a >> hack >> where you index date fields as strings and hacking together date >> functions >> to generate proper queries/format results. >> >> Does this remain the recommended solution to this issue? >> >> Thanks >> >> --- >> Alok K. Dhir >> Symplicity Corporation >> www.symplicity.com >> (703) 351-0200 x 8080 >> [EMAIL PROTECTED] >> >>
