Thank you Erick. This solution fits part of our queries, will adopt it for those. Yet we have use-cases in which the results can not be cached.
Everyone, What do you think about our assumptions and conclusions? As a general rule of thumb, at least in our case, would you please comment on the following assumptions/conclusions (note, all assuming that we don't want to cache filters, and the 'fq' part is only used to avoid scoring): 1) If the query sorts by any other field than score (e.g. date), we can put the 'fq' part in 'q'. Scoring won't be done, and we won't pay the cost of building the filter, and then discarding it when the query completes. 2) In fact, if we don't intend to cache the filter, we might as well just use only 'q'. At least, on our dataset (this may definitely *not* be a general statement). 3) If we sort by relevance, but want to avoid scoring of the 'filter' clauses, is there anything we can do on 4.7? 3.1) The ^= operator is only available in 5.1, which seems exactly what we need. 3.2) Adding the filter clauses to the query w/ boost 0 will still compute their score, only they won't affect the overall document score correct? 4) A more general question -- with the addition of ^= to query clauses in 5.1 (resolved to ConstantScoreQuery down stream), what is the use case for using fq w/ !cache=false? As we understand it, users who use this want to compute a filter but not cache it. As we see, there is some added cost to building a filter, so if you pay this cost over and over, would it not be better to just use ^=? Have a good day, Esther From: Erick Erickson <erickerick...@gmail.com> To: solr-user@lucene.apache.org Date: 25/06/2015 03:27 PM Subject: Re: fq versus q Side note on dates and fqs. If you're using NOW in your date expressions you may be able to re-use fqs by using "date math", see: https://lucidworks.com/blog/date-math-now-and-filter-queries/ Of course this may not be applicable in your situation... FWIW, Erick On Thu, Jun 25, 2015 at 8:03 AM, Shai Erera <ser...@gmail.com> wrote: > The tables came across corrupt, here they are (times in ms): > > Caches enabled: > > q fq delta > original query 28 295 267 > w/o grouping 58 325 267 > w/o sort on date 28 293 265 > > Caches disabled: > > q fq delta > original query 4113 4381 268 > w/o grouping 131 407 276 > w/o sort on date 4217 4400 183 > > Shai > > On Thu, Jun 25, 2015 at 2:04 PM, Esther Goldbraich <estherg...@il.ibm.com> > wrote: > >> Thank you all for collaborative thinking! >> >> Ran additional benchmarks as proposed. Some results: >> >> All solr caches are enabled (queryResultCache hit ratio = 0.02): >> >> >> q >> fq {!cache=false} >> delta >> original query >> 28 >> 295 >> 267 >> w/o grouping >> 58 >> 325 >> 267 >> w/o sort on date >> 28 >> 293 >> 265 >> >> All solr caches are disabled (except built in lucene field cache): >> >> >> q >> fq {!cache=false} >> delta >> original query >> 4113 >> 4381 >> 268 >> w/o grouping >> 131 >> 407 >> 276 >> w/o sort on date >> 4217 >> 4400 >> 183 >> >> *median runtime in ms >> >> As you can see, disabling grouping and/or sorting does not affect the >> results much. That is, the difference between running with >> 'fq{!cache=false}' or with 'q' is the same, while 'fq' performs slower in >> all cases. >> >> Is it correct to assume then that the performance difference comes from >> computing the filter (traversing the posting lists and building the >> bitset)? >> Does it also mean that not caching the filter does not affect grouping? >> I.e. perhaps the second pass of grouping uses the already computed filter, >> and does not attempt to fetch it from the cache? >> >> As a general rule of thumb, at least in our case, would you please comment >> on the following assumptions/conclusions (note, all assuming that we don't >> want to cache filters, and the 'fq' part is only used to avoid scoring): >> >> 1) If the query sorts by any other field than score (e.g. date), we can >> put the 'fq' part in 'q'. Scoring won't be done, and we won't pay the cost >> of building the filter, and then discarding it when the query completes. >> >> 2) In fact, if we don't intend to cache the filter, we might as well just >> use only 'q'. At least, on our dataset (this may definitely *not* be a >> general statement). >> >> 3) If we sort by relevance, but want to avoid scoring of the 'filter' >> clauses, is there anything we can do on 4.7? >> 3.1) The ^= operator is only available in 5.1, which seems exactly what we >> need. >> 3.2) Adding the filter clauses to the query w/ boost 0 will still compute >> their score, only they won't affect the overall document score correct? >> >> 4) A more general question -- with the addition of ^= to query clauses in >> 5.1 (resolved to ConstantScoreQuery down stream), what is the use case for >> using fq w/ !cache=false? As we understand it, users who use this want to >> compute a filter but not cache it. As we see, there is some added cost to >> building a filter, so if you pay this cost over and over, would it not be >> better to just use ^=? >> >> Best regards, >> Esther >> >> >> >> >> From: >> Erick Erickson <erickerick...@gmail.com> >> To: >> solr-user@lucene.apache.org >> Date: >> 25/06/2015 02:38 AM >> Subject: >> Re: fq versus q >> >> >> >> Tell us a bit more about your test setup. 1 or 2 tests >> don't mean much. For instance, if the fq query has to >> load the low-level caches from disk then the q-only >> query is run and doesn't that could skew the results. >> Or if somehow you're hitting the queryResultCache. Or.... >> >> Frankly I'd disable all my caches for running tests like >> this, and be sure to mix-n-match the tests so I wasn't >> getting bitten by caches. >> >> And please tell us what the actual numbers are. 5-10X >> doesn't mean much at all if it's 25ms .vs. 5 ms. It means >> a lot (and something's very wrong) if it means >> 200ms .vs. 1,000ms. >> >> Best, >> Erick >> >> On Wed, Jun 24, 2015 at 5:30 PM, Upayavira <u...@odoko.co.uk> wrote: >> > Are you wanting to do no scoring at all, or just have a portion of the >> > query not contribute to the score? >> > >> > If you don't want scoring at all, just sort by another field. If you >> > don't have a field, I just tried "&sort=1 desc", and it worked! This >> > should, if I'm right, pull documents out of the index in index order. >> > >> > Upayavira >> > >> > On Wed, Jun 24, 2015, at 08:26 PM, Shai Erera wrote: >> >> Ah thanks. I see it was added in 5.1 - is there any other way prior to >> >> that >> >> (like 4.7)? >> >> >> >> if not, I guess the only option is to not use fq if we don't intend to >> >> cache it, and on 5.1 use the ^= syntax. >> >> >> >> Shai >> >> >> >> On Wed, Jun 24, 2015 at 9:21 PM, Jack Krupansky >> >> <jack.krupan...@gmail.com> >> >> wrote: >> >> >> >> > Yonik added syntax to request a constant score query in Solr with the >> ^= >> >> > operator. >> >> > >> >> > For example: +color:blue^=1 text:shoes >> >> > >> >> > See: >> >> > https://issues.apache.org/jira/browse/SOLR-7218 >> >> > >> >> > -- Jack Krupansky >> >> > >> >> > On Wed, Jun 24, 2015 at 1:41 PM, Shai Erera <ser...@gmail.com> wrote: >> >> > >> >> > > Thanks Shawn, >> >> > > >> >> > > What's Solr equivalence to ConstantScoreQuery? I.e., what if you >> want to >> >> > > run a query that does not score, but only filter. The rationale >> behind >> >> > > using a non-cached 'fq' was just that. >> >> > > >> >> > > Shai >> >> > > >> >> > > On Wed, Jun 24, 2015 at 4:29 PM, Shawn Heisey <apa...@elyograg.org> >> >> > wrote: >> >> > > >> >> > > > On 6/24/2015 5:28 AM, Esther Goldbraich wrote: >> >> > > > > We are comparing the performance of fq versus q for queries >> that are >> >> > > > > actually filters and should not be cached. >> >> > > > > In part of queries we see strange behavior where q performs >> 5-10x >> >> > > better >> >> > > > > than fq. The question is why? >> >> > > > > >> >> > > > > An example1: >> >> > > > > q=maildate:{DATE1 to DATE2} COMPARED TO >> >> > > fq={!cache=false}maildate:{DATE1 >> >> > > > > to DATE2} >> >> > > > > sort=maildate_sort* desc >> >> > > > >> >> > > > <snip> >> >> > > > >> >> > > > > <field name="maildate" stored="true" indexed="true" >> type="tdate"/> >> >> > > > > <field name="maildate_sort" stored="false" indexed="false" >> >> > type="tdate" >> >> > > > > docValues="true"/> >> >> > > > >> >> > > > For simplicity, I would probably just use one field for that, >> rather >> >> > > > than a separate sort field. The disk space required would >> probably be >> >> > > > the same either way, but your interaction with the index will not >> be as >> >> > > > complex. There's nothing wrong with doing it the way you have, >> though. >> >> > > > >> >> > > > I'm not at all an expert, but I've been a member of this >> community for >> >> > a >> >> > > > long time. Here's my guess about why your query is faster in the >> q >> >> > > > parameter than a non-cached filter: >> >> > > > >> >> > > > The result of a standard query is the stored fields from the top >> N >> >> > > > documents, where N is the value in the rows parameter. The >> default for >> >> > > > N is typically set to 10, and for most people will normally be >> 200 or >> >> > > less. >> >> > > > >> >> > > > The result of a filter is very different -- it is a bitset of all >> the >> >> > > > documents in your entire index, with binary 0 for documents that >> don't >> >> > > > match the filter and binary 1 for documents that do match. >> >> > > > >> >> > > > If your index has 100 million documents, every single one of >> those 100 >> >> > > > million documents must be checked against the filter query to >> produce a >> >> > > > filter bitset, but when it's in the q parameter, shortcuts can be >> taken >> >> > > > which will get the top N results quickly. >> >> > > > >> >> > > > The filterCache levels the playing field when filters are >> re-used. If >> >> > a >> >> > > > requested filter is already in the cache, it can be retrieved and >> >> > > > applied to a result VERY quickly. >> >> > > > >> >> > > > You have turned off the caching for your filter. I'm not sure >> why you >> >> > > > did this, but you know your use case a lot better than I do. If >> it >> >> > were >> >> > > > me, I would use filter queries and do everything possible to >> re-use the >> >> > > > same filters, and I would cache them. >> >> > > > >> >> > > > Thanks, >> >> > > > Shawn >> >> > > > >> >> > > > >> >> > > >> >> > >> >> >> >>
