bq. How slow is "around commit points really slow"? You could at least lessen the pain here by committing less often if you can stand the latency....
They are shamelessly slow, like 60-70 seconds. While normal searches are within 1-3 seconds range. And, yes. your idea is right and what we are pursuing: less commits. However we do have shards that are hot because we need to keep them that hot, i.e. we commit as often as data arrives. This is where the slow searches pop up. bq. Often users are more disturbed by getting (numbers from thin air) 2 second responses occasionally spiking to 20 seconds with an average of 3 seconds than getting all responses between 4 and 6 seconds with an average of 5. yes, I believe so too. So at the moment, the call for using post-filtering or cache is more or less for business folks to make. We have been looking into other things, like making our shards as small as possible. This a parallel route to making our cache efficient. Thanks, Dmitry On Thu, Dec 5, 2013 at 3:59 PM, Erick Erickson <erickerick...@gmail.com>wrote: > bq: To be sure we are using cost 101 and no cache > > The guy who wrote the code is really good, but I'm paranoid too so I use > 101. Based on the number of off-by-one errors I've coded :)... > > How slow is "around commit points really slow"? You could at least lessen > the pain here by committing less often if you can stand the latency.... > > But otherwise you've pretty much nailed your options. One approach is to > give users _predictable_ responses, not necessarily the best average. Often > users are more disturbed by getting (numbers from thin air) 2 second > responses occasionally spiking to 20 seconds with an average of 3 seconds > than getting all responses between 4 and 6 seconds with an average of 5. > > FWIW, > Erick > > > On Thu, Dec 5, 2013 at 7:39 AM, Dmitry Kan <solrexp...@gmail.com> wrote: > > > Thanks Erick! > > To be sure we are using cost 101 and no cache. It seems to affect on > > searches as we expected. > > > > Basically with cache on we see more "fat" spikes around commit points, as > > cache is getting flushed (we don't rerun too many entries from old > cache). > > But when the post-filtering is involved, those spikes are thinner, but > the > > rest of the queries take about 2 seconds longer (our queries are pretty > > heavy duty stuff). > > > > So the post-filtering gives an option of making trade-offs between query > > times for all users during normal execution and query times during > commits. > > To rephrase we have 2 options: > > > > 1. Make all searches somewhat slower for all users and avoid really slow > > searches around commit points: post-filtering option > > > > OR > > > > 2. Make majority of searches really fast, but around commit points really > > slow: normal with cache option > > > > Dmitry > > > > > > On Wed, Dec 4, 2013 at 3:34 PM, Erick Erickson <erickerick...@gmail.com > > >wrote: > > > > > OK, so cache=false and cost=100 should do it, see: > > > http://searchhub.org/2012/02/22/custom-security-filtering-in-solr/ > > > > > > Best, > > > Erick > > > > > > > > > On Wed, Dec 4, 2013 at 5:56 AM, Dmitry Kan <solrexp...@gmail.com> > wrote: > > > > > > > Thanks Yonik. > > > > > > > > For our use case, we would like to skip caching only one particular > > > filter > > > > cache, yet apply a high cost for it to make sure it executes last of > > all > > > > filter queries. > > > > > > > > So this means, the rest of the fqs will execute and cache as usual. > > > > > > > > > > > > > > > > > > > > On Tue, Dec 3, 2013 at 9:58 PM, Yonik Seeley <yo...@heliosearch.com> > > > > wrote: > > > > > > > > > On Tue, Dec 3, 2013 at 4:45 AM, Dmitry Kan <solrexp...@gmail.com> > > > wrote: > > > > > > ok, we were able to confirm the behavior regarding not caching > the > > > > filter > > > > > > query. It works as expected. It does not cache with > {!cache=false}. > > > > > > > > > > > > We are still looking into clarifying the cost assignment: i.e. > > > whether > > > > it > > > > > > works as expected for long boolean filter queries. > > > > > > > > > > Yes, filters should be ordered by cost (cheapest first) whenever > you > > > > > use {!cache=false} > > > > > > > > > > -Yonik > > > > > http://heliosearch.com -- making solr shine > > > > > > > > > > > > > > > > > > > > > -- > > > > Dmitry > > > > Blog: http://dmitrykan.blogspot.com > > > > Twitter: twitter.com/dmitrykan > > > > > > > > > > > > > > > -- > > Dmitry > > Blog: http://dmitrykan.blogspot.com > > Twitter: twitter.com/dmitrykan > > > -- Dmitry Blog: http://dmitrykan.blogspot.com Twitter: twitter.com/dmitrykan
