Well, I’m not sure what to say. I’ve been observing a noticeable latency decrease over the first few thousand queries. I’m not doing anything too tricky either. Same exact query pattern, only one fq, always on the same field, no faceting. The only potential suspects that occur to me could be that it’s a large index (although properly sharded to fit in system memory), and that it’s doing geo filtering & ordering.
Since I don’t have a good mechanism for many queries, I’ll probably just do a few queries in firstSearcher for now and cross my fingers, but I’m not optimistic. For what it’s worth, I did verify that a replica doesn’t make itself available to other nodes until after the firstSearcher queries are completed. On 7/21/14, 5:57 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: >I've never seen it necessary to run "thousands of queries" >to warm Solr. Usually less than a dozen will work fine. My >challenge would be for you to measure performance differences >on queries after running, say, 12 well-chosen queries as >opposed to hundreds/thousands. I bet that if >1> you search across all the relevant fields, you'll fill up the > low-level caches for those fields. >2> you facet on all the fields you intend to facet on. >3> you sort on all the fields you intend to sort on. >4> you specify some filter queries. This is fuzzy since > really depends on you being able to predict what > those will be for firstSearcher. Things like "in the > last day/week/month" can be pre-configured, but > others you won't get. BTW, here's a blog about > why "in the last day" fq clauses can be tricky. > http://searchhub.org/2012/02/23/date-math-now-and-filter-queries/ > >that you'll pretty much nail warmup and be fine. Note that >you can do all the faceting on a single query. Specifying >the primary, secondary & etc. sorts will fill those caches. > >Best, >Erick > > >On Mon, Jul 21, 2014 at 5:07 PM, Jeff Wartes <jwar...@whitepages.com> >wrote: > >> >> On 7/21/14, 4:50 PM, "Shawn Heisey" <s...@elyograg.org> wrote: >> >> >On 7/21/2014 5:37 PM, Jeff Wartes wrote: >> >> I¹d like to ensure an extended warmup is done on each SolrCloud node >> >>prior to that node serving traffic. >> >> I can do certain things prior to starting Solr, such as pump the >>index >> >>dir through /dev/null to pre-warm the filesystem cache, and >>post-start I >> >>can use the ping handler with a health check file to prevent the node >> >>from entering the clients load balancer until I¹m ready. >> >> What I seem to be missing is control over when a node starts >> >>participating in queries sent to the other nodes. >> >> >> >> I can, of course, add solrconfig.xml firstSearcher queries, which I >> >>assume (and fervently hope!) happens before a node registers itself in >> >>ZK clusterstate.json as ready for work, but that doesn¹t scale so well >> >>if I want that initial warmup to run thousands of queries, or run them >> >>with some paralleism. I¹m storing solrconfig.xml in ZK, so I¹m >>sensitive >> >>to the size. >> >> >> >> Any ideas, or corrections to my assumptions? >> > >> >I think that firstSearcher/newSearcher (and making sure useColdSearcher >> >is set to false) is going to be the only way you can do this in a way >> >that's compatible with SolrCloud. If you were doing manual distributed >> >search without SolrCloud, you'd have more options available. >> > >> >If useColdSearcher is set to false, that should keep *everything* from >> >using the searcher until the warmup has finished. I cannot be certain >> >that this is the case, but I have some reasonable confidence that this >> >is how it works. If you find that it doesn't behave this way, I'd call >> >it a bug. >> > >> >Thanks, >> >Shawn >> >> >> Thanks for the quick reply. Since distributed search latency is the max >>of >> the shard sub-requests, I¹m trying my best to minimize any spikes in >> cluster latency due to node restarts. >> I double-checked useColdSearcher was false, but the doc says this means >> requests ³block until the first searcher is done warming², which >> translates pretty clearly to ³latency spike². The more I think about it, >> the more worried I am that a node might indeed register itself in >> live_nodes and get distributed requests before it¹s got a searcher to >>work >> with. *Especially* if I have lots of serial firstSearcher queries. >> >> I¹ll look through the code myself tomorrow, but if anyone can help >> confirm/deny the order of operations here, I¹d appreciate it. >>
