Thanks Erick.
I have added queries for "firstSearcher" and "newSearcher". After
startup, pmap shows well populated mmap entries and have better QTimes
than before.
However, phrase queries (edismax with pf2) are still sluggish. The
QTimes increase as the number of words in a phrase increase. None of the
mmap "warming" seem to have any impact on this. Am I missing anything?
Thanks.
On 9/24/16 5:20 PM, Erick Erickson wrote:
Hmm..
About <1>: Yep, GC is one of the "more art than science" bits of
Java/Solr. Siiiggggh.
About <2>: that's what autowarming is about. Particularly the
filterCache and queryResultCache. My guess is that you have the
autowarm count on those two caches set to zero. Try setting it to some
modest number like 16 or 32. The whole _point_ of those parameters is
to smooth out these kinds of spikes. Additionally, the newSearcher
event (also in solrconfig.xml) is explicitly intended to allow you to
hard-code queries that fill the internal caches as well as the mmap OS
memory from disk, people include facets, sorts and the like in that
event. It's fired every time a new searcher is opened (i.e. whenever
you commit and open a new searcher)...
FirstSearcher is for restarts. The difference is that newSearcher
presumes Solr has been running for a while and the autowarm counts
have something to work from. OTOH, when you start Solr there's no
history to autowarm so firstSeracher can be quite a bit more complex
than newSearcher. Practically, most people just copy newSearcher into
firstSearcher on the assumption that restarting Solr is pretty
rare.....
about <3> MMap stuff will be controlled by the OS I think. I actually
worked with a much more primitive system at one point that would be
dog-slow during off-hours. Someone wrote an equivalent of a cron job
to tickle the app upon occasion to prevent periodic slowness.....
for a nauseating set of details about hard and soft commits, see:
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Best,
Erick
On Sat, Sep 24, 2016 at 11:35 AM, Rallavagu <rallav...@gmail.com> wrote:
On 9/22/16 5:59 AM, Shawn Heisey wrote:
On 9/22/2016 5:46 AM, Muhammad Zahid Iqbal wrote:
Did you find any solution to slow searches? As far as I know jetty
container default configuration is bit slow for large production
environment.
This might be true for the default configuration that comes with a
completely stock jetty downloaded from eclipse.org, but the jetty
configuration that *Solr* ships with is adequate for just about any Solr
installation. The Solr configuration may require adjustment as the
query load increases, but the jetty configuration usually doesn't.
Thanks,
Shawn
It turned out to be a "sequence of performance testing sessions" in order to
locate slowness. Though I am not completely done with it, here are my
finding so far. We are using NRT configuration (warm up count to 0 for
caches and NRTCachingDirectoryFactory for index directory)
1. Essentially, solr searches (particularly with edismax and relevance)
generate lot of "garbage" that makes GC activity to kick in more often. This
becomes even more when facets are included. This has huge impact on QTimes
(I have 12g heap and configured 6g to NewSize).
2. After a fresh restart (or core reload) when searches are performed, Solr
would initially "populate" mmap entries and this is adding to total QTimes
(I have made sure that index files are cached at filesystem layer using
vmtouch - https://hoytech.com/vmtouch). When run the same test again with
mmap entries populated from previous tests, it shows improved QTimes
relative to previous test.
3. Seems the populated mmap entries are flushed away after certain idle time
(not sure if it is controlled by Solr or underlying OS). This will make
subsequent searches to fetch from "disk" (even though the disk items are
cached by OS).
So, what I am gonna try next is to tune the field(s) for facets to reduce
the index size if possible. Though I am not sure, if it will have impact but
would attempt to change the "caches" even though they will be invalidated
after a softCommit (every 10 minutes in my case).
Any other tips/clues/suggestions are welcome. Thanks.
