Another option is to remove autowarming, and instead create a small bunch of queries that go most of the way. If you sort on a field, do that sort; facet queries also. This will load the basic Lucene data structures. Also, just getting the index data loaded into the OS disk cache helps a lot.
On Wed, May 9, 2012 at 1:53 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 5/9/2012 7:01 AM, richard.pog...@holidaylettings.co.uk wrote: >> >> We are testing an updated version of our Solr server running solr 3.5.0 >> and we are experiencing some performance issues with regard to updates and >> commits. >> >> Searches are working well. >> >> There are approximately 80,000 documents and the index is about 2.5 GB. >> This does not seem to be extreme based on other implementations I have seen. > > > There seems to be no such thing as typical. My 11.5 million document > indexes are nearing 21 GB. Your documents are huge when compared to mine, > but I'm sure that someone out there has even larger documents. > > >> An attempt to add a single document and commit is taking many minutes but >> the time taken is not consistent. >> >> Despite being in very light usage the java process seems to consistently >> require an entire CPU core. >> >> Optimising the index produced a limited improvement but this also takes a >> good deal of time to run. >> >> I am at a slight loss as to what the issue may be, I am interested in what >> others think would be a normal commit time with this sort of data volume. >> >> I am thinking there may be an issue with the schema but I am just >> guessing. > > > Frequent optimizes should be avoided. Optimizing is not inherently bad, but > it should happen during off hours, and index updates/deletes should be > suspended until the optimize is complete. > > I believe it's fairly normal for a commit operation to use one core, at > least with 3.x. I have not yet used trunk, so I cannot say whether the > situation will be better on 4.0. > > Without concrete information, I would guess that it's cache warming that's > making it go slow. You can see the total warmupTime on the Statistics page > in the admin GUI: > > http://HOST:PORT/solr/CORENAME/admin/stats.jsp > > Once there, search the page for warmup. The first entries found (in the > Core section) will be the total warmup time for the entire searcher. > Searching further, you will find additional entries in the Cache section, > those will tell you which of the caches is taking the majority of the > warmupTime. > > If it is indeed the warmup that is going slowly, you can make it go faster > by reducing the autowarm counts for your caches. If you find (as I have) > that even a small number of warming queries still goes slowly, you'll need > to see if you can identify the slow queries before they get executed and use > LocalParams options to not cache those queries. > > If the total warmupTime is low, then you can ignore everything I've said > above and keep looking for the cause, in which case answers to the following > questions may be helpful: > > How often are you updating your index? How many documents are in a typical > update? Are you using explicit commits, autoCommit, or perhaps both? If > autoCommit is enabled, what are the settings? > > Another thing that results in very slow queries is not having enough free > memory to cache the majority (or entirety) of the index. > > Thanks, > Shawn > -- Lance Norskog goks...@gmail.com
