On 10/1/07, Robert Purdy [EMAIL PROTECTED] wrote:
Hi there, I am having some major CPU performance problems with heavy user
load with solr 1.2. I currently have approximately 4 million documents in
the index and I am doing some pretty heavy faceting on multi-valued columns.
I know that doing facets are expensive on multi-valued columns but the CPU
seems to max out (400%) with apache bench with just 5 identical concurrent
requests
One can always max out CPU (unless one is IO bound) with concurrent
requests greater than the number of CPUs on the system. This isn't a
problem by itself and would exist even if Solr were an order of
magnitude slower or faster.
You should be looking at things the peak throughput (queries per sec)
you need to support and the latency of the requests (look at the 90
percentile, or whatever).
and I have the potential for a lot more concurrent requests then
that with my large number of users that hit our site per day and I am
wondering if there are any workarounds. Currently I am running the out of
the box solr solution (Example jetty application with my own schema.xml and
solrconfig.xml) on a dual Intel Duo core 64 bit box with 8 gigs of ram
allocated to the start.jar process dedicated to solr with no slaves.
I have set up some aggressive caching in the solrconfig.xml for the
filtercache (class=solr.LRUCachesize=300 initialSize=200) and
have the HashDocSet set to 1 to help with faceting, but still I am
getting some pretty poor performance. I have also tried autowarming the
facets by performing a query that hits all my multivalued facets with no
facet limits across all the documents in the index. This does seem to reduce
my query times by a lot because the filtercache grows to about 2.1 million
lookups and finishes the query in about 70 secs.
OK, that's long. So focus on the latency of a single request instead
of jumping straight to load testing.
2.1 million is a lot - what's the field with the largest number of
unique values that you are faceting on?
However I have noticed an
issue with this because each time I do an optimize or a commit after
prewarming the facets the cache gets cleared, according to the stats on the
admin page, but the RSize does not shink for the process, and the queries
get slow again, so I prewarm the facets again and the memory usage keeps
growing like the cache is not being recycled
The old searcher and cache won't be discarded until all requests using
it have completed.
and as a results the prewarm
query starts to get slower and slower as each time this occurs (after about
5 times of prewarms and then commit the query takes about 30 mins... ugh)
and almost run out of memory.
Any thoughts on how to help improve this and fix the memory issue?
You could try the minDf param to reduce the number of facets stored in
the cache and reduce memory consumption.
-Yonik
